Sentiment analysis at the Fringe, part 1

With all the audience reviews of shows at this year’s Edinburgh Fringe Festival scraped and counted, I decided to dig a little deeper into what they actually represent. Lots of reviews does not necessarily mean lots of good reviews. Over the next several posts, I’ll be trying a number of different approaches to sentiment analysis

I’m starting with a brute force approach, using sentiment lexicons applied word-by-word. The idea is that some words can be categorised as positive and some as negative, so if you count the number of those words in your text, you can calculate if is overall positive or negative. This doesn’t consider context, or deal with negation (for example, “not good” would have the word “good” considered as positive). Nonetheless, it is a pretty simple way to get a sense of your data: some reviews might be wrongly classified, but a lot will be close enough.

For this work, I used the tidytext package in R, and drew a lot from this chapter of the book by Julia Silge and Dave Robinson, who created the package. Unlike the example there, though, I’m using data.table and at this stage am looking at a review level rather than a show (or chapter/book) level.

Comparing lexicons

The lexicon you use is important, because they will give you different results. If you have a particular context for your work, you might need a lexicon that deals with a different vocabulary. It’s also worth considering that language is constantly evolving; words can move from neutral to highly emotive, or even totally flip. Therefore lexicons can go out of date if not maintained, but also might not be appropriate for historical text.

There are three general use lexicons included in tidytext: Bing, AFINN and NRC.

Table 1: Sentiment lexicons
Sentiment lexicon Scoring approach Positive words Negative words
Bing Words categorised as positive or negative 2006 4782
AFINN Words scored between -5 and 5, with -5 being extremely negative and 5 being extremely positive 878 1597
NRC Words categorised as belonging to any of: positive, negative, anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. They can belong to multiple categories. 2312 3324

You can see straight away that these will produce different results. For example, Bing has fewer positive words but more negative words than NRC. Meanwhile, AFINN has far fewer words assigned as positive or negative than either of the others. However, AFINN does consider a scale of positivity and negativity, which perhaps might give it a more subtle approach.

I’ll be trying out all three of these in order to compare the outcomes on the Fringe reviews data.

Analaysing for sentiment

To start with, I took the text of the reviews I’d already scraped and prepared for a previous post. As I want to evaluate how accurate the analysis is, I had to label a portion of these reviews to have something to assess against. This would need to be the same set of reviews for each approach I use, including those I cover in future posts. I’ll also need labelled data for some other approaches that involve supervised machine learning; for the current approach this is unnecessary. Therefore, I took a random sample of 400 reviews, labelled them manually as positive, negative or mixed, and then randomly split these so I had a 200 review dataset to use for testing.

Taking this test dataset (called test_labelled in my code), I proceeded in my analysis.

# Load required libraries

library(data.table)
library(tidytext)

# Create a dataset splitting the reviews into one word per line

words_in_reviews <- unnest_tokens(test_labelled, word, review)

### Using bing --------------

# Add bing sentiment to dataset

bing_sentiment <- merge(words_in_reviews, get_sentiments("bing"))

# Count the number of positive/negative words by review
# (identified by an original_index number)

positive_bing_words <- bing_sentiment[sentiment == "positive", 
                                      .(positive_words = .N), 
                                      by = original_index]
negative_bing_words <- bing_sentiment[sentiment == "negative", 
                                      .(negative_words = .N), 
                                      by = original_index]

all_bing_words <- merge(positive_bing_words, negative_bing_words, all = TRUE)

all_bing_words[is.na(all_bing_words)] <- 0

# Calculate net positive/negative score
# Less than 0 is negative, more than 0 is positive

all_bing_words[, combined_sentiment := positive_words - negative_words]
all_bing_words[, overall_sentiment := ifelse(combined_sentiment < 0, 
                                             "negative", 
                                             ifelse(combined_sentiment == 0, 
                                                    "mixed", 
                                                    "positive"))]

# Combine sentiment score with reviews dataset

bing_sentiment_outcome <- merge(test_labelled,
                                all_bing_words[, .(original_index, 
                                                   overall_sentiment)],
                           by = "original_index", all.x = TRUE)

# No sentiment is assigned mixed

bing_sentiment_outcome[is.na(overall_sentiment), 
                       overall_sentiment := "mixed"]

# New column to show if the manually assigned sentiment_label matches the 
# overall_sentiment calculated from the lexicon

bing_sentiment_outcome[, match := sentiment_label == overall_sentiment]

The above code can be repeated for the NRC lexicon, because it is similar to Bing in having words assigned as “positive” or “negative”. However, with the AFINN lexicon, the words all have scores instead, so the approach is slightly different.

### Using AFINN ---------------

# Add AFINN sentiment to dataset

afinn_sentiment <- merge(words_in_reviews, get_sentiments("afinn"))

# Sum the score of words by review#
# (This is the different section)

afinn_reviews <- afinn_sentiment[, .(sentiment_score = sum(score)), 
                                 by = original_index]

# Calculate net positive/negative score
# Less than 0 is negative, more than 0 is positive

afinn_reviews[, overall_sentiment := ifelse(sentiment_score < 0, 
                                            "negative", 
                                            ifelse(sentiment_score == 0, 
                                                   "mixed", 
                                                   "positive"))]

# Combine sentiment score with reviews dataset

afinn_sentiment_outcome <- merge(test_labelled,
                                 afinn_reviews[, .(original_index, 
                                                   overall_sentiment)],
                                by = "original_index", all.x = TRUE)

# No sentiment is assigned mixed

afinn_sentiment_outcome[is.na(overall_sentiment), 
                        overall_sentiment := "mixed"]

# New column to show if the manually assigned sentiment_label matches the 
# overall_sentiment calculated from the lexicon

afinn_sentiment_outcome[, match := sentiment_label == overall_sentiment]

Assessing the outcomes

By quickly counting the number of reviews where the manually assigned sentiment label matched that derived from the sentiment lexicon scores, we can compare how the lexicons did overall.

# Count how many matches each lexicon achieved

bing_sentiment_outcome[, .("Bing outcome" = .N), by = match]
##    match Bing outcome
## 1: FALSE           31
## 2:  TRUE          169

afinn_sentiment_outcome[, .("AFINN outcome" = .N), by = match]
##    match AFINN outcome
## 1: FALSE            27
## 2:  TRUE           173

nrc_sentiment_outcome[, .("NRC outcome" = .N), by = match]
##    match NRC outcome
## 1: FALSE          44
## 2:  TRUE         156

AFINN does best, with 173/200 reviews matching their label. Bing is very close behind, though NRC misses almost a quarter of the labels. Some more detailed breakdown shows us this:

Table 2: Sentiment lexicons and their results
Dataset True positive True negative True mixed False positive False negative False mixed
Actual labels 184 12 4 NA NA NA
Bing 164 5 0 9 9 13
AFINN 170 3 0 10 6 11
NRC 152 3 1 9 13 22

The first thing to note is that the original datset is overwhelmingly positive, with 184/200 positive reviews. For all lexicons, most of the confusion comes around the ‘mixed’ category - they all classify a LOT more reviews as mixed, which would mean that they had an equal number of positive and negative words, or a balanced score in the case of the AFINN lexicon. They would also count as mixed in this instance if there were no known positive or negative words in the review. Of course, my categorisation of what counts as a mixed review is subjective, so that is also a possible issue.

Interestingly, AFINN, which does best overall, returns the most false positives and joint least true negatives, so its success might be down to a very unbalanced dataset and a tendency to bias positive rather than a really excellent ability to distinguish between reviews.

What went wrong?

Having looked through reviews that were miscategorised, here are some reasons why.

“What a show.These girls are stars.”

All the lexicons struggled with this one. Although clearly positive to a reader, its concision doesn’t give the lexicons much to work with, and none of them consider “stars” a positive, which points to the fact these are general purpose lexicons.

“Fantastic performance by such a talented young cast. Disturbing subject matter, well delivered. Watch out for the stand out performance by the worryingly creepy doctor.”

For this review, the writer has referred to some subject matter and character elements, which are only meant as a description rather then relating to their reaction, but this leads some of the lexicons to add some unintented negative scoring.

“Didn’t know that to get multiple stars and happy reviews at the EdFringe, performers have to resort to sex and toilet ‘jokes’. This is one of the two shows I wanted to walk out on. The spousal gag was old after 2 minutes and then there was another 56-58 to tolerate! While many reviews below are positive , the audience I sat in was quite silent throughout and the burning anger at the show from the audience was palpable. Maybe EdFringe and its reviewers think they are breaking taboos and boundaries by fostering an environment of low level genital and toilet ‘humour’, and EdFringe is certainly raking in pounds this way; but it really ruined my 2 weeks at the Fringe to (a) see sensitive and truly funny and whimsical shows ignored by the venues in their tweets, (b) read inaccurate teenage ‘professional’ reviews from outfits like Broadwaybaby and Theatre Weekly while (c) promoting low level puerile humour as in this show. Overall I would give 0 stars to this show and to EdFringe itself for its imbalance in promoting puerile standup ‘comedy’ vs other genres in the arts.”

This review is almost the opposite, in that everything it says about the show is negative, but there are a few mentions of other hypotehtical better shows and other reviews that must balance out the sentiment score. I noticed that this was labelled differently by all lexicons: AFINN said it was positve, Bing said it was mixed, and only NRC correctly identified it as negative.

Had small audience but didn’t short change us. A very funny interactive show. Went as seen at Cambridge footlights show and we weren’t disappointed. Recommended.

Finally, this is an example of a review where negation changes the meaning (“weren’t disappointed”). By scoring sentiment on a word-by-word basis, the lexicon analysis misses this context and assigns a negative score for that word.

Conclusion

Firstly, I think it is worth pointing out that the results for the sentiment lexicons weren’t terrible. The AFINN lexicon was right 86.5% of the time. Its results would tell you that the reviews were overwhelmingly positive, which is true.

But there are clearly ways this approach could be improved. Next time, I’ll look at how considering multiple words together impacts the results when using sentiment lexicons.

comments powered by Disqus