sentiment-analysis-with-vader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/CC_BY.png\"><br />\n",
    "\n",
    "Created by [Nathan Kelber](http://nkelber.com) and Ted Lawless for [JSTOR Labs](https://labs.jstor.org/) under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/)<br />\n",
    "For questions/comments/improvements, email nathan.kelber@ithaka.org.<br />\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis with VADER\n",
    "\n",
    "**Description:** This notebook describes Sentiment Analysis and demonstrates basic applications using:\n",
    "* VADER (Valence Aware Dictionary for sEntiment Reasoning), a rule-based algorithm\n",
    "* sci-kit learn's Multinomial Naive Bayes, a machine learning classifier\n",
    "\n",
    "**Use Case:** For Learners (Detailed explanation, not ideal for researchers)\n",
    "\n",
    "**Difficulty:** Beginner\n",
    "\n",
    "**Completion Time:** 60 minutes\n",
    "\n",
    "**Knowledge Required:** \n",
    "* Python Basics Series ([Start Python Basics I](./python-basics-1.ipynb))\n",
    "\n",
    "**Knowledge Recommended:** None\n",
    "\n",
    "**Data Format:** None\n",
    "\n",
    "**Libraries Used:** vaderSentiment, sklearn\n",
    "\n",
    "**Research Pipeline:** None\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methods for Sentiment Analysis\n",
    "\n",
    "Sentiment analysis can help an analyst discover whether feedback is positive, negative, or mixed. For example, a large company like Amazon or Walmart could use sentiment analysis on user reviews to determine whether a featured product should be promoted or discontinued. Sentiment analysis generally falls into two categories:\n",
    "\n",
    "* Rule-based algorithms\n",
    "* Machine Learning models \n",
    "\n",
    "### Rule-Based Algorithms\n",
    "\n",
    "Rule-based algorithms assign sentiment scores to particular words or multi-word constructions. Simple algorithms may simply assess each word individually in a feedback document and add up an overall score. More complex algorithms may assess multi-word (or n-gram) constructions and have special rules for addressing issues such as negation, emojis, and emoticons. They can detect the difference between \"bad\", \"not bad\", and \"bad ass\". Some algorithms also support emojis and emoticons, such as \"=)\" and \"😁\".\n",
    "\n",
    "### Machine Learning Models\n",
    "\n",
    "Machine learning models rely on feedback data that has already been assessed by humans to have a particular sentiment. Each piece of feedback is **labeled** by a human reader who may place the feedback into a particular category. The categories could be as simple as positive, negative, or neutral. As long as there exists **labeled** data, a machine learning model can often identify complex concepts. For example, a car manufacturer may desire to classify the sentiment of feedback from past buyers as: \"budget-conscious\", \"eco-conscious\", \"tech-enthusiastic\", \"luxury-driven\", \"performance-driven\", etc. Assuming there is an adequately labeled **training data** for each of these categories, a machine learning model could assign a score for each category. This could help analysts understand the brand better, answering questions about what consumers do or do not like about a particular vehicle.\n",
    "\n",
    "In the humanities, sentiment analysis could be used to track emerging trends on social media. For example, we might ask: \"How are Twitter or Reddit users responding to a particular government policy or public event?\" We could look at a hashtag like \"#blm\" and get a sense of national sentiment on the Black Lives Matter movement. The project [On the Books: Jim Crow and Algorithms of Resistance](https://onthebooks.lib.unc.edu/) is using machine classification to detect racist laws based on the pioneering work of [Pauli Murray](https://en.wikipedia.org/wiki/Pauli_Murray) and [Safiya Noble](https://en.wikipedia.org/wiki/Safiya_Noble)'s concept of \"algorithmic oppression\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VADER\n",
    "\n",
    "This notebook uses a rule-based algorithm named VADER (Valence Aware Dictionary and sEntiment Reasoner). VADER is a rule-based algorithm that is \"specifically attuned to sentiments expressed in social media.\" It relies on a specialized **lexicon** of words, phrases, and emojis. Each token in the lexicon is assigned a \"mean-sentiment rating\" between -4 (extremely negative) to 4 (extremely positive). Here are a few examples:\n",
    "\n",
    "|Token|Mean-Sentiment Rating|\n",
    "|---|---|\n",
    "|(:|2.2|\n",
    "|/:|-1.3|\n",
    "|):<|-1.9|\n",
    "|rotflmao|2.8|\n",
    "|aghast|-1.9|\n",
    "|awesome|3.1|\n",
    "|awful|-2.0|\n",
    "\n",
    "There are over 7500 tokens listed in VADER lexicon. (You can also add your own if you like.) VADER also considers grammatical and syntactical rules to measure intensity based on word order and sensitive relationships between terms. For example, it increases or decreases a sentiment based on degree modifers such as: \"The product is good\" versus \"the product is very good\" versus \"the product is marginally good.\" To read more about VADER, including how it works and to see its code, [visit the github page](https://github.com/cjhutto/vaderSentiment)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying the VADER Algorithm\n",
    "First, we need to import the SentimentIntensityAnalyzer. Here we assign the VADER lexicon object to a variable `sa`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the SentimentIntensityAnalyzer\n",
    "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
    "\n",
    "# Creat the variable sa to hold the VADER lexicon object \n",
    "sa = SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can preview the contents of the lexicon by using `sa.lexicon`. This will return a dictionary, where each key is a token and each value is a sentiment rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preview the lexicon contents\n",
    "# There are over 7500 tokens in the lexicon\n",
    "sa.lexicon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if a word is in the lexicon\n",
    "test_word = 'sweet' # The word to check for\n",
    "\n",
    "# Get the word's score or print a message for missing words\n",
    "sa.lexicon.get(test_word, 'No score for that word') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to do our analysis, we will use a very small sample of 8 user reviews. Each review is a simple text string inside a list variable called `product_reviews`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a list of product reviews\n",
    "\n",
    "product_reviews = [\n",
    "    'I love this product. It helps me get so much work done. I tell everyone about what a great thing it is.',\n",
    "    'This product is defective. I feel like it is broken because it does not do what it promises. Do not buy this.',\n",
    "    'Do yourself a favor and buy this product as soon as possible. I recommend it to everyone I know. It has saved me so much time!',\n",
    "    'This product is overpriced and useless. It was a waste of money and it made all my hair fall out.',\n",
    "    'Works like a dream and it is a bargain! It solves my problems with ease. I bought two!',\n",
    "    'Do not buy! This product is a ripoff. I wish it was better, but it fails constantly. What a mistake!',\n",
    "    'This thing is garbage. Do yourself a favor and save the money. Mine is a dumpster fire and fell apart.',\n",
    "    'I adore this product. =) It makes my life so much easier. And it is a deal!'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will analyze each product and assign it a \"normalized, weighted composite score\" based on summing the valence scores of each word in the lexicon (with some adjustments based on word order and other rules). VADER measures the proportion of text that falls into positive, negative, and neutral sentiment. The result is a sentiment score that falls between -1 (the most negative) and +1 (the most positive). (This is different from the lexicon scores that fall between -4 to +4!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For each review in our `product_reviews` list\n",
    "# Store a polarity score in `scores`\n",
    "# Then print the score followed by the review\n",
    "for review in product_reviews:\n",
    "    scores = sa.polarity_scores(review)\n",
    "    print(scores['compound'], review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our simple analysis does a fairly good job of assessing positive and negative sentiment. Notice that our second to last review was not very accurate though:\n",
    "> 0.5423 This thing is garbage. Do yourself a favor and save the money. Mine started on fire and fell apart.\n",
    "\n",
    "The VADER lexicon contains the following entries:\n",
    "\n",
    "|Token|Mean-Sentiment Rating|\n",
    "|---|---|\n",
    "|favor|1.7|\n",
    "|fire|-1.4|\n",
    "\n",
    "VADER assigns a value of -1.4 for \"fire\" but \"fire\" can also have a positive connotation, such as \"straight fire.\" However, words like \"garbage\" and \"dumpster,\" as in \"dumpster fire,\" are less ambiguous. If a specific token is not found in the VADER lexicon, it is considered to be neutral. Like any other statistical approach, the process benefits from having more data. In this case, the sentences are very short and several significant words do not happen to exist in our lexicon. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Tokens to the VADER Lexicon\n",
    "\n",
    "The `sa.lexicon` is a simple dictionary, so we can add words that we want included. There are some guidelines for best scoring practices included in the academic paper linked on [VADER's github repository](https://github.com/cjhutto/vaderSentiment). (Remember that lexicon tokens are scored from -4 to +4.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding the dictionary of `new_words`\n",
    "# to sa.lexicon\n",
    "\n",
    "new_words = {\n",
    "    'garbage': -2.0,\n",
    "    'dumpster': -3.1,\n",
    "}\n",
    "\n",
    "sa.lexicon.update(new_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try our analysis again with the new lexicon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For each review in our `product_reviews` list\n",
    "# Store a polarity score in `scores`\n",
    "# Then print the score followed by the review\n",
    "\n",
    "for review in product_reviews:\n",
    "    scores = sa.polarity_scores(review)\n",
    "    print(scores['compound'], review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment analysis with machine learning\n",
    "\n",
    "The primary advantage of using a machine learning classifier for sentiment analysis is there is no need to maintain a lexicon, assign sentiment scores to particular words, develop linguistic rules based on grammatical structures (negation, intensifiers), or keep track of novel expressions (slang, emoticons, etc.). \n",
    "\n",
    "In order to deploy a machine learning classifier, however, we need accurate, human-labeled data. If there is no existing labeled data, then collecting and labeling the data can be a laborious task. Without labeled data, there is no way for us to know if a machine learning classifier is accurate.\n",
    "\n",
    "There are a variety of classifiers, such as Naive Bayes, Decision Trees, Logistic Regression, K-Nearest Neighbor, Deep Learning, Support Vector Machines. We will use a [Naive Bayes](https://en.wikipedia.org/wiki/Naive_Bayes_classifier) approach with the Python library [scikit-learn](https://scikit-learn.org/stable/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train/Validation/Test Split\n",
    "\n",
    "While we could throw all our data into training our machine, we would have no way to know if the machine's predictions were accurate. For this reason, it is common practice to split the labeled data into two or three groups. \n",
    "\n",
    "A simple train/test split is usually 80/20 or 70/30. About 75% of the data is used to train the machine while 25% of the data is held out. This \"held out\" test data is then used to discover whether the machine's predictions generalize to data it has not yet seen.\n",
    "\n",
    "Another, more rigorous, approach is to split the data into three groups: Train, Validation, Test. The mix is usually about 60/20/20. In practice, there should be little difference between the Validation and Test sets. However, usually the model is trained on Train set, the hyperparameters are tuned on the Validation set, and the Test set confirms the model's accuracy on novel data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifying film review sentiment snippets\n",
    "\n",
    "The fundamental methods for employing machine learning are fairly straightforward. The hard part, by far, is getting access to good data. VADER contains several labeled datasets for testing and demonstration including tweets, NY Times editorials, movie reviews, and Amazon reviews. We could certainly use this data to improve the VADER model through modifying our lexicon and implementing new linguistic rules, but we can also use them to train a machine learning classifer.\n",
    "\n",
    "First, we need to download and extract the data from the VADER GitHub repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download sample datasets from VADER and decompress them\n",
    "import urllib.request\n",
    "import tarfile\n",
    "import os\n",
    "\n",
    "# Move to the data directory\n",
    "os.chdir('./data')\n",
    "\n",
    "# Retrieve the file\n",
    "file_name = 'sentiment_data.tar.gz'\n",
    "\n",
    "url = 'https://github.com/cjhutto/vaderSentiment/raw/master/additional_resources/hutto_ICWSM_2014.tar.gz'\n",
    "urllib.request.urlretrieve(url, file_name)\n",
    "print('Sample datasets retrieved.')\n",
    "\n",
    "# Uncompress the file\n",
    "tar = tarfile.open(file_name, 'r:gz')\n",
    "tar.extractall()\n",
    "tar.close\n",
    "print('Datasets uncompressed.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a pandas dataframe to store our film review snippets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import data into Pandas\n",
    "import pandas as pd\n",
    "movies = pd.read_csv('./hutto_ICWSM_2014/movieReviewSnippets_GroundTruth.txt', sep=\"\\t\", header=None)\n",
    "movies.columns = [\"id\", \"sentiment\", \"text\"]\n",
    "movies = movies.set_index('id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's expand the column width in Pandas so we can read the full snippets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Expand Pandas display\n",
    "pd.set_option('max_colwidth', 400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Preview the data\n",
    "movies.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look at the `'sentiment'` column, we can see the total number of snippets and the min/max for the sentiment scoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['sentiment'].describe().round(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a bag of words corpus from the text of the `movies` dataframe using a tokenizer from the Natural Language Toolkit called `casual_tokenize` which specializes in informal language, particularly Twitter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tokenize our dataset, create a bag of words corpus using Counter() objects\n",
    "from nltk.tokenize import casual_tokenize\n",
    "bow_corpus = []\n",
    "from collections import Counter\n",
    "for text in movies.text:\n",
    "    bow_corpus.append(Counter(casual_tokenize(text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preview our Python list of bags of words.\n",
    "# Each snippet is converted into a Counter() object\n",
    "# which counts the number of occurrences of its tokens\n",
    "bow_corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert our dataframe into a sparse matrix \n",
    "# where each row represents a snippet\n",
    "# and each row is a particular token\n",
    "df_bows = pd.DataFrame.from_records(bow_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Preview our dataframe \n",
    "df_bows.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_bows.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data has a lot of NaN entries which will throw an error when we start our training, so we fill these in with 0s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change NaNs to 0\n",
    "df_bows = df_bows.fillna(0).astype(int)\n",
    "df_bows.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preview the data frame with columnns from a given snippet\n",
    "columns = list(bow_corpus[3].keys())\n",
    "\n",
    "df_bows.head()[columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we actually train the model. We import sklearn and fit the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "nb = MultinomialNB()\n",
    "\n",
    "# Convert the output variable (sentiment float) to a discrete label\n",
    "nb = nb.fit(df_bows, movies.sentiment > 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert our binary classification (0 or 1) to -4 or 4\n",
    "# This will help us compare it to the \"ground truth\" sentiment\n",
    "movies['predicted_sentiment'] = nb.predict(df_bows) * 8 - 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We calculate the average absolute value of the prediction error\n",
    "# This is the Mean Absolute Error (MAE)\n",
    "movies['error'] = (movies.predicted_sentiment - movies.sentiment).abs()\n",
    "movies.error.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['sentiment_ispositive'] = (movies.sentiment > 0).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['predicted_ispositive'] = (movies.predicted_sentiment > 0).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movies['sentiment predicted_sentiment sentiment_ispositive predicted_ispositive'.split()].head(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the accuracy of our predictions (positive/negative)\n",
    "(movies.predicted_ispositive == movies.sentiment_ispositive).sum() / len(movies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "199px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}