Chapter13/02_nlp_with_textblob.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP with TextBlob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextBlob is a python library that provides a simple API for common NLP tasks and builds on the Natural Language Toolkit (nltk) and the Pattern web mining libraries. TextBlob facilitates part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and others."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports & Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:26.696873Z",
     "start_time": "2018-11-26T02:54:26.461634Z"
    }
   },
   "outputs": [],
   "source": [
    "% matplotlib inline\n",
    "import warnings\n",
    "from pathlib import Path\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Visualization\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# spacy, textblob and nltk for language processing\n",
    "from textblob import TextBlob, Word\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "\n",
    "# sklearn for feature extraction & modeling\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer, TfidfTransformer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB         # Naive Bayes\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import metrics\n",
    "from sklearn.externals import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:26.700048Z",
     "start_time": "2018-11-26T02:54:26.698100Z"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "pd.set_option('float_format', '{:,.2f}'.format)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load BBC Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate the use of TextBlob, we sample a BBC sports article with the headline ‘Robinson ready for difficult task’. Similar to spaCy and other libraries, the first step is to pass the document through a pipeline represented by the TextBlob object to assign annotations required for various tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:28.476829Z",
     "start_time": "2018-11-26T02:54:28.400043Z"
    }
   },
   "outputs": [],
   "source": [
    "path = Path('data', 'bbc')\n",
    "files = path.glob('**/*.txt')\n",
    "doc_list = []\n",
    "for i, file in enumerate(files):\n",
    "    topic = file.parts[-2]\n",
    "    article = file.read_text(encoding='latin1').split('\\n')\n",
    "    heading = article[0].strip()\n",
    "    body = ' '.join([l.strip() for l in article[1:]]).strip()\n",
    "    doc_list.append([topic, heading, body])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:28.805397Z",
     "start_time": "2018-11-26T02:54:28.786497Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2225 entries, 0 to 2224\n",
      "Data columns (total 3 columns):\n",
      "topic      2225 non-null object\n",
      "heading    2225 non-null object\n",
      "body       2225 non-null object\n",
      "dtypes: object(3)\n",
      "memory usage: 52.2+ KB\n"
     ]
    }
   ],
   "source": [
    "docs = pd.DataFrame(doc_list, columns=['topic', 'heading', 'body'])\n",
    "docs.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-21T15:03:08.577908Z",
     "start_time": "2018-11-21T15:03:08.572433Z"
    }
   },
   "source": [
    "## Introduction to TextBlob\n",
    "\n",
    "You should already have downloaded TextBlob, a Python library used to explore common NLP tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select random article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:32.080548Z",
     "start_time": "2018-11-26T02:54:32.072173Z"
    }
   },
   "outputs": [],
   "source": [
    "article = docs.sample(1).squeeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:32.490561Z",
     "start_time": "2018-11-26T02:54:32.482638Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic:\tSport\n",
      "\n",
      "Robinson ready for difficult task\n",
      "\n",
      "England coach Andy Robinson faces the first major test of his tenure as he tries to get back to winning ways after the Six Nations defeat by Wales.  Robinson is likely to make changes in the back row and centre after the 11-9 loss as he contemplates Sunday's set-to with France at Twickenham. Lewis Moody and Martin Corry could both return after missing the game with hamstring and shoulder problems. And the midfield pairing of Mathew Tait and Jamie Noon is also under threat. Olly Barkley immediately allowed England to generate better field position with his kicking game after replacing debutant Tait just before the hour. The Bath fly-half-cum-centre is likely to start against France, with either Tait or Noon dropping out.  Tait, given little opportunity to shine in attack, received praise from Robinson afterwards, even if the coach admitted Cardiff was an \"unforgiving place\" for the teenage prodigy. Robinson now has a tricky decision over whether to withdraw from the firing line, after just one outing, a player he regards as central to England's future. Tait himself, at least outwardly, appeared unaffected by the punishing treatment dished out to him by Gavin Henson in particular. \"I want more of that definitely,\" he said. \"Hopefully I can train hard this week and get selected for next week but we'll have to look at the video and wait and see. \"We were playing on our own 22 for a lot of the first half so it was quite difficult. I thought we defended reasonably well but we've just got to pick it up for France.\"  His Newcastle team-mate Noon hardly covered himself in glory in his first major Test. He missed a tackle on Michael Owen in the build-up to Wales' try, conceded a penalty at the breakdown, was turned over in another tackle and fumbled Gavin Henson's cross-kick into touch, all inside the first quarter. His contribution improved in the second half, but England clearly need more of a playmaker in the inside centre role. Up front, the line-out remains fallible, despite a superb performance from Chris Jones, whose athleticism came to the fore after stepping into the side for Moody. It is more likely the Leicester flanker will return on the open side for the more physical challenge posed by the French forwards, with Andy Hazell likely to make way. Lock Ben Kay also justified his recall with an impressive all-round display on his return to the side, but elsewhere England positives were thin on the ground.\n"
     ]
    }
   ],
   "source": [
    "print(f'Topic:\\t{article.topic.capitalize()}\\n\\n{article.heading}\\n')\n",
    "print(article.body.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:37.512003Z",
     "start_time": "2018-11-26T02:54:37.510065Z"
    }
   },
   "outputs": [],
   "source": [
    "parsed_body = TextBlob(article.body)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:38.178003Z",
     "start_time": "2018-11-26T02:54:38.131744Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordList(['England', 'coach', 'Andy', 'Robinson', 'faces', 'the', 'first', 'major', 'test', 'of', 'his', 'tenure', 'as', 'he', 'tries', 'to', 'get', 'back', 'to', 'winning', 'ways', 'after', 'the', 'Six', 'Nations', 'defeat', 'by', 'Wales', 'Robinson', 'is', 'likely', 'to', 'make', 'changes', 'in', 'the', 'back', 'row', 'and', 'centre', 'after', 'the', '11-9', 'loss', 'as', 'he', 'contemplates', 'Sunday', \"'s\", 'set-to', 'with', 'France', 'at', 'Twickenham', 'Lewis', 'Moody', 'and', 'Martin', 'Corry', 'could', 'both', 'return', 'after', 'missing', 'the', 'game', 'with', 'hamstring', 'and', 'shoulder', 'problems', 'And', 'the', 'midfield', 'pairing', 'of', 'Mathew', 'Tait', 'and', 'Jamie', 'Noon', 'is', 'also', 'under', 'threat', 'Olly', 'Barkley', 'immediately', 'allowed', 'England', 'to', 'generate', 'better', 'field', 'position', 'with', 'his', 'kicking', 'game', 'after', 'replacing', 'debutant', 'Tait', 'just', 'before', 'the', 'hour', 'The', 'Bath', 'fly-half-cum-centre', 'is', 'likely', 'to', 'start', 'against', 'France', 'with', 'either', 'Tait', 'or', 'Noon', 'dropping', 'out', 'Tait', 'given', 'little', 'opportunity', 'to', 'shine', 'in', 'attack', 'received', 'praise', 'from', 'Robinson', 'afterwards', 'even', 'if', 'the', 'coach', 'admitted', 'Cardiff', 'was', 'an', 'unforgiving', 'place', 'for', 'the', 'teenage', 'prodigy', 'Robinson', 'now', 'has', 'a', 'tricky', 'decision', 'over', 'whether', 'to', 'withdraw', 'from', 'the', 'firing', 'line', 'after', 'just', 'one', 'outing', 'a', 'player', 'he', 'regards', 'as', 'central', 'to', 'England', \"'s\", 'future', 'Tait', 'himself', 'at', 'least', 'outwardly', 'appeared', 'unaffected', 'by', 'the', 'punishing', 'treatment', 'dished', 'out', 'to', 'him', 'by', 'Gavin', 'Henson', 'in', 'particular', 'I', 'want', 'more', 'of', 'that', 'definitely', 'he', 'said', 'Hopefully', 'I', 'can', 'train', 'hard', 'this', 'week', 'and', 'get', 'selected', 'for', 'next', 'week', 'but', 'we', \"'ll\", 'have', 'to', 'look', 'at', 'the', 'video', 'and', 'wait', 'and', 'see', 'We', 'were', 'playing', 'on', 'our', 'own', '22', 'for', 'a', 'lot', 'of', 'the', 'first', 'half', 'so', 'it', 'was', 'quite', 'difficult', 'I', 'thought', 'we', 'defended', 'reasonably', 'well', 'but', 'we', \"'ve\", 'just', 'got', 'to', 'pick', 'it', 'up', 'for', 'France', 'His', 'Newcastle', 'team-mate', 'Noon', 'hardly', 'covered', 'himself', 'in', 'glory', 'in', 'his', 'first', 'major', 'Test', 'He', 'missed', 'a', 'tackle', 'on', 'Michael', 'Owen', 'in', 'the', 'build-up', 'to', 'Wales', 'try', 'conceded', 'a', 'penalty', 'at', 'the', 'breakdown', 'was', 'turned', 'over', 'in', 'another', 'tackle', 'and', 'fumbled', 'Gavin', 'Henson', \"'s\", 'cross-kick', 'into', 'touch', 'all', 'inside', 'the', 'first', 'quarter', 'His', 'contribution', 'improved', 'in', 'the', 'second', 'half', 'but', 'England', 'clearly', 'need', 'more', 'of', 'a', 'playmaker', 'in', 'the', 'inside', 'centre', 'role', 'Up', 'front', 'the', 'line-out', 'remains', 'fallible', 'despite', 'a', 'superb', 'performance', 'from', 'Chris', 'Jones', 'whose', 'athleticism', 'came', 'to', 'the', 'fore', 'after', 'stepping', 'into', 'the', 'side', 'for', 'Moody', 'It', 'is', 'more', 'likely', 'the', 'Leicester', 'flanker', 'will', 'return', 'on', 'the', 'open', 'side', 'for', 'the', 'more', 'physical', 'challenge', 'posed', 'by', 'the', 'French', 'forwards', 'with', 'Andy', 'Hazell', 'likely', 'to', 'make', 'way', 'Lock', 'Ben', 'Kay', 'also', 'justified', 'his', 'recall', 'with', 'an', 'impressive', 'all-round', 'display', 'on', 'his', 'return', 'to', 'the', 'side', 'but', 'elsewhere', 'England', 'positives', 'were', 'thin', 'on', 'the', 'ground'])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsed_body.words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence boundary detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:38.943758Z",
     "start_time": "2018-11-26T02:54:38.940200Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence(\"England coach Andy Robinson faces the first major test of his tenure as he tries to get back to winning ways after the Six Nations defeat by Wales.\"),\n",
       " Sentence(\"Robinson is likely to make changes in the back row and centre after the 11-9 loss as he contemplates Sunday's set-to with France at Twickenham.\"),\n",
       " Sentence(\"Lewis Moody and Martin Corry could both return after missing the game with hamstring and shoulder problems.\"),\n",
       " Sentence(\"And the midfield pairing of Mathew Tait and Jamie Noon is also under threat.\"),\n",
       " Sentence(\"Olly Barkley immediately allowed England to generate better field position with his kicking game after replacing debutant Tait just before the hour.\"),\n",
       " Sentence(\"The Bath fly-half-cum-centre is likely to start against France, with either Tait or Noon dropping out.\"),\n",
       " Sentence(\"Tait, given little opportunity to shine in attack, received praise from Robinson afterwards, even if the coach admitted Cardiff was an \"unforgiving place\" for the teenage prodigy.\"),\n",
       " Sentence(\"Robinson now has a tricky decision over whether to withdraw from the firing line, after just one outing, a player he regards as central to England's future.\"),\n",
       " Sentence(\"Tait himself, at least outwardly, appeared unaffected by the punishing treatment dished out to him by Gavin Henson in particular.\"),\n",
       " Sentence(\"\"I want more of that definitely,\" he said.\"),\n",
       " Sentence(\"\"Hopefully I can train hard this week and get selected for next week but we'll have to look at the video and wait and see.\"),\n",
       " Sentence(\"\"We were playing on our own 22 for a lot of the first half so it was quite difficult.\"),\n",
       " Sentence(\"I thought we defended reasonably well but we've just got to pick it up for France.\"\"),\n",
       " Sentence(\"His Newcastle team-mate Noon hardly covered himself in glory in his first major Test.\"),\n",
       " Sentence(\"He missed a tackle on Michael Owen in the build-up to Wales' try, conceded a penalty at the breakdown, was turned over in another tackle and fumbled Gavin Henson's cross-kick into touch, all inside the first quarter.\"),\n",
       " Sentence(\"His contribution improved in the second half, but England clearly need more of a playmaker in the inside centre role.\"),\n",
       " Sentence(\"Up front, the line-out remains fallible, despite a superb performance from Chris Jones, whose athleticism came to the fore after stepping into the side for Moody.\"),\n",
       " Sentence(\"It is more likely the Leicester flanker will return on the open side for the more physical challenge posed by the French forwards, with Andy Hazell likely to make way.\"),\n",
       " Sentence(\"Lock Ben Kay also justified his recall with an impressive all-round display on his return to the side, but elsewhere England positives were thin on the ground.\")]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsed_body.sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To perform stemming, we instantiate the SnowballStemmer from the nltk library, call its .stem() method on each token and display tokens that were modified as a result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:39.684948Z",
     "start_time": "2018-11-26T02:54:39.645914Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Andy', 'andi'),\n",
       " ('faces', 'face'),\n",
       " ('tenure', 'tenur'),\n",
       " ('tries', 'tri'),\n",
       " ('winning', 'win'),\n",
       " ('ways', 'way'),\n",
       " ('Nations', 'nation'),\n",
       " ('Wales', 'wale'),\n",
       " ('likely', 'like'),\n",
       " ('changes', 'chang'),\n",
       " ('centre', 'centr'),\n",
       " ('contemplates', 'contempl'),\n",
       " ('France', 'franc'),\n",
       " ('Lewis', 'lewi'),\n",
       " ('Moody', 'moodi'),\n",
       " ('Corry', 'corri'),\n",
       " ('missing', 'miss'),\n",
       " ('hamstring', 'hamstr'),\n",
       " ('problems', 'problem'),\n",
       " ('pairing', 'pair'),\n",
       " ('Jamie', 'jami'),\n",
       " ('Olly', 'olli'),\n",
       " ('immediately', 'immedi'),\n",
       " ('allowed', 'allow'),\n",
       " ('generate', 'generat'),\n",
       " ('position', 'posit'),\n",
       " ('kicking', 'kick'),\n",
       " ('replacing', 'replac'),\n",
       " ('debutant', 'debut'),\n",
       " ('before', 'befor'),\n",
       " ('fly-half-cum-centre', 'fly-half-cum-centr'),\n",
       " ('likely', 'like'),\n",
       " ('France', 'franc'),\n",
       " ('dropping', 'drop'),\n",
       " ('little', 'littl'),\n",
       " ('opportunity', 'opportun'),\n",
       " ('received', 'receiv'),\n",
       " ('praise', 'prais'),\n",
       " ('afterwards', 'afterward'),\n",
       " ('admitted', 'admit'),\n",
       " ('unforgiving', 'unforgiv'),\n",
       " ('teenage', 'teenag'),\n",
       " ('prodigy', 'prodigi'),\n",
       " ('tricky', 'tricki'),\n",
       " ('decision', 'decis'),\n",
       " ('firing', 'fire'),\n",
       " ('regards', 'regard'),\n",
       " ('future', 'futur'),\n",
       " ('outwardly', 'outward'),\n",
       " ('appeared', 'appear'),\n",
       " ('unaffected', 'unaffect'),\n",
       " ('punishing', 'punish'),\n",
       " ('dished', 'dish'),\n",
       " ('definitely', 'definit'),\n",
       " ('Hopefully', 'hope'),\n",
       " ('selected', 'select'),\n",
       " (\"'ll\", 'll'),\n",
       " ('playing', 'play'),\n",
       " ('quite', 'quit'),\n",
       " ('defended', 'defend'),\n",
       " ('reasonably', 'reason'),\n",
       " (\"'ve\", 've'),\n",
       " ('France', 'franc'),\n",
       " ('Newcastle', 'newcastl'),\n",
       " ('team-mate', 'team-mat'),\n",
       " ('hardly', 'hard'),\n",
       " ('covered', 'cover'),\n",
       " ('glory', 'glori'),\n",
       " ('missed', 'miss'),\n",
       " ('tackle', 'tackl'),\n",
       " ('Wales', 'wale'),\n",
       " ('try', 'tri'),\n",
       " ('conceded', 'conced'),\n",
       " ('penalty', 'penalti'),\n",
       " ('turned', 'turn'),\n",
       " ('another', 'anoth'),\n",
       " ('tackle', 'tackl'),\n",
       " ('fumbled', 'fumbl'),\n",
       " ('inside', 'insid'),\n",
       " ('contribution', 'contribut'),\n",
       " ('improved', 'improv'),\n",
       " ('clearly', 'clear'),\n",
       " ('playmaker', 'playmak'),\n",
       " ('inside', 'insid'),\n",
       " ('centre', 'centr'),\n",
       " ('remains', 'remain'),\n",
       " ('fallible', 'fallibl'),\n",
       " ('despite', 'despit'),\n",
       " ('performance', 'perform'),\n",
       " ('Jones', 'jone'),\n",
       " ('athleticism', 'athletic'),\n",
       " ('stepping', 'step'),\n",
       " ('Moody', 'moodi'),\n",
       " ('likely', 'like'),\n",
       " ('Leicester', 'leicest'),\n",
       " ('physical', 'physic'),\n",
       " ('challenge', 'challeng'),\n",
       " ('posed', 'pose'),\n",
       " ('forwards', 'forward'),\n",
       " ('Andy', 'andi'),\n",
       " ('Hazell', 'hazel'),\n",
       " ('likely', 'like'),\n",
       " ('justified', 'justifi'),\n",
       " ('recall', 'recal'),\n",
       " ('impressive', 'impress'),\n",
       " ('elsewhere', 'elsewher'),\n",
       " ('positives', 'posit')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize stemmer.\n",
    "stemmer = SnowballStemmer('english')\n",
    "\n",
    "# Stem each word.\n",
    "[(word, stemmer.stem(word)) for i, word in enumerate(parsed_body.words) \n",
    " if word.lower() != stemmer.stem(parsed_body.words[i])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:47.978888Z",
     "start_time": "2018-11-26T02:54:46.897896Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('faces', 'face'),\n",
       " ('as', 'a'),\n",
       " ('tries', 'try'),\n",
       " ('ways', 'way'),\n",
       " ('changes', 'change'),\n",
       " ('as', 'a'),\n",
       " ('problems', 'problem'),\n",
       " ('was', 'wa'),\n",
       " ('has', 'ha'),\n",
       " ('regards', 'regard'),\n",
       " ('as', 'a'),\n",
       " ('was', 'wa'),\n",
       " ('was', 'wa'),\n",
       " ('forwards', 'forward'),\n",
       " ('positives', 'positive')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(word, word.lemmatize()) for i, word in enumerate(parsed_body.words) \n",
    " if word != parsed_body.words[i].lemmatize()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lemmatization relies on parts-of-speech (POS) tagging; `spaCy` performs POS tagging, here we make assumptions, e.g. that each token is verb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:54:47.987180Z",
     "start_time": "2018-11-26T02:54:47.980209Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('faces', 'face'),\n",
       " ('tries', 'try'),\n",
       " ('winning', 'win'),\n",
       " ('is', 'be'),\n",
       " ('changes', 'change'),\n",
       " ('contemplates', 'contemplate'),\n",
       " ('missing', 'miss'),\n",
       " ('pairing', 'pair'),\n",
       " ('is', 'be'),\n",
       " ('allowed', 'allow'),\n",
       " ('kicking', 'kick'),\n",
       " ('replacing', 'replace'),\n",
       " ('is', 'be'),\n",
       " ('dropping', 'drop'),\n",
       " ('given', 'give'),\n",
       " ('received', 'receive'),\n",
       " ('admitted', 'admit'),\n",
       " ('was', 'be'),\n",
       " ('has', 'have'),\n",
       " ('firing', 'fire'),\n",
       " ('outing', 'out'),\n",
       " ('regards', 'regard'),\n",
       " ('appeared', 'appear'),\n",
       " ('punishing', 'punish'),\n",
       " ('dished', 'dish'),\n",
       " ('said', 'say'),\n",
       " ('selected', 'select'),\n",
       " ('were', 'be'),\n",
       " ('playing', 'play'),\n",
       " ('was', 'be'),\n",
       " ('thought', 'think'),\n",
       " ('defended', 'defend'),\n",
       " ('got', 'get'),\n",
       " ('covered', 'cover'),\n",
       " ('missed', 'miss'),\n",
       " ('conceded', 'concede'),\n",
       " ('was', 'be'),\n",
       " ('turned', 'turn'),\n",
       " ('fumbled', 'fumble'),\n",
       " ('improved', 'improve'),\n",
       " ('remains', 'remain'),\n",
       " ('came', 'come'),\n",
       " ('stepping', 'step'),\n",
       " ('is', 'be'),\n",
       " ('posed', 'pose'),\n",
       " ('forwards', 'forward'),\n",
       " ('justified', 'justify'),\n",
       " ('were', 'be'),\n",
       " ('ground', 'grind')]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(word, word.lemmatize(pos='v')) for i, word in enumerate(parsed_body.words) \n",
    " if word != parsed_body.words[i].lemmatize(pos='v')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment & Polarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextBlob provides polarity and subjectivity estimates for parsed documents using dictionaries provided by the Pattern library. These dictionaries lexicon map adjectives frequently found in product reviews to sentiment polarity scores, ranging from -1 to +1 (negative ↔ positive) and a similar subjectivity score (objective ↔ subjective).\n",
    "\n",
    "The .sentiment attribute provides the average for each over the relevant tokens, whereas the .sentiment_assessments attribute lists the underlying values for each token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:57:50.363858Z",
     "start_time": "2018-11-26T02:57:50.359319Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sentiment(polarity=0.088031914893617, subjectivity=0.46456433637284694)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsed_body.sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-26T02:57:25.305753Z",
     "start_time": "2018-11-26T02:57:25.301670Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sentiment(polarity=0.088031914893617, subjectivity=0.46456433637284694, assessments=[(['first'], 0.25, 0.3333333333333333, None), (['major'], 0.0625, 0.5, None), (['tries'], -0.1, 0.4, None), (['back'], 0.0, 0.0, None), (['winning'], 0.5, 0.75, None), (['likely'], 0.0, 1.0, None), (['back'], 0.0, 0.0, None), (['missing'], -0.2, 0.05, None), (['game'], -0.4, 0.4, None), (['better'], 0.5, 0.5, None), (['game'], -0.4, 0.4, None), (['likely'], 0.0, 1.0, None), (['little'], -0.1875, 0.5, None), (['teenage'], 0.0, 0.0, None), (['central'], 0.0, 0.25, None), (['future'], 0.0, 0.125, None), (['least'], -0.3, 0.4, None), (['unaffected'], -0.05, 0.1, None), (['particular'], 0.16666666666666666, 0.3333333333333333, None), (['more'], 0.5, 0.5, None), (['definitely'], 0.0, 0.5, None), (['hard'], -0.2916666666666667, 0.5416666666666666, None), (['next'], 0.0, 0.0, None), (['own'], 0.6, 1.0, None), (['first'], 0.25, 0.3333333333333333, None), (['half'], -0.16666666666666666, 0.16666666666666666, None), (['difficult'], -0.5, 1.0, None), (['reasonably'], 0.2, 0.6, None), (['hardly'], -0.2916666666666667, 0.5416666666666666, None), (['first'], 0.25, 0.3333333333333333, None), (['major'], 0.0625, 0.5, None), (['first'], 0.25, 0.3333333333333333, None), (['second'], 0.0, 0.0, None), (['half'], -0.16666666666666666, 0.16666666666666666, None), (['clearly'], 0.10000000000000002, 0.3833333333333333, None), (['more'], 0.5, 0.5, None), (['superb'], 1.0, 1.0, None), (['more'], 0.5, 0.5, None), (['likely'], 0.0, 1.0, None), (['open'], 0.0, 0.5, None), (['more'], 0.5, 0.5, None), (['physical'], 0.0, 0.14285714285714285, None), (['french'], 0.0, 0.0, None), (['likely'], 0.0, 1.0, None), (['justified'], 0.4, 0.9, None), (['impressive'], 1.0, 1.0, None), (['thin'], -0.4, 0.8500000000000001, None)])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parsed_body.sentiment_assessments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine Textblob Lemmatization with `CountVectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:26:04.152859Z",
     "start_time": "2018-11-22T18:26:04.150466Z"
    }
   },
   "outputs": [],
   "source": [
    "def lemmatizer(text):\n",
    "    words = TextBlob(text.lower()).words\n",
    "    return [word.lemmatize() for word in words]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-22T18:26:04.160781Z",
     "start_time": "2018-11-22T18:26:04.154322Z"
    }
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(analyzer=lemmatizer, decode_error='replace')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}