api-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Data APIs\n",
    "\n",
    "**Sam Maurer // maurer@berkeley.edu // Oct. 9, 2017**\n",
    "\n",
    "This notebook provides a demonstration of data-access APIs that operate over the web. See README.md for setup instructions.\n",
    "\n",
    "In Part 1, we'll load and parse results from an API feed of earthquake data.  \n",
    "In Part 2, we'll add query parameters to the workflow, using the Google Maps Geolocation API as an example.  \n",
    "In Part 3, we'll use an authenticated API to query public Twitter posts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Reading from an automated data feed\n",
    "\n",
    "### USGS real-time earthquake feeds\n",
    "\n",
    "This is an API for near-real-time data about earthquakes. Data is provided in JSON format over the web. No authentication is needed, and there's no way to customize the output. Instead, the API has a separate endpoint for each permutation of the data that users might want.\n",
    "\n",
    "**API documentation:**  \n",
    "http://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php\n",
    "\n",
    "**Sample API endpoint, for magnitude 4.5+ earthquakes in past day:**  \n",
    "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_day.geojson  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "import json      # library for working with JSON-formatted text strings\n",
    "import requests  # library for accessing content from web URLs\n",
    "\n",
    "import pprint    # library for cleanly printing Python data structures\n",
    "pp = pprint.PrettyPrinter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# download data on magnitude 2.5+ quakes from the past week\n",
    "\n",
    "endpoint_url = \"http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojson\"\n",
    "response = requests.get(endpoint_url)\n",
    "results = response.text\n",
    "\n",
    "# what is the data type of the results?\n",
    "\n",
    "print(type(results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print the first 500 characters to see a sample of the data\n",
    "\n",
    "print(results[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# it looks like the results are a string with JSON-formatted data inside\n",
    "\n",
    "# parse the string into a Python dictionary (loads = \"load string\")\n",
    "data = json.loads(results)\n",
    "\n",
    "print(type(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print the dictionary\n",
    "\n",
    "pp.pprint(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# subsets of the data can be accessed with square brackets, using labels for the \n",
    "# named elements and numerals for the lists\n",
    "\n",
    "# save the list of quakes to a new variable\n",
    "\n",
    "quakes = data['features']\n",
    "\n",
    "# print the most recent quake\n",
    "\n",
    "pp.pprint(quakes[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# pull out the title from each earthquake listing\n",
    "\n",
    "for q in quakes:\n",
    "    print(q['properties']['title'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# pull out magnitudes and depths into a Pandas dataframe, using\n",
    "# a more compact Python syntax for iterating through lists\n",
    "\n",
    "d = {'magnitude': [q['properties']['mag'] for q in quakes],\n",
    "     'depth': [q['geometry']['coordinates'][2] for q in quakes]}\n",
    "\n",
    "df = pd.DataFrame.from_dict(d)\n",
    "\n",
    "# how many earthquakes were loaded into the dataframe?\n",
    "\n",
    "print(len(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print the first few lines of data\n",
    "\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print some descriptive statistics\n",
    "\n",
    "print(df.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# plot the depth vs. magnitude\n",
    "\n",
    "df.plot(x='magnitude', y='depth', kind='scatter')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# save the dataframe to disk\n",
    "\n",
    "df.to_csv('usgs_earthquake_data.csv')\n",
    "\n",
    "print('file saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# read it back later\n",
    "\n",
    "new_df = pd.DataFrame.from_csv('usgs_earthquake_data.csv')\n",
    "\n",
    "print(new_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Querying an API endpoint\n",
    "\n",
    "### Google Maps Geocoding API\n",
    "\n",
    "Google has various APIs that let you access its services through code instead of through GUI apps. This one from Google Maps lets you look up the latitude-longitude coordinates of street addresses.\n",
    "\n",
    "It works similarly to the earthquakes example, but with query parameters added to the URL endpoint!\n",
    "\n",
    "**API documentation:**  \n",
    "https://developers.google.com/maps/documentation/geocoding/intro\n",
    "\n",
    "**API endpoint:**  \n",
    "https://maps.googleapis.com/maps/api/geocode/json\n",
    "\n",
    "**API endpoint with query parameters:**  \n",
    "https://maps.googleapis.com/maps/api/geocode/json?address=Wurster+Hall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json      # library for working with JSON-formatted text strings\n",
    "import requests  # library for accessing content from web URLs\n",
    "\n",
    "import pprint    # library for cleanly printing Python data structures\n",
    "pp = pprint.PrettyPrinter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# we have to encode the search query so that it can be passed as a URL, \n",
    "# with spaces and other special characters removed\n",
    "\n",
    "endpoint = 'https://maps.googleapis.com/maps/api/geocode/json'\n",
    "\n",
    "params = {'address': 'Wurster Hall, Berkeley, CA'}\n",
    "\n",
    "url = requests.Request('GET', endpoint, params=params).prepare().url\n",
    "print(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# download and parse the results\n",
    "\n",
    "response = requests.get(url)\n",
    "results = response.text\n",
    "data = json.loads(results)\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# print it more nicely\n",
    "\n",
    "pp.pprint(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# pull out the lat-lon coordinates\n",
    "\n",
    "for r in data['results']:\n",
    "    coords = r['geometry']['location']\n",
    "    print(coords['lat'], coords['lng'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1. Search for some other addresses or landmarks!\n",
    "2. Take a look at the [API documentation](https://developers.google.com/maps/documentation/geocoding/intro). What are the usage limits?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Querying an API with authentication\n",
    "\n",
    "### Twitter search APIs\n",
    "\n",
    "Twitter's APIs operate over the web as well, but they require a back-and-forth authentication process at the beginning of each connection. It's easier to have a Python library handle this than to create the query URLs ourselves.\n",
    "\n",
    "The Twitter \"REST\" APIs perform stand-alone operations: you submit a query and receive results, like in earlier examples. ([REST](https://en.wikipedia.org/wiki/Representational_state_transfer) is a particular set of guidelines that many APIs follow.) Twitter also has a \"streaming\" API that continues sending results in real time until you disconnect.\n",
    "\n",
    "**API documentation:**  \n",
    "https://dev.twitter.com/rest/public  \n",
    "https://dev.twitter.com/overview/api/tweets\n",
    "\n",
    "**Documentation for the Python helper library**:  \n",
    "http://geduldig.github.io/TwitterAPI/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from TwitterAPI import TwitterAPI\n",
    "\n",
    "import pprint\n",
    "pp = pprint.PrettyPrinter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# import API credentials from keys.py file in the\n",
    "# same directory as this notebook\n",
    "\n",
    "from keys import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set up an API connection using credentials from the keys file\n",
    "\n",
    "api = TwitterAPI(consumer_key, consumer_secret, \n",
    "                 access_token, access_token_secret)\n",
    "\n",
    "print(\"Connection is set up but not tested\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making a simple data request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# most recent tweet from CED\n",
    "\n",
    "endpoint = 'statuses/user_timeline'\n",
    "params = {\n",
    "    'screen_name': 'CEDNews', \n",
    "    'count': 1\n",
    "}\n",
    "r = api.request(endpoint, params)\n",
    "\n",
    "for tweet in r.get_iterator():\n",
    "    print(tweet['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# what other data is there?\n",
    "\n",
    "pp.pprint(tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other API endpoints allow different types of searches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# search for recent tweets with your favorite emoji\n",
    "\n",
    "endpoint = 'search/tweets'\n",
    "params = {\n",
    "    'q': '👻🎃', \n",
    "    'count': 5\n",
    "}\n",
    "r = api.request(endpoint, params)\n",
    "\n",
    "for tweet in r.get_iterator():\n",
    "    print(tweet['text'] + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# search for public tweets in Korean\n",
    "\n",
    "endpoint = 'search/tweets'\n",
    "params = {\n",
    "    'q': '*', \n",
    "    'lang': 'ko', \n",
    "    'count': 5\n",
    "} \n",
    "r = api.request(endpoint, params)\n",
    "\n",
    "for tweet in r.get_iterator():\n",
    "    print(tweet['text'] + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# search for public tweets geotagged near the UC Berkeley campus\n",
    "\n",
    "endpoint = 'search/tweets'\n",
    "params = {\n",
    "    'q': '*', \n",
    "    'geocode': '37.873,-122.260,0.5km', \n",
    "    'count': 5\n",
    "} \n",
    "r = api.request(endpoint, params)\n",
    "\n",
    "for tweet in r.get_iterator():\n",
    "    print(tweet['text'] + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "1. Try some different search queries!\n",
    "2. Display some more data fields in addition to the tweet text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus: Streaming live tweets in real time "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Twitter allows only one or two simultaneous streaming connections for \n",
    "# each set of API credentials, so this part may not work during class\n",
    "\n",
    "endpoint = 'statuses/filter'\n",
    "params = {'locations': '-180,-90,180,90'}\n",
    "r = api.request(endpoint, params)\n",
    "LIMIT = 20\n",
    "\n",
    "# 'enumerate' lets us count tweets as we receive them\n",
    "\n",
    "for i, tweet in enumerate(r.get_iterator()):\n",
    "    print(tweet['created_at'])\n",
    "    print(tweet['place']['full_name'] + ', ' + tweet['place']['country'])\n",
    "    print(tweet['text'] + '\\n')\n",
    "    if (i > LIMIT): break\n",
    "\n",
    "# close the streaming connection\n",
    "r.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises for the remainder of class\n",
    "\n",
    "Choose your favorite:\n",
    "\n",
    "1. Make a scatter plot of the lat-lon coordinates of earthquakes.  \n",
    "   &nbsp;\n",
    "   \n",
    "2. The earthquakes API is actually returning a specific data format called [GeoJSON](https://en.wikipedia.org/wiki/GeoJSON), also used by many other geospatial data feeds. Try saving the raw GeoJSON file and opening it in QGIS or in the [geojson.io](http://geojson.io) web viewer.  \n",
    "   &nbsp;\n",
    "\n",
    "2. Using the geocoding example as a starting point, try searching Google's Directions API or Elevation API instead. You can read more about them on the [Google Maps web services](https://developers.google.com/maps/web-services/) page.  \n",
    "   &nbsp;\n",
    "\n",
    "3. Try out another API that you're interested in. Can you figure out how to connect to it using Python?  \n",
    "\n",
    "   With municipal data it's often easiest to just download a data file, but APIs are great for querying big data sets or tracking live updates. Here are some resources.\n",
    "\n",
    "   - San Francisco:  https://data.sfgov.org/developers  \n",
    "   - Alameda County:  https://data.acgov.org/developers  \n",
    "   - UC Berkeley:  https://api-central.berkeley.edu  \n",
    "   - US Census:  http://www.census.gov/data/developers/data-sets.html  \n",
    "   - Open Data Network:  https://www.opendatanetwork.com  \n",
    "   - CivicData:  http://www.civicdata.io/  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}