python-intermediate-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/CC_BY.png\"><br />\n",
    "\n",
    "Created by [Nathan Kelber](http://nkelber.com) and Ted Lawless for [JSTOR Labs](https://labs.jstor.org/) under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/)<br />\n",
    "For questions/comments/improvements, email nathan.kelber@ithaka.org.<br />\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Intermediate 2\n",
    "\n",
    "**Description:** This notebook describes how to:\n",
    "* Read and write files (.txt, .csv, .json)\n",
    "* Use the `constellate` client to read in metadata\n",
    "* Use the `constellate` client to read in data\n",
    "\n",
    "This notebook describes how to read and write text, CSV, and JSON files using Python. Additionally, it explains how the `constellate` client can help users easily load and analyze their datasets.\n",
    "\n",
    "**Use Case:** For Learners (Detailed explanation, not ideal for researchers)\n",
    "\n",
    "**Difficulty:** Intermediate\n",
    "\n",
    "**Completion Time:** 90 minutes\n",
    "\n",
    "**Knowledge Required:** \n",
    "* Python Basics ([Start Python Basics I](./python-basics-1.ipynb))\n",
    "\n",
    "**Knowledge Recommended:**\n",
    "* [Python Intermediate 1](./python-intermediate-1.ipynb)\n",
    "\n",
    "**Data Format:** Text (.txt), CSV (.csv), JSON (.json)\n",
    "\n",
    "**Libraries Used:**\n",
    "* [pandas](https://constellate.org/docs/key-terms/#pandas) to read and write CSV files\n",
    "* `json` to read and write JSON files\n",
    "* [constellate](https://constellate.org/docs/key-terms/#constellate-client) client to retrieve and read data\n",
    "\n",
    "**Research Pipeline:** None\n",
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Download Sample Files for this Lesson\n",
    "import urllib.request\n",
    "download_urls = [\n",
    "    'https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/sample.csv',\n",
    "    'https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/sample.txt',\n",
    "    'https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/adaptation.txt'\n",
    "]\n",
    "\n",
    "for url in download_urls:\n",
    "    urllib.request.urlretrieve(url, './data/' + url.rsplit('/', 1)[-1])\n",
    "    \n",
    "print('Sample files ready.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Files in Python\n",
    "Working with files is an essential part of Python programming. When we execute code in Python, we manipulate data through the use of variables. When the program is closed, however, any data stored in those variables is erased. To save the information stored in variables, we must learn how to write it to a file.\n",
    "\n",
    "At the same time, we may have notebooks for applying specific analyses, but we need to have a way to bring data into the notebook for analysis. Otherwise, we would have to type all the data into the program ourselves! Both reading-in from files and writing-out data to files are important skills for data science and the digital humanities.\n",
    "\n",
    "This section describes how to work with three kinds of common data files in Python:\n",
    "* Plain Text Files (.txt)\n",
    "* Comma-Separated Value files (.csv)\n",
    "* Javascript Object Notation files (.json)\n",
    "\n",
    "Each of these filetypes are in wide use in data science, digital humanities, and general programming. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Three Common Data File Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plain Text Files (.txt)\n",
    "A plain text file is one of the simplest kinds of computer files. Plain text files can be opened with a text editor like Notepad (Windows 10) or TextEdit (OS X). The file can contain only basic textual characters such as: letters, numbers, spaces, and line breaks. Plain text files do not contain styling such as: heading sizes, italic, bold, or specialized fonts. (To including styling in a text file, writers may use other file formats such as rich text format (.rtf) or markdown (.md).)\n",
    "\n",
    "Plain text files (.txt) can be easily viewed and modified by humans by changing the text within. This is an important distinction from binary files such as images (.jpg), archives (.gzip), audio (.wav), or video (.mp4). If a binary file is opened with a text editor, the content will be largely unreadable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comma-Separated Value Files (.csv)\n",
    "A comma-separated value file is also a text file that can easily be modifed with a text editor. A CSV file is generally used to store data that fits in a series or table (like a list or spreadsheet). A spreadsheet application (like Microsoft Excel or Google Sheets) will allow you to view and edit a CSV data in the form of a table.\n",
    "\n",
    "Each row of a CSV file represents a single row of a table. The values in a CSV are separated by commas (with no space between), but other delimiters can be chosen such as a tab or pipe (|). A tab-separated value file is called a TSV file (.tsv). Using tabs or pipes may be preferable if the data being stored contains commas (since this could make it confusing whether a comma is part of a single entry or a delimiter between entries).\n",
    "\n",
    "#### The text contents of a sample CSV file\n",
    "```\n",
    "Username,Login email,Identifier,First name,Last name\n",
    "booker12,rachel@example.com,9012,Rachel,Booker\n",
    "grey07,,2070,Laura,Grey\n",
    "johnson81,,4081,Craig,Johnson\n",
    "jenkins46,mary@example.com,9346,Mary,Jenkins\n",
    "smith79,jamie@example.com,5079,Jamie,Smith\n",
    "```\n",
    "#### The same CSV file represented in Google Sheets:\n",
    "\n",
    "![CSV table view in Google Sheets](https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/csv_in_sheets.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JavaScript Object Notation (.json)\n",
    "A Javascript Object Notation file is also a text file that can be modified with a text editor. A JSON file stores data in key/value pairs, very similar to a Python dictionary. One of the key benefits of JSON is its compactness which makes it ideal for exchanging data between web browsers and servers.\n",
    "\n",
    "While smaller JSON files can be opened in a text editor, larger files can be difficult to read. Viewing and editing JSON is easier in specialized editors, available online at sites like: \n",
    "\n",
    "* [JSON Formatter](http://jsonformatter.org)\n",
    "* [JSON Editor Online](https://jsoneditoronline.org/)\n",
    "\n",
    "A JSON file has a nested structure, where smaller concepts are grouped under larger ones. Like extensible markup language (.xml), a JSON file can be checked to determine that it is valid (follows the proper format for JSON) and/or well-formed (follows the proper format defined in a specialized example, called a schema). \n",
    "\n",
    "#### The text contents of a sample JSON file\n",
    "\n",
    "```\n",
    "{\n",
    "    \"firstName\": \"Julia\",\n",
    "    \"lastName\": \"Smith\",\n",
    "    \"gender\": \"woman\",\n",
    "    \"age\": 57,\n",
    "    \"address\": {\n",
    "        \"streetAddress\": \"11434\",\n",
    "        \"city\": \"Detroit\",\n",
    "        \"state\": \"Mi\",\n",
    "        \"postalCode\": \"48202\"\n",
    "    },\n",
    "    \"phoneNumbers\": [\n",
    "        { \"type\": \"home\", \"number\": \"7383627627\" }\n",
    "    ]\n",
    "}\n",
    "```\n",
    "#### The same JSON file represented in JSON Editor Online\n",
    "![An image of the JSON file showing the structure](https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/json_editor.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening, Reading, and Writing Text Files (.txt) \n",
    "\n",
    "Before we can read or write to a text file, we must open the file. Normally, when we open a file, a new window appears where we can see the file contents. In Python, opening a file means creating a *file object* that references the particular file. When the file has been opened, we can read or write to the file using the file object. Finally, we must close the file. Here are the three steps:\n",
    "\n",
    "1. Open the file using `with open()` using read `r`, write `w`, read+write `r+` or append `a` mode\n",
    "2. Use the `.read()` or `.write()` method on the file object\n",
    "3. Let the code block close which automatically closes the file\n",
    "\n",
    "Let's practice on `sample.txt`, a sample text file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Opening a file in read mode\n",
    "with open('data/sample.txt', 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have created a file object called `f`. The first argument (`'data/sample.txt'`) passed into the `open()` function is a string containing the file name. You can see the sample.txt is in the data directory by choosing `File > Open` in the file menu. If your file was called reports.txt, you would replace that argument with `'data/reports.txt'`. The second argument (`'r'`) determines that we are opening the file in \"read\" mode, where the file can be read but not modified. There are four main modes that can be specified when creating a file object:\n",
    "\n",
    "|Argument|Mode Name|Description|\n",
    "|---|---|---|\n",
    "|'r'|read|Reads file but no writing allowed (protects file from modification)|\n",
    "|'w'|write|Writes to file, overwriting any information in the file (saves over the current version of the file|\n",
    "|'r+'|read+write|Makes the file object available for both reading and writing.|\n",
    "|'a'|append|Appends (or adds) information to the end of the file (new information is added below old information)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading Very Large Text Files (Line by Line)\n",
    "\n",
    "If the file is very large, we may want to read it one line at a time. To read a single line at a time, we can use the use a for loop instead of the `.read()` method. This is useful when you are working with large files that approach the amount of memory in your computer. This depends on your computer, but is probably about 8 GB or larger. (In a plain text file, that would be roughly 5.5 million pages.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Opening a file in read mode\n",
    "with open('data/sample.txt', 'r') as f:\n",
    "    for line in f:\n",
    "        print(line, end='') # By default each print inserts a line break, pass `end = ''` to remove this break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4 style=\"color:red; display:inline\">Coding Challenge! &lt; / &gt; </h4>\n",
    "\n",
    "In the data folder is a poem called Adaptation by the 2022 American Poet Laureate, Ada Limón. The file is called `adaptation.txt`. Open the file and print it out line by line.\n",
    "\n",
    "For a more difficult challenge, print the title and author before the poem along with a number for each line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read and print the poem found in /data/adaptation.txt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Chunking file reading with `.read()` \n",
    "\n",
    "We can also read large files in chunks using `.read()` passing an argument that specifies the number of bytes to read in each chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading a set number of bytes with .read()\n",
    "\n",
    "with open('data/sample.txt', 'r') as f:\n",
    "    while True:\n",
    "        file_chunk = f.read(4) # Try changing the argument here\n",
    "        if len(file_chunk) > 0:\n",
    "            print(file_chunk)\n",
    "        else: \n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to and Creating Files with `.write()`\n",
    "The `.write()` method can be used with a file opened in write mode ('w'), read+write mode ('r+), or append mode ('a'). If you try to use the `.write()` method on a file opened in read mode ('r'), the write will fail and you will get a \"not writable\" error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Trying to use `.write()` on a file in read mode\n",
    "# will create an error\n",
    "\n",
    "with open('data/sample.txt', 'r') as f:\n",
    "    f.write('Add this text to the file')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to use the `.write()` method to write to a file, we demonstrate three common modes here: write ('w'), read+write ('r+') and append ('a') mode. \n",
    "\n",
    "**Choose write mode ('w') if:**\n",
    "* You want to create a new text file and/or write data to it\n",
    "* You want to overwrite all data in the file\n",
    "\n",
    "Be careful with write mode, since it will write over any existing file!\n",
    "\n",
    "**Choose read+write mode ('r+') if:**\n",
    "* You need to both read and write data to a file\n",
    "* You are okay overwriting the file if it already exists\n",
    "* You want the file pointer at the beginning of the file\n",
    "\n",
    "Again, be careful with read+write mode ('r+') because it can overwrite existing data.\n",
    "\n",
    "**Choose append mode ('a') if:**\n",
    "* You simply need to add more data to a file\n",
    "* You want to protect existing data in the file from being overwritten\n",
    "\n",
    "There are also additional modes for working with a text file:\n",
    "\n",
    "|Mode|Effect|\n",
    "|---|---|\n",
    "| w | Opens a file for writing only. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing. |\n",
    "| r+ | Opens a file for both reading and writing. The file pointer will be at the beginning of the file. |\n",
    "| a | Opens a file for appending. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing. |\n",
    "| w+ | Opens a file for both writing and reading. Overwrites the existing file if the file exists. If the file does not exist, it creates a new file for reading and writing. |\n",
    "| a+ | Opens a file for both appending and reading. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing. |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a new file using `.write()` method and write mode ('w')\n",
    "To create a new file, open a new file object in write mode. You may then use the `.write()` method to add some initial data to the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new file and write a string to the file\n",
    "\n",
    "with open('data/sample_new.txt', 'w') as f:\n",
    "    f.write('Put this string into the new file. ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Modifying an existing file with using `.write()` method and read+write mode ('r+') \n",
    "You can read and write to the same file by opening it in read+write mode ('r+'). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4 style=\"color:red; display:inline\">Coding Challenge! &lt; / &gt; </h4>\n",
    "\n",
    "Open our `sample_new.txt` file, overwrite the word 'Put', the first three characters, with 'Add'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading and writing to a file using read+write mode\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Appending to an existing file using `.write()` method and append mode ('a')\n",
    "\n",
    "Use the append mode if you want to be sure to preserve existing data in the file. Try appending some text onto the end of a file using append mode ('a')."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4 style=\"color:red; display:inline\">Coding Challenge! &lt; / &gt; </h4>\n",
    "\n",
    "Open our `sample_new.txt` file again. Add a second sentence to the file without overwriting the current file contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Appending to a file using the append mode ('a')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening, Reading, and Writing CSV Files (.csv)\n",
    "CSV file data can be easily opened, read, and written using the `pandas` library. (For large CSV files (>500 mb), you may wish to use the `csv` library to read in a single row at a time to reduce the memory footprint.) Pandas is flexible for working with tabular data, and the process for importing and exporting to CSV is simple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas \n",
    "import pandas as pd\n",
    "\n",
    "# Create our dataframe\n",
    "df = pd.read_csv('data/sample.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the dataframe\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you've made any necessary changes in Pandas, write the dataframe back to the CSV file. (Remember to always back up your data before writing over the file.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write data to new file\n",
    "# Keeping the Header but removing the index\n",
    "df.to_csv('data/new_sample.csv', header=True, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening, Reading, and Writing JSON Files (.json)\n",
    "\n",
    "JSON files use a key/value structure very similar to a Python dictionary. We will start with a Python dictionary called `py_dict` and then write the data to a JSON file using the `json` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining sample data in a Python dictionary\n",
    "py_dict = {\n",
    "    \"firstName\": \"Julia\",\n",
    "    \"lastName\": \"Smith\",\n",
    "    \"gender\": \"woman\",\n",
    "    \"age\": 57,\n",
    "    \"address\": {\n",
    "        \"streetAddress\": \"11434\",\n",
    "        \"city\": \"Detroit\",\n",
    "        \"state\": \"Mi\",\n",
    "        \"postalCode\": \"48202\"\n",
    "    },\n",
    "    \"phoneNumber\": \"3133627627\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To write our dictionary to a JSON file, we will use the `with open` technique we learned that automatically closes file objects. We also need the `json` library to help dump our dictionary data into the file object. The `json.dump` function works a little differently than the write method we saw with text files. \n",
    "\n",
    "We need to specify two arguments: \n",
    "\n",
    "* The data to be dumped\n",
    "* The file object where we are dumping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open/create sample.json in write mode\n",
    "# as the file object `f`. The data in py_dict\n",
    "# is dumped into `f` and then `f` is closed\n",
    "\n",
    "import json\n",
    "with open('data/sample.json', 'w') as f:\n",
    "    json.dump(py_dict, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read data in from a JSON file, we can use the `json.load` function on our file object. Here we load all the content into a variable called `content`. We can then print values based on particular keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Open the .json file in read mode\n",
    "# and print the loaded contents\n",
    "\n",
    "with open('data/sample.json', 'r') as f:\n",
    "    print(json.load(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the current data from the file into a dictionary\n",
    "# and add an entry to the dictionary\n",
    "\n",
    "with open('data/sample.json', 'r') as f:\n",
    "    file_contents = json.load(f)\n",
    "    file_contents['pet'] = 'dog'\n",
    "\n",
    "# Write the contents of the dictionary over the existing data \n",
    "    \n",
    "with open('data/sample.json', 'w') as f:  \n",
    "    json.dump(file_contents, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.jsonload()` method creates a Python dictionary from the .json file object. This means that we can query a particular key/value pair after loading the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the data from the json file into a dictionary\n",
    "# Check to see if there is value associated with pet\n",
    "\n",
    "with open('data/sample.json', 'r') as f:\n",
    "    json_contents = json.load(f)\n",
    "    print(json_contents.get('pet'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3 style=\"color:red; display:inline\">Coding Challenge! &lt; / &gt; </h3>\n",
    "\n",
    "Add an entry to Julia Smith that indicates her favorite ice cream is chocolate peanut butter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add an ice cream entry to Julia Smith's record\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening datasets with `constellate` client\n",
    "\n",
    "The `constellate` client helps retrieve a given dataset and/or its associated metadata. The metadata is supplied in a CSV file and the full dataset is supplied in a compressed JSON Lines file (.jsonl.gz). For analysis focused on metadata, we recommend users start with the CSV file since it is both easier and faster to view, parse, and manipulate. If you need access to n-grams or full-text, then you want the full JSON lines file.\n",
    "\n",
    "### Metadata CSV vs. JSON Lines Data File\n",
    "All of the textual data and metadata is available inside of the JSON Lines files, but we have chosen to offer the metadata CSV for two primary reasons:\n",
    "\n",
    "1. The JSON Lines data is a little more complex to parse since it is nested. It cannot be easily represented in a table form in something like Pandas. It is nice to be able to easily view all the metadata in a Pandas dataframe.\n",
    "2. The JSON Lines data can be very large. Each file contains all of the metadata *plus* unigram counts, bigram counts, trigram counts, and full-text (when available). Manipulating all that data takes significant computer time and costs. Even a modest dataset (~5000 files) can be over 1 GB in size uncompressed.\n",
    "\n",
    "More information is available, including the metadata categories, in the FAQ [\"What is the data file format?\"](https://constellate.org/docs/what-format-are-jstor-portico-datasets/). There's also reference documentation for [the `constellate` client](https://constellate.org/docs/constellate-client).\n",
    "\n",
    "### Retrieving sampled data (`constellate` client methods)\n",
    "\n",
    "By passing the `constellate` client a dataset ID (here called `dataset_id`), we can automatically download the metadata CSV file or the full JSON Lines dataset created by the [dataset builder](https://constellate.org/builder/). By default, these datasets are randomly sampled to 1500 documents. Experience has shown us that this is the right size for teaching/learning where data loads quickly and still gives interesting results.\n",
    "\n",
    "* Use the `.get_metadata()` method to retrieve the sampled metadata CSV file\n",
    "* Use the `.get_dataset()` method to retrieve the sampled JSON data file\n",
    "\n",
    "|Code|Result|\n",
    "|---|---|\n",
    "|f = constellate.get_metadata(dataset_id)|Automatically retrieves a sampled metadata CSV file and creates a file object `f`|\n",
    "|f = constellate.get_dataset(dataset_id)| Automatically retrieves a sampled, compressed JSON Lines dataset file (jsonl.gz) and creates a file object `f`|\n",
    "\n",
    "The JSON Lines file will be downloaded in a compressed gzip format (jsonl.gz). We can iterate over each document in the corpus by using the `dataset_reader()` method.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Downloading and reading a Constellate dataset\n",
    "# If using locally, pip install constellate-client\n",
    "import constellate\n",
    "\n",
    "# Setting a dataset ID based on the journal Shakespeare Quarterly.\n",
    "dataset_id = \"2880ef54-b3b5-c0af-2ef1-ad557f236b5e\"\n",
    "\n",
    "# Pull in our dataset CSV using\n",
    "# The .get_metadata() method downloads the CSV file for our metadata\n",
    "# to the /data folder and returns a string for the file name and location\n",
    "# dataset_metadata will be a string containing that file name and location\n",
    "dataset_metadata = constellate.get_metadata(dataset_id)\n",
    "\n",
    "# Pull in the sampled dataset (1500 documents) that matches `dataset_id`\n",
    "# in the form of a gzipped JSON lines file.\n",
    "# The .get_dataset() method downloads the gzipped JSONL file\n",
    "# to the /data folder and returns a string for the file name and location\n",
    "dataset_file = constellate.get_dataset(dataset_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieving the full dataset with `constellate` client\n",
    "\n",
    "Researchers wishing to work with larger datasets (up to 25,000 items), first need to request these datasets be built.\n",
    "\n",
    "![From the dataset dashboard, select \"download\" then \"More Download Options\"](https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/request-dataset.png)\n",
    "\n",
    "Then you can request the full dataset using the `constellate.download` method. For more information, see [the Constellate Client reference documentation](https://constellate.org/docs/constellate-client).\n",
    "\n",
    "|Code|Result|\n",
    "|---|---|\n",
    "|f = constellate.download(dataset_id, 'metadata')|Automatically retrieves a full metadata CSV file and creates a file object `f`|\n",
    "|f = constellate.download(dataset_id, 'jsonl')| Automatically retrieves the full compressed JSON Lines dataset file (jsonl.gz) and creates a file object `f`|\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To download the full dataset metadata (up to a limit of 25,000 documents),\n",
    "# request it first in the builder environment. See the Constellate Client\n",
    "# documentation at: https://constellate.org/docs/constellate-client\n",
    "# Then use the `constellate.download` method show below.\n",
    "dataset_metadata = constellate.download(dataset_id, 'metadata')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To download the full dataset (up to a limit of 25,000 documents),\n",
    "# request it first in the builder environment. See the Constellate Client\n",
    "# documentation at: https://constellate.org/docs/constellate-client\n",
    "# Then use the `constellate.download` method show below.\n",
    "dataset_file = constellate.download(dataset_id, 'jsonl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to work with a CSV file, pass the file object into Pandas `read_csv` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pandas dataframe from a metadata CSV file\n",
    "import pandas as pd\n",
    "\n",
    "# Create our dataframe\n",
    "df = pd.read_csv(dataset_metadata)\n",
    "\n",
    "# Preview our dataframe\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading in data from a dataset object\n",
    "\n",
    "When working with a Constellate JSON file, it is best to iterate over each line (representing all the data for a given document) using the dataset_reader method and a `for` loop.\n",
    "\n",
    "The `dataset_reader()` method will read in data from the compressed JSON dataset file object. By keeping the data in a compressed format and reading in a single line at a time, we reduce the processing time and memory use. These can be substantial for large datasets.\n",
    "\n",
    "The `dataset_reader()` essentially unzips each row/document and loads it into a dictionary. Each dictionary will contain all the metadata and data available for the current document. \n",
    "\n",
    "In practice, that looks like:\n",
    "\n",
    "|Code|Result|\n",
    "|---|---|\n",
    "|for document in constellate.dataset_reader(dataset_file):| Load all the data for a document into a dictionary, one by one|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loop over every document in the dataset\n",
    "# and print the title\n",
    "\n",
    "for document in constellate.dataset_reader(dataset_file):\n",
    "    print(document['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Lesson Complete\n",
    "\n",
    "Congratulations! You have completed *Python Intermediate 2*.\n",
    "\n",
    "### Start Next Lesson: [Python Intermediate 3](./python-intermediate-3.ipynb)\n",
    "\n",
    "### Exercise Solutions\n",
    "Here are a few solutions for exercises in this lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read and print the poem found in /data/adaptation.txt\n",
    "\n",
    "# Create a variable to store the line\n",
    "line_number = 1\n",
    "\n",
    "print('ADAPTATION')\n",
    "print('Ada Limón\\n')\n",
    "\n",
    "with open('data/adaptation.txt', 'r') as f:\n",
    "    for line in f:\n",
    "        print(line_number, line, end='')\n",
    "        line_number += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading and writing to a file using read+write mode ('r+')\n",
    "\n",
    "with open('data/sample_new.txt', 'r+') as f:\n",
    "    f.write('Add')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Appending to a file using the append mode ('a')\n",
    "with open('data/sample_new.txt', 'a') as f:\n",
    "    f.write('A new sentence is added now.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the current data from the file into a dictionary\n",
    "# and add an entry to the dictionary\n",
    "\n",
    "with open('data/sample.json', 'r') as f:\n",
    "    file_contents = json.load(f)\n",
    "    file_contents['favoriteIceCream'] = 'chocolate peanut butter'\n",
    "\n",
    "# Write the contents of the dictionary over the existing data \n",
    "    \n",
    "with open('data/sample.json', 'w') as f:  \n",
    "    json.dump(file_contents, f)\n",
    "    \n",
    "with open('data/sample.json', 'r') as f:\n",
    "    json_contents = json.load(f)\n",
    "    print(json_contents.get('favoriteIceCream'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "312px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}