GPT_Allama.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "GPT - Allama",
      "provenance": [],
      "collapsed_sections": [
        "pel-uBULXO2L"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hk21042/GPT/blob/master/GPT_Allama.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7LoMj4GA4n_",
        "colab_type": "text"
      },
      "source": [
        "#  Train a GPT-2 Text-Generating Model w/ GPU For Free \n",
        "\n",
        "by [Max Woolf](http://minimaxir.com)\n",
        "\n",
        "*Last updated: August 28th, 2019*\n",
        "\n",
        "Retrain an advanced text generating neural network on any text dataset **for free on a GPU using Collaboratory** using `gpt-2-simple`!\n",
        "\n",
        "For more about `gpt-2-simple`, you can visit [this GitHub repository](https://github.com/minimaxir/gpt-2-simple).\n",
        "\n",
        "\n",
        "To get started:\n",
        "\n",
        "1. Copy this notebook to your Google Drive to keep it and save your changes. (File -> Save a Copy in Drive)\n",
        "2. Make sure you're running the notebook in Google Chrome.\n",
        "3. Run the cells below:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KBkpRgBCBS2_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 233
        },
        "outputId": "77ee2c13-f2b6-4f30-bbb2-6b353df5cd05"
      },
      "source": [
        "!pip install -q gpt-2-simple\n",
        "import gpt_2_simple as gpt2\n",
        "from datetime import datetime\n",
        "from google.colab import files"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[?25l\r\u001b[K     |▌                               | 10kB 28.5MB/s eta 0:00:01\r\u001b[K     |█                               | 20kB 31.3MB/s eta 0:00:01\r\u001b[K     |█▌                              | 30kB 38.8MB/s eta 0:00:01\r\u001b[K     |██                              | 40kB 3.5MB/s eta 0:00:01\r\u001b[K     |██▌                             | 51kB 4.3MB/s eta 0:00:01\r\u001b[K     |███                             | 61kB 5.1MB/s eta 0:00:01\r\u001b[K     |███▋                            | 71kB 5.8MB/s eta 0:00:01\r\u001b[K     |████                            | 81kB 6.6MB/s eta 0:00:01\r\u001b[K     |████▋                           | 92kB 7.3MB/s eta 0:00:01\r\u001b[K     |█████                           | 102kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████▋                          | 112kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████                          | 122kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████▋                         | 133kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████▏                        | 143kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████▋                        | 153kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████▏                       | 163kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████▋                       | 174kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████▏                      | 184kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████▊                      | 194kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████▏                     | 204kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████▊                     | 215kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████▏                    | 225kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████▊                    | 235kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 245kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████▊                   | 256kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████▎                  | 266kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████▊                  | 276kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████▎                 | 286kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████▊                 | 296kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████▎                | 307kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████▉                | 317kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████▎               | 327kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████▉               | 337kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████████▎              | 348kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████████▉              | 358kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████▎             | 368kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████▉             | 378kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████▍            | 389kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████▉            | 399kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████████▍           | 409kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████████▉           | 419kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████████████▍          | 430kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████████          | 440kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████████▍         | 450kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 460kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████████▍        | 471kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 481kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████████████▍       | 491kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 501kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▌      | 512kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████████████      | 522kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▌     | 532kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 542kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▌    | 552kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████████████████    | 563kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▌   | 573kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████   | 583kB 8.0MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▌  | 593kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 604kB 8.0MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▌ | 614kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 624kB 8.0MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▋| 634kB 8.0MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 645kB 8.0MB/s \n",
            "\u001b[?25h  Building wheel for gpt-2-simple (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:\n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bj2IJLHP3KwE",
        "colab_type": "text"
      },
      "source": [
        "## GPU\n",
        "\n",
        "Colaboratory uses either a Nvidia T4 GPU or an Nvidia K80 GPU. The T4 is slightly faster than the old K80 for training GPT-2, and has more memory allowing you to train the larger GPT-2 models and generate more text.\n",
        "\n",
        "You can verify which GPU is active by running the cell below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sUmTooTW3osf",
        "colab_type": "code",
        "outputId": "6aa37ddc-f8f3-4510-d59f-28c4e2667ba1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mon Nov 11 23:07:57 2019       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 430.50       Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   73C    P8    33W / 149W |      0MiB / 11441MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wXB05bPDYxS",
        "colab_type": "text"
      },
      "source": [
        "## Downloading GPT-2\n",
        "\n",
        "If you're retraining a model on new text, you need to download the GPT-2 model first. \n",
        "\n",
        "There are three released sizes of GPT-2:\n",
        "\n",
        "* `124M` (default): the \"small\" model, 500MB on disk.\n",
        "* `355M`: the \"medium\" model, 1.5GB on disk.\n",
        "* `774M`: the \"large\" model, cannot currently be finetuned with Colaboratory but can be used to generate text from the pretrained model (see later in Notebook)\n",
        "\n",
        "Larger models have more knowledge, but take longer to finetune and longer to generate text. You can specify which base model to use by changing `model_name` in the cells below.\n",
        "\n",
        "The next cell downloads it from Google Cloud Storage and saves it in the Colaboratory VM at `/models/<model_name>`.\n",
        "\n",
        "This model isn't permanently saved in the Colaboratory VM; you'll have to redownload it if you want to retrain it at a later time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P8wSlgXoDPCR",
        "colab_type": "code",
        "outputId": "a568a231-660c-4d0c-91e5-74bb9bd4817e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "gpt2.download_gpt2(model_name=\"124M\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 317Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 86.7Mit/s]                                                   \n",
            "Fetching hparams.json: 1.05Mit [00:00, 563Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:06, 75.4Mit/s]                                  \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 542Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 1.05Mit [00:00, 126Mit/s]                                                 \n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 106Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N8KXuKWzQSsN",
        "colab_type": "text"
      },
      "source": [
        "## Mounting Google Drive\n",
        "\n",
        "The best way to get input text to-be-trained into the Colaboratory VM, and to get the trained model *out* of Colaboratory, is to route it through Google Drive *first*.\n",
        "\n",
        "Running this cell (which will only work in Colaboratory) will mount your personal Google Drive in the VM, which later cells can use to get data in/out. (it will ask for an auth code; that auth is not saved anywhere)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puq4iC6vUAHc",
        "colab_type": "code",
        "outputId": "c618bf47-4d8b-4123-98c4-ea25659b6960",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "gpt2.mount_gdrive()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BT__brhBCvJu",
        "colab_type": "text"
      },
      "source": [
        "## Uploading a Text File to be Trained to Colaboratory\n",
        "\n",
        "In the Colaboratory Notebook sidebar on the left of the screen, select *Files*. From there you can upload files:\n",
        "\n",
        "![alt text](https://i.imgur.com/TGcZT4h.png)\n",
        "\n",
        "Upload **any smaller text file**  (<10 MB) and update the file name in the cell below, then run the cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i1YQSLpls9YJ",
        "colab_type": "code",
        "outputId": "7e899cc7-b872-41ac-f5f0-62738f3f0f28",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6OFnPCLADfll",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "file_name = \"input.txt\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeeSKtNWUedE",
        "colab_type": "text"
      },
      "source": [
        "If your text file is larger than 10MB, it is recommended to upload that file to Google Drive first, then copy that file from Google Drive to the Colaboratory VM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Z6okFD8VKtS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#gpt2.copy_file_to_gdrive(file_name)\n",
        "gpt2.copy_file_from_gdrive(file_name)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdpZQXknFNY3",
        "colab_type": "text"
      },
      "source": [
        "## Finetune GPT-2\n",
        "\n",
        "The next cell will start the actual finetuning of GPT-2. It creates a persistent TensorFlow session which stores the training config, then runs the training for the specified number of `steps`. (to have the finetuning run indefinitely, set `steps = -1`)\n",
        "\n",
        "The model checkpoints will be saved in `/checkpoint/run1` by default. The checkpoints are saved every 500 steps (can be changed) and when the cell is stopped.\n",
        "\n",
        "The training might time out after 4ish hours; make sure you end training and save the results so you don't lose them!\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files.\n",
        "\n",
        "Other optional-but-helpful parameters for `gpt2.finetune`:\n",
        "\n",
        "\n",
        "*  **`restore_from`**: Set to `fresh` to start training from the base GPT-2, or set to `latest` to restart training from an existing checkpoint.\n",
        "* **`sample_every`**: Number of steps to print example output\n",
        "* **`print_every`**: Number of steps to print training progress.\n",
        "* **`learning_rate`**:  Learning rate for the training. (default `1e-4`, can lower to `1e-5` if you have <1MB input data)\n",
        "*  **`run_name`**: subfolder within `checkpoint` to save the model. This is useful if you want to work with multiple models (will also need to specify  `run_name` when loading the model)\n",
        "* **`overwrite`**: Set to `True` if you want to continue finetuning an existing model (w/ `restore_from='latest'`) without creating duplicate copies. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aeXshJM-Cuaf",
        "colab_type": "code",
        "outputId": "dcb4e17f-08d6-4105-d2ea-830dcd4fd443",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 428
        }
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.finetune(sess,\n",
        "              dataset=file_name,\n",
        "              model_name='124M',\n",
        "              steps=1000,\n",
        "              restore_from='fresh',\n",
        "              run_name='run1',\n",
        "              print_every=10,\n",
        "              sample_every=200,\n",
        "              save_every=500\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:17: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Loading checkpoint models/124M/model.ckpt\n",
            "INFO:tensorflow:Restoring parameters from models/124M/model.ckpt\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r  0%|          | 0/1 [00:00<?, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading dataset...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\r100%|██████████| 1/1 [00:01<00:00,  1.67s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "dataset has 275380 tokens\n",
            "Training...\n",
            "[10 | 53.55] loss=3.40 avg=3.40\n",
            "[20 | 99.09] loss=2.95 avg=3.17\n",
            "[30 | 144.57] loss=2.81 avg=3.05\n",
            "[40 | 189.76] loss=2.79 avg=2.98\n",
            "[50 | 234.97] loss=2.90 avg=2.97\n",
            "[60 | 280.42] loss=2.58 avg=2.90\n",
            "[70 | 325.65] loss=2.72 avg=2.87\n",
            "[80 | 370.98] loss=2.94 avg=2.88\n",
            "[90 | 416.33] loss=2.60 avg=2.85\n",
            "[100 | 461.70] loss=2.62 avg=2.83\n",
            "[110 | 507.14] loss=2.29 avg=2.77\n",
            "[120 | 552.57] loss=2.50 avg=2.75\n",
            "[130 | 598.05] loss=2.69 avg=2.75\n",
            "[140 | 643.37] loss=2.54 avg=2.73\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXSuTNERaw6K",
        "colab_type": "text"
      },
      "source": [
        "After the model is trained, you can copy the checkpoint folder to your own Google Drive.\n",
        "\n",
        "If you want to download it to your personal computer, it's strongly recommended you copy it there first, then download from Google Drive. The checkpoint folder is copied as a `.rar` compressed file; you can download it and uncompress it locally."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VHdTL8NDbAh3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gpt2.copy_checkpoint_to_gdrive(run_name='run1')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qQJgV_b4bmzd",
        "colab_type": "text"
      },
      "source": [
        "You're done! Feel free to go to the **Generate Text From The Trained Model** section to generate text based on your retrained model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pel-uBULXO2L",
        "colab_type": "text"
      },
      "source": [
        "## Load a Trained Model Checkpoint\n",
        "\n",
        "Running the next cell will copy the `.rar` checkpoint file from your Google Drive into the Colaboratory VM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DCcx5u7sbPTD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gpt2.copy_checkpoint_from_gdrive(run_name='run1')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTa6zf3e_9gV",
        "colab_type": "text"
      },
      "source": [
        "The next cell will allow you to load the retrained model checkpoint + metadata necessary to generate text.\n",
        "\n",
        "**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-fxL77nvAMAX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "gpt2.load_gpt2(sess, run_name='run1')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClJwpF_ACONp",
        "colab_type": "text"
      },
      "source": [
        "## Generate Text From The Trained Model\n",
        "\n",
        "After you've trained the model or loaded a retrained model from checkpoint, you can now generate text. `generate` generates a single text from the loaded model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4RNY6RBI9LmL",
        "colab_type": "code",
        "outputId": "3be4c658-1d97-44ff-fb0a-67ca3e9e3329",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        }
      },
      "source": [
        "gpt2.generate(sess, run_name='run1')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\"71\",\"He gave me a day and place he wants to meet. Is this an okay response?     \n",
            "     \n",
            "Hi Mr. X,     \n",
            "Monday, Nov 18th works great for me. I’m free anytime after 3pm, so let me know what time works best for you. Is there anything you’d like me to bring, such as a transcript or resumé?    \n",
            "Thanks,     \n",
            "aishstalksme\",\"Help responding to my interview invitation email\"\n",
            "\"72\",\"Ok I know it’s not my business but there’s a kid at my school (non-competitive Midwest) who’s a Questbridge finalist and I’m REALLY questioning that. Not his application or his credentials or anything like that, but the fact that he was financially eligible to be a finalist AT ALL. \n",
            "\n",
            "\n",
            "First, he’s an only child with married parents, lives in my neighborhood which is most definitely significantly pricier than a Questbridge-qualifying income should be able to afford (assuming he was at the max range for a Pell grant, his family would have to spend nothing for the good part of a decade to buy this...), has like 3 fucking brand new iPads, drives an Audi, travels overseas every summer to China, just to name a few conflicting things. \n",
            "\n",
            "\n",
            "His family moved from China a few years ago and I’m having a very hard time believing that they’re being honest about their overseas income. To be clear, I’m not applying for Questbridge or directly competing with him in that sense, but this feels absolutely unfair for everyone else who actually deserves to match through QB. He’s had access to opportunities (big internships in China, really expensive sports, etc.) that most QB qualifying applicants would never get to even attempt, and yet his family is intentionally making themselves look significantly poorer than they actually are so he can compete with them. I don’t know, it just doesn’t seem right...\n",
            "\n",
            "\n",
            "Thanks for coming to my TED talk\",\"I don’t know how to feel about this...\"\n",
            "\"73\",\"This is purely out of curiosity but have you guys been getting ur decisions for those that applied rolling? \n",
            "Just hella anxious rn\",\"Hello fellow penn state applicants?\"\n",
            "\"74\",\"I had no idea it would take 2 weeks to process it and I applied ED to a school that has a deadline at November 15th.\",\"Is it too late to send the CSS profile?\"\n",
            "\"75\",\"My school is very competitive, and we treat those who get into t20s as idols. My stats are below average among my class, and I know I’ll be judged negatively if I don’t get into a “top college” (top 30 at least). How do I stop feeling inferior to everyone at school once college letters are sent out? I can already envision myself sitting next to people with Harvard, MIT, and Stanford hoodies in class. I know I won’t see these people in a year, but still....\",\"How to not be embarrassed of the college you end up going to... ?\"\n",
            "\"76\",\"So I'm kind of confused as to what I should be doing with my transfer UC PIQs. Every UC rep I've talked to has said that they \\\"aren't looking for anything in particular\\\" and that they just want to get to know you, but they also say to treat the essays like an interview. I've shown several people my responses and each time I get conflicting feedback! I have one essay where I basically list a lot of my related experience, and one rep loved that essay, but others didn't. So I guess I have a couple questions.\n",
            "\n",
            "1. How do y'all feel about detailed, narrative hooks for the intro? \n",
            "2. For the mandatory question (what have you done to prepare), I list at least four things, it sounds a little matter of fact but I feel like they're all important. Should I focus on one thing and make it more flow-y/improve the writing, or sacrifice on literary aesthetic so I can talk about everything?\n",
            "3. While most of my essays are really matter of fact, eg. I did XYZ because XYZ and it had XYZ impact, I have one that has this central metaphor and catchy story. I feel like my personality shines through and I still talk about my achievements. People that I show tend to LOVE it, but I feel like that's not what the UC's are looking for.\n",
            "\n",
            "Overall I'm just so confused. I have absolutely no idea what I'm doing and everyone has a different idea of what my essays should be. I'd greatly appreciate any advice!\",\"UC Transfer PIQs\"\n",
            "\"77\",\"Is economics a good major?\",\"\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF4-PqF0Fl7R",
        "colab_type": "text"
      },
      "source": [
        "If you're creating an API based on your model and need to pass the generated text elsewhere, you can do `text = gpt2.generate(sess, return_as_list=True)[0]`\n",
        "\n",
        "You can also pass in a `prefix` to the generate function to force the text to start with a given character sequence and generate text from there (good if you add an indicator when the text starts).\n",
        "\n",
        "You can also generate multiple texts at a time by specifing `nsamples`. Unique to GPT-2, you can pass a `batch_size` to generate multiple samples in parallel, giving a massive speedup (in Colaboratory, set a maximum of 20 for `batch_size`).\n",
        "\n",
        "Other optional-but-helpful parameters for `gpt2.generate` and friends:\n",
        "\n",
        "*  **`length`**: Number of tokens to generate (default 1023, the maximum)\n",
        "* **`temperature`**: The higher the temperature, the crazier the text (default 0.7, recommended to keep between 0.7 and 1.0)\n",
        "* **`top_k`**: Limits the generated guesses to the top *k* guesses (default 0 which disables the behavior; if the generated output is super crazy, you may want to set `top_k=40`)\n",
        "* **`top_p`**: Nucleus sampling: limits the generated guesses to a cumulative probability. (gets good results on a dataset with `top_p=0.9`)\n",
        "* **`truncate`**: Truncates the input text until a given sequence, excluding that sequence (e.g. if `truncate='<|endoftext|>'`, the returned text will include everything before the first `<|endoftext|>`). It may be useful to combine this with a smaller `length` if the input texts are short.\n",
        "*  **`include_prefix`**: If using `truncate` and `include_prefix=False`, the specified `prefix` will not be included in the returned text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8DKMc0fiej4N",
        "colab_type": "code",
        "outputId": "2cad024b-5b85-49f3-c2f3-2497a0bc2620",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              length=650,\n",
        "              temperature=0.5,\n",
        "              prefix=\"HELLO\",\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "HELLO: [Monday-Thursday at 10:30 PM AEDT](https://www.reddit.com/r/ApplyingToCollege/comments/a9gydl/iama_high_school_senior_who_will_be_attending_the/)\n",
            "* [Tuesday at 11:30 PM AEDT](https://www.reddit.com/r/ApplyingToCollege/comments/7n6yez/iama_high_school_senior_who_will_be_attending_the/)\n",
            "* [Friday at 11:30 PM AEDT](https://www.reddit.com/r/ApplyingToCollege/comments/6h0dy1/iama_high_school_senior_who_will_be_attending_the/)\n",
            "* [Puerto Rico: A Quest for Peace](https://www.reddit.com/r/ApplyingToCollege/comments/55j4p5/iama_a_rising_puerto_ricerist_ama_and_a_rising_puerto_student_aid/)\n",
            "* [Part 1:  How To Start An Essay, \\\"Show Don't Tell,\\\" And Showcase Yourself In A Compelling Way](https://www.reddit.com/r/ApplyingToCollege/comments/bs9f9f/the_scholargrade_essay_series_part_1_how_to_start/)\n",
            "* [Part 2: Throw Away Everything You Learned In English Class](https://www.reddit.com/r/ApplyingToCollege/comments/cgwexx/the_scholargrade_essay_series_part_2_throw_away/)\n",
            "* [Part 3: Conquering the \\\"Why \\[School\\]\\\" Essay](https://www.reddit.com/r/ApplyingToCollege/comments/c9gydl/the_scholargrade_essay_series_part_3_conquering/)\n",
            "* [Part 4: What Makes An Essay Outstanding?](https://www.reddit.com/r/ApplyingToCollege/comments/9b7uow/the_scholargrade_essay_series_part_4_what_makes/)\n",
            "* [What to do when you're over the word count](https://www.reddit.com/r/ApplyingToCollege/comments/9bkooser/what_to_do_when_youre_over_the_word_count/)\n",
            "* [What to do when your essay is too short](https://www.reddit.com/r/ApplyingToCollege/comments/aefjas/what_to_do_when_your_essay_is_too_short/)\n",
            "* [How To End An Essay Gracefully](https://www.reddit.com/r/ApplyingToCollege/comments/8upen2/how_to_end_an_essay_gracefully/)\n",
            "\n",
            "====================\n",
            "HELLO_BY_MORNING_UP_AND_DYING_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_ABOUT_\n",
            "\n",
            "I know some of you are still looking for basic advice, so be sure to read through this. Don’t allow yourself to get overwhelmed by it; it’s just a guide to let you know what you might need to be doing. Here's my [post with a To-Do list.](https://www.reddit.com/r/ApplyingToCollege/comments/dam4mg/seniors_need_a_college_admissions_checklist_for/?utm_source=share&amp;utm_medium=web2x)\n",
            "\n",
            "**SOME SIMPLE STRESS RELIEVERS**\n",
            "\n",
            "This can be a stressful time, so I want you to be sure to take the time to recharge your batteries. It’s essential -- your essays will be stronger and your schoolwork will be better (and yes, you have to keep up your grades senior year.). Be sure to take time every day to sit back and reflect, get outside and take a walk, be in nature, take 3 - 5 minutes just to breathe -- really feel your inhales and exhales. Count to 4 when you inhale and count to 4 while you exhale. Other ways to recharge batteries -- a short Netflix binge or gaming period (brief. amounts.), a dance party in your room, walk and sing as loud as you can to your favorites. When I’m stressed, I listen to Bob Marley and sing along as loud as I can. His reminder that “none but ourselves can free our minds” in Redemption Song helps me every time. Maybe y’all can list some of yours down below.\n",
            "\n",
            "**KEEP DOING WHAT YOU'RE DOING**\n",
            "\n",
            "Keep supporting each other, advising each other, and sharing your thoughts, worries, concerns, excitement, words of encouragement, and advice! Don't forget to take time to breathe and recharge your batteries every once in a while :)\n",
            "\n",
            "Take care and remember to breathe. You got this,\n",
            "\n",
            "AdmissionsMom\",\"Monday News and Updates: AMA About Testing, Interview Posts, and more\"\n",
            "\"12\",\"I think it went pretty well! He was very nice and really encouraged me to be more casual and comfortable. I was worried beforehand because he has a real estate development degree and I didn't really know what to ask him, but it was fine because we talked about career and interest in general. We also talked about civil engineering and architecture (which I am interested in).\n",
            "\n",
            "He had 5 main questions:\n",
            "\n",
            "1) What attracted you to MIT\n",
            "\n",
            "2) What major are you interested in\n",
            "\n",
            "3) What have you done that you are most proud of\n",
            "\n",
            "4) Who has had the most influence on you\n",
            "\n",
            "====================\n",
            "HELLO_BELONG_TO_JUNIOR_ATHLETICS/)\n",
            "* [June 2015](https://www.reddit.com/r/teenagers/comments/3icqle/i_am_a_former_undergraduate_admissions_counselor/)\n",
            "* [June 2015 on other sub](https://www.reddit.com/r/UTAustin/comments/3f78l3/what_are_my_chances_how_do_i_apply_receive/)\n",
            "* [June 2015 on other other sub](https://www.reddit.com/r/IAmA/comments/3b3hwm/iama_former_undergraduate_admissions_counselor/)\n",
            "* [November 2011 while employed for UT](https://www.reddit.com/r/IAmA/comments/nz96p/iama_college_admissions_essay_reader_and/)\n",
            "\n",
            "Steve\\_nyc's AMAs: **College Admissions Counselor and Founder of A2C:**\n",
            "\n",
            "* [From 2016](https://www.reddit.com/r/ApplyingToCollege/comments/3nyvpv/iama_college_admission_counselor_mod/)\n",
            "* [From 2015](https://www.reddit.com/r/ApplyingToCollege/comments/3d2dji/iama_creator_and_moderator_of_rapplyingtocollege/)\n",
            "\n",
            "WilliamTheReader's AMA: [Top 5 USNews University Alum, Worked in Alma Mater's Admissions Office, Part-Time Elite Admissions Consultant](https://www.reddit.com/r/ApplyingToCollege/comments/9f8zl3/ama_top_5_usnews_university_alum_worked_in_alma/)\n",
            "\n",
            "[Ethan Sawyer: the College Essay Guy's AMA](https://www.reddit.com/r/ApplyingToCollege/comments/9i177k/im_ethan_sawyer_the_college_essay_guy_and_ive/). He wrote the first essay guide I shared.\n",
            "\n",
            "Copied from steve\\_nyc (big shoutout here):\n",
            "\n",
            "**Admissions Officers:**\n",
            "\n",
            "* [Former Cornell admission officer, Nelson Ureña](https://www.reddit.com/r/ApplyingToCollege/comments/3fmqfm/hi_im_nelson_ure%C3%B1a_i_am_a_former_admissions/)\n",
            "* [Current Reed admission officer, Milyon Trulove](https://www.reddit.com/r/ApplyingToCollege/comments/40ypsw/im_milyon_trulove_vice_president_and_dean_of/)\n",
            "* [Accepted to all 8 Ivy Leagues, Kwasi Enin](https://www.reddit.com/r/ApplyingToCollege/comments/\n",
            "====================\n",
            "HELLO_BY_MELBOURNE_AND_JUNIOR_ABOUT_SENIOR_ABOUT_SOME_CHANGES_AND_REVISITED_BY_MELBOURNE_AND_JUNIOR_ABOUT_SOME_CHANGES)\n",
            "* [This post](https://www.reddit.com/r/ApplyingToCollege/comments/bydczc/list_of_competitions_and_programs_to_pursue_to/) links several resources to find competitions/programs for your ECs or to find ECs based on your academic interest! I don't think you should be basing your activities on prestigious awards, but if you *are* doing something and you want to find ways to get more involved or get rewarded, this is a good resource.\n",
            "\n",
            "# LORS:\n",
            "\n",
            "steve\\_nyc: [How to Ask Teachers for College Recommendation Letters](https://www.reddit.com/r/ApplyingToCollege/comments/1nqrwp/how_to_ask_teachers_for_college_recommendation/)\n",
            "\n",
            "novembrr: [The secret to having excellent LORs](https://www.reddit.com/r/ApplyingToCollege/comments/6damac/the_secret_to_having_excellent_letters_of/)\n",
            "\n",
            "ScholarGrade: [How to get top LORs that stand out from the stack](https://www.reddit.com/r/ApplyingToCollege/comments/bw5h8u/rising_seniors_now_is_the_time_to_ask_about/)\n",
            "\n",
            "# AP Score Reporting:\n",
            "\n",
            "novembrr: [When AP Scores Matter and When They Don't (in my experience as an admissions reader at Berkeley and UChicago)](https://www.reddit.com/r/ApplyingToCollege/comments/c9vmfk/when_ap_scores_matter_and_when_they_dont_in_my/)\n",
            "\n",
            "u/admissionsmom: [Let's Talk about your AP Score](https://www.reddit.com/r/ApplyingToCollege/comments/c9gydl/lets_talk_about_your_ap_scores/)\n",
            "\n",
            "# Interviews:\n",
            "\n",
            "ScholarGrade: [There have been many questions about interviews. Here's my guide](https://www.reddit.com/r/ApplyingToCollege/comments/9ujnzd/there_have_been_many_questions_about_interviews/)\n",
            "\n",
            "WilliamTheReader: [Interview Tips!](https://www.reddit.com/r/ApplyingToCollege/comments/admjvg/williamthereader_interview_tips/)\n",
            "\n",
            "novembrr:  [How to prepare for an interview: a guide by Novembrr, former UChicago admissions reader &amp; alumna interviewer](https://www.reddit.\n",
            "====================\n",
            "HELLO. I know I’m not the most competitive subreddit out there, but I really want to get my points figured, so I can send them out. Can you give me some advice on how to start and tackle Emory University's supplemental essay?\",\"Provided I can pay my college admission taxes by the 19th?\"\n",
            "\"84\",\"So I’m pretty sure my family does not qualify for any need-based aid, but my parents want to file FAFSA anyway in case our financial circumstances change and since many colleges require you to file it to be eligible for merit scholarships. But I was wondering whether colleges will view this negatively and/or if this will negatively impact my application (since I’m filing FAFSA even though I’m from an upper-middle class family that doesn’t qualify for aid). \n",
            "\n",
            "Thank you.\",\"Will colleges view me negatively if I file FAFSA and am from a middle-upper class family?\"\n",
            "\"85\",\"What if my actual CBSE percentage might be less than my predicted score  \n",
            "Let’s say I get 83%-85% than a 90% \n",
            "\n",
            "Will I be revoked then ?\",\"To all those who are in college and gave CBSE boards .....\"\n",
            "\"86\",\"If you have any information about these. I would be really happy if you can answer them.\",\"What should I do to get recruited and can you share your recruiting video on additional part of common app? I don’t know what can I write on the additional part? For example, can I write that my school doesn’t offer ap/ib so i take online college courses?\"\n",
            "\"87\",\"Hi, I’m looking at Canada schools as I live fairly close to the border. What is the best evs science/ studies program focused on policy? Also is international tuition too much?\",\"What is the best college in Canada for Environmental Science?\"\n",
            "\"88\",\"means i was exposed to STEM (Science, technology, engineering, and math) during high school, and now I'm exposed to all those things.\n",
            "\n",
            "I'm not going to copy or anything, I just have absolutely no clue what they are. I know what colleges are and aren’t going to do about it.\n",
            "\n",
            "I know what classes are usually done, and I don’t even care about what major is doing me in. But is it worth it for me to just assume it is,\"5is what it is and don’t take it as seriously as I could've.\"\n",
            "\"89\",\"What if my actual CBSE percentage might be less than my predicted score  \n",
            "\n",
            "Let’s say I get 83%-85% than a 90% \n",
            "\n",
            "Will I be revoked then ?\",\"To all those who are in college and gave CBSE boards .....\"\n",
            "\"90\",\"If you have any information about these. I would be really happy if you can answer them.\",\"What should I do to get recruited and can you share your recruiting video on additional part of common app? I don’t know what can I write on the additional part\n",
            "====================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjjEN2Tafhl2",
        "colab_type": "text"
      },
      "source": [
        "For bulk generation, you can generate a large amount of text to a file and sort out the samples locally on your computer. The next cell will generate a generated text file with a unique timestamp.\n",
        "\n",
        "You can rerun the cells as many times as you want for even more generated texts!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fa6p6arifSL0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())\n",
        "\n",
        "gpt2.generate_to_file(sess,\n",
        "                      destination_path=gen_file,\n",
        "                      length=500,\n",
        "                      temperature=0.7,\n",
        "                      nsamples=100,\n",
        "                      batch_size=20\n",
        "                      )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-LRex8lfv1g",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# may have to run twice to get file to download\n",
        "files.download(gen_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQAN3M6RT7Kj",
        "colab_type": "text"
      },
      "source": [
        "## Generate Text From The Pretrained Model\n",
        "\n",
        "If you want to generate text from the pretrained model, not a finetuned model, pass `model_name` to `gpt2.load_gpt2()` and `gpt2.generate()`.\n",
        "\n",
        "This is currently the only way to generate text from the 774M \"large\" model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hsUd_jHgUZnD",
        "colab_type": "code",
        "outputId": "4e0c8a3f-3527-41c4-e3fe-3357f3f8f6c2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 158
        }
      },
      "source": [
        "model_name = \"774M\"\n",
        "\n",
        "gpt2.download_gpt2(model_name=model_name)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fetching checkpoint: 1.05Mit [00:00, 354Mit/s]                                                      \n",
            "Fetching encoder.json: 1.05Mit [00:00, 131Mit/s]                                                    \n",
            "Fetching hparams.json: 1.05Mit [00:00, 279Mit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 3.10Git [00:23, 131Mit/s]                                  \n",
            "Fetching model.ckpt.index: 1.05Mit [00:00, 380Mit/s]                                                \n",
            "Fetching model.ckpt.meta: 2.10Mit [00:00, 226Mit/s]                                                 \n",
            "Fetching vocab.bpe: 1.05Mit [00:00, 199Mit/s]                                                       \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BAe4NpKNUj2C",
        "colab_type": "code",
        "outputId": "b09bfe1d-2ff8-4b8a-fffb-273d28d5d4ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        }
      },
      "source": [
        "sess = gpt2.start_tf_sess()\n",
        "\n",
        "gpt2.load_gpt2(sess, model_name=model_name)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0828 18:37:58.571830 139905369159552 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Loading pretrained model models/774M/model.ckpt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-xInIZKaU104",
        "colab_type": "code",
        "outputId": "56348e28-7d08-45e3-c859-f26c0efd066d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 797
        }
      },
      "source": [
        "gpt2.generate(sess,\n",
        "              model_name=model_name,\n",
        "              prefix=\"The secret of life is\",\n",
        "              length=100,\n",
        "              temperature=0.7,\n",
        "              top_p=0.9,\n",
        "              nsamples=5,\n",
        "              batch_size=5\n",
        "              )"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The secret of life is that it's really easy to make it complicated,\" said Bill Nye, the host of the popular science show \"Bill Nye the Science Guy.\" \"And this is one of the reasons why we all need to be smarter about science, because we can't keep up with the amazing things that are going on all the time.\"\n",
            "\n",
            "While Nye is correct that \"everything that's going on all the time\" is making the world a better place, he misses the point. This is not\n",
            "====================\n",
            "The secret of life is in the rhythm of the universe. It's not a mystery. It's not a mystery to me. It's the nature of the universe. It's the beauty of the universe. It's the way the universe works. It's the way the universe is. It's the way the universe is going to work. It's the way the universe is. It's the way the universe is. It's the way the universe is. It's the way the universe is. It's the way\n",
            "====================\n",
            "The secret of life is in the universe.\n",
            "\n",
            "\n",
            "-\n",
            "\n",
            "The Red Devil\n",
            "\n",
            "It's the end of the world as we know it, and the only thing that can save us is a band of super-powered individuals known as the Red Devil.\n",
            "\n",
            "\n",
            "The Red Devil is a group of super-powered individuals who are seeking the secret of life and the only way they know how to do it is by taking on the roles of a variety of different super-powered individuals, each of which has their own\n",
            "====================\n",
            "The secret of life is in the mixing of the elements, and it is the mixing of the elements that makes life possible.\"\n",
            "\n",
            "But in the world of food science, the idea of a \"complex\" or \"complexity\" is almost entirely imaginary.\n",
            "\n",
            "As a scientist, I'm fascinated by the question of how life first began.\n",
            "\n",
            "It's the question that drives my work and the work of the scientists who work on it.\n",
            "\n",
            "My current research is exploring how microbes work in the first moments\n",
            "====================\n",
            "The secret of life is the journey of life, the search for the truth.\n",
            "\n",
            "4.4.2. The last thing you know\n",
            "\n",
            "There is nothing more important than the last thing you know.\n",
            "\n",
            "4.4.3. The little things that make all the difference\n",
            "\n",
            "The little things that make all the difference.\n",
            "\n",
            "4.4.4. The truth is the best teacher\n",
            "\n",
            "The truth is the best teacher.\n",
            "\n",
            "4.4.5. The truth is what\n",
            "====================\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ig-KVgkCDCKD",
        "colab_type": "text"
      },
      "source": [
        "# Etcetera\n",
        "\n",
        "If the notebook has errors (e.g. GPU Sync Fail), force-kill the Colaboratory virtual machine and restart it with the command below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rIHiVP53FnsX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!kill -9 -1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmTXWNUygS5E",
        "colab_type": "text"
      },
      "source": [
        "# LICENSE\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Max Woolf\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE."
      ]
    }
  ]
}