Retrained_train_again_GPT_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Retrained_train_again_GPT-2.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vsnupoudel/CarHuman-ObjectDetection-yolov5-training-and-inference/blob/main/Retrained_train_again_GPT_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pzxl1vYX-1kk"
      },
      "source": [
        "Setup:\n",
        "\n",
        "1) Make sure GPU is enabled, go to edit->notebook settings->Hardware Accelerator GPU\n",
        "\n",
        "2) Make a copy to your google drive, click on copy to drive in panel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iW0abT07ZkhZ"
      },
      "source": [
        "Note: Colab will reset after 12 hours make sure to save your model checkpoints to google drive around 10-11 hours mark or before, then go to runtime->reset all runtimes. Now copy your train model back into colab and start training again from the previous checkpoint."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iLXW02eIYpcB"
      },
      "source": [
        "clone and cd into repo, nshepperd's fork https://github.com/nshepperd/gpt-2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ICYu3w9hIJkC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c31e38e9-fc0d-4df7-fc87-24931b22d6f6"
      },
      "source": [
        "!git clone https://github.com/nshepperd/gpt-2.git"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'gpt-2'...\n",
            "remote: Enumerating objects: 435, done.\u001b[K\n",
            "remote: Counting objects: 100% (64/64), done.\u001b[K\n",
            "remote: Compressing objects: 100% (51/51), done.\u001b[K\n",
            "remote: Total 435 (delta 20), reused 45 (delta 13), pack-reused 371\u001b[K\n",
            "Receiving objects: 100% (435/435), 4.48 MiB | 19.34 MiB/s, done.\n",
            "Resolving deltas: 100% (221/221), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6eEIs3ApZUVO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3df3237f-a0e3-4b30-ac43-e2ab0ec13946"
      },
      "source": [
        "cd gpt-2"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/gpt-2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qtn1qZPgZLb0"
      },
      "source": [
        "Install requirements"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "434oOx0bZH6J",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c94e2de0-1757-4464-ce18-72df3b1a676d"
      },
      "source": [
        "!pip3 install -r requirements.txt"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting fire>=0.1.3\n",
            "  Downloading fire-0.4.0.tar.gz (87 kB)\n",
            "\u001b[K     |████████████████████████████████| 87 kB 5.4 MB/s \n",
            "\u001b[?25hCollecting regex==2017.4.5\n",
            "  Downloading regex-2017.04.05.tar.gz (601 kB)\n",
            "\u001b[K     |████████████████████████████████| 601 kB 38.6 MB/s \n",
            "\u001b[?25hCollecting requests==2.21.0\n",
            "  Downloading requests-2.21.0-py2.py3-none-any.whl (57 kB)\n",
            "\u001b[K     |████████████████████████████████| 57 kB 5.8 MB/s \n",
            "\u001b[?25hCollecting tqdm==4.31.1\n",
            "  Downloading tqdm-4.31.1-py2.py3-none-any.whl (48 kB)\n",
            "\u001b[K     |████████████████████████████████| 48 kB 4.7 MB/s \n",
            "\u001b[?25hCollecting toposort==1.5\n",
            "  Downloading toposort-1.5-py2.py3-none-any.whl (7.6 kB)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests==2.21.0->-r requirements.txt (line 3)) (2021.10.8)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests==2.21.0->-r requirements.txt (line 3)) (1.24.3)\n",
            "Collecting idna<2.9,>=2.5\n",
            "  Downloading idna-2.8-py2.py3-none-any.whl (58 kB)\n",
            "\u001b[K     |████████████████████████████████| 58 kB 6.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests==2.21.0->-r requirements.txt (line 3)) (3.0.4)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from fire>=0.1.3->-r requirements.txt (line 1)) (1.15.0)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.7/dist-packages (from fire>=0.1.3->-r requirements.txt (line 1)) (1.1.0)\n",
            "Building wheels for collected packages: regex, fire\n",
            "  Building wheel for regex (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for regex: filename=regex-2017.4.5-cp37-cp37m-linux_x86_64.whl size=534445 sha256=987ebfac687f17e75d4a49520054b0567c191d57cc4d84910e0e410b4c1c4ae7\n",
            "  Stored in directory: /root/.cache/pip/wheels/3d/e8/a5/d4894e7ef29935f75c6074409ce8ca80a0271f0ce2a30da5d3\n",
            "  Building wheel for fire (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115943 sha256=64952b543df1745779702487b9b3d109010bb63739986334c053ae1c7505a672\n",
            "  Stored in directory: /root/.cache/pip/wheels/8a/67/fb/2e8a12fa16661b9d5af1f654bd199366799740a85c64981226\n",
            "Successfully built regex fire\n",
            "Installing collected packages: idna, tqdm, toposort, requests, regex, fire\n",
            "  Attempting uninstall: idna\n",
            "    Found existing installation: idna 2.10\n",
            "    Uninstalling idna-2.10:\n",
            "      Successfully uninstalled idna-2.10\n",
            "  Attempting uninstall: tqdm\n",
            "    Found existing installation: tqdm 4.62.3\n",
            "    Uninstalling tqdm-4.62.3:\n",
            "      Successfully uninstalled tqdm-4.62.3\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.23.0\n",
            "    Uninstalling requests-2.23.0:\n",
            "      Successfully uninstalled requests-2.23.0\n",
            "  Attempting uninstall: regex\n",
            "    Found existing installation: regex 2019.12.20\n",
            "    Uninstalling regex-2019.12.20:\n",
            "      Successfully uninstalled regex-2019.12.20\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "spacy 2.2.4 requires tqdm<5.0.0,>=4.38.0, but you have tqdm 4.31.1 which is incompatible.\n",
            "panel 0.12.1 requires tqdm>=4.48.0, but you have tqdm 4.31.1 which is incompatible.\n",
            "google-colab 1.0.0 requires requests~=2.23.0, but you have requests 2.21.0 which is incompatible.\n",
            "fbprophet 0.7.1 requires tqdm>=4.36.1, but you have tqdm 4.31.1 which is incompatible.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "Successfully installed fire-0.4.0 idna-2.8 regex-2017.4.5 requests-2.21.0 toposort-1.5 tqdm-4.31.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WvUQhgK3PQ4L"
      },
      "source": [
        "Mount drive to access google drive for saving and accessing checkpoints later. Have to log in to your google account"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FNpf6R4ahYSN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4790c8f4-0462-47f0-a2a1-032700974381"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o1hrgeKFYsuE"
      },
      "source": [
        "Download the model data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A498TySgHYyF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "513d2add-7a58-455f-8e17-de7c3157257a"
      },
      "source": [
        "!python3 download_model.py 117M"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fetching checkpoint: 1.00kit [00:00, 741kit/s]                                                      \n",
            "Fetching encoder.json: 1.04Mit [00:01, 990kit/s]                                                    \n",
            "Fetching hparams.json: 1.00kit [00:00, 566kit/s]                                                    \n",
            "Fetching model.ckpt.data-00000-of-00001: 498Mit [00:41, 11.9Mit/s]                                  \n",
            "Fetching model.ckpt.index: 6.00kit [00:00, 3.85Mit/s]                                               \n",
            "Fetching model.ckpt.meta: 472kit [00:00, 619kit/s]                                                  \n",
            "Fetching vocab.bpe: 457kit [00:00, 620kit/s]                                                        \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5UDpEGjfO8Q2"
      },
      "source": [
        "# !python3 download_model.py 345M"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zq-YwRnNOBYO"
      },
      "source": [
        "encoding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7oJPQtdLbbeK"
      },
      "source": [
        "!export PYTHONIOENCODING=UTF-8"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KzSbAvePgsI"
      },
      "source": [
        "Fetch checkpoints if you have them saved in google drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cA2Wk7yIPmS6"
      },
      "source": [
        "!cp -r /content/drive/My\\ Drive/checkpoint/ /content/gpt-2/models"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!ls -alh /content/drive/My\\ Drive/checkpoint/run1"
      ],
      "metadata": {
        "id": "q3VFMMoJIq70",
        "outputId": "b5553078-bc00-4367-afea-650c96753dc0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total 1.9G\n",
            "-rw------- 1 root root  200 Jan  9 11:24 checkpoint\n",
            "-rw------- 1 root root    5 Jan  9 11:24 counter\n",
            "-rw------- 1 root root 266K Jan  9 11:24 events.out.tfevents.1641722773.f2d20458afe9\n",
            "-rw------- 1 root root 475M Jan  9 11:24 model-1000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 11:24 model-1000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 11:24 model-1000.meta\n",
            "-rw------- 1 root root 475M Jan  9 11:24 model-2000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 11:24 model-2000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 11:24 model-2000.meta\n",
            "-rw------- 1 root root 475M Jan  9 11:24 model-3000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 11:24 model-3000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 11:24 model-3000.meta\n",
            "-rw------- 1 root root 475M Jan  9 11:24 model-4000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 11:24 model-4000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 11:24 model-4000.meta\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0p--9zwqQRTc"
      },
      "source": [
        "\n",
        "Let's get our train on! In this case the file is A Tale of Two Cities (Charles Dickens) from Project Gutenberg. To change the dataset GPT-2 models will fine-tune on, change this URL to another .txt file, and change corresponding part of the next cell. Note that you can use small datasets if you want but you will have to be sure not to run the fine-tuning for too long or you will overfit badly. Roughly, expect interesting results within minutes to hours in the 1-10s of megabyte ballpark, and below this you may want to stop the run early as fine-tuning can be very fast."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QOCvrs-DHvxa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8c62653-091a-4f7f-df77-f7e85ee6f3a2"
      },
      "source": [
        "# !wget https://www.gutenberg.org/files/98/98-0.txt\n",
        "!gdown --id 1hlvOXJqe3fdxjQilNfLRwPud_KWcOVEE -O '/content/en_US.zip'\n",
        "!unzip -o /content/en_US.zip "
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1hlvOXJqe3fdxjQilNfLRwPud_KWcOVEE\n",
            "To: /content/en_US.zip\n",
            "100% 262M/262M [00:03<00:00, 75.4MB/s]\n",
            "Archive:  /content/en_US.zip\n",
            "  inflating: en_US/en_US.blogs.txt   \n",
            "  inflating: en_US/en_US.news.txt    \n",
            "  inflating: en_US/en_US.twitter.txt  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "# cd /content/gpt-2/en_US/\n",
        "# cat en_US.blogs.txt en_US.news.txt en_US.twitter.txt > train_text.txt"
      ],
      "metadata": {
        "id": "JyZKO7XhDCZZ"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!cd /content/gpt-2/"
      ],
      "metadata": {
        "id": "eTTl4jBxDj5Y"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with open(\"/content/gpt-2/en_US/en_US.twitter.txt\", 'r') as f:\n",
        "  smallerfile = f.readlines(99999)\n",
        "  smallerfile = f.readlines(50000)\n",
        "  with open(\"/content/gpt-2/en_US/smallerfile.txt\", \"w\") as file1:\n",
        "    # Writing data to a file\n",
        "    file1.writelines(smallerfile)"
      ],
      "metadata": {
        "id": "RTU73IP9Q955"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yPfJ5b3CQXqr"
      },
      "source": [
        "\n",
        "Start training, add --model_name '345M' to use 345 model"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls -alh /content/gpt-2/en_US/"
      ],
      "metadata": {
        "id": "QAkLHp1dE4zt",
        "outputId": "d5b44416-1a0f-4438-916e-fc7f60e47145",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total 557M\n",
            "drwxr-xr-x 2 root root 4.0K Jan  9 13:32 .\n",
            "drwxr-xr-x 7 root root 4.0K Jan  9 13:31 ..\n",
            "-rw-r--r-- 1 root root 201M Jul 22  2014 en_US.blogs.txt\n",
            "-rw-r--r-- 1 root root 197M Jul 22  2014 en_US.news.txt\n",
            "-rw-r--r-- 1 root root 160M Jul 22  2014 en_US.twitter.txt\n",
            "-rw-r--r-- 1 root root  49K Jan  9 13:32 smallerfile.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pEn_ihcGI00T"
      },
      "source": [
        "# !PYTHONPATH=src ./train.py --dataset /content/gpt-2/en_US/smallerfile.txt --model_name '117M'"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!ls -alh /content/gpt-2/models/checkpoint/run1/"
      ],
      "metadata": {
        "id": "hVghaI5OGNUE",
        "outputId": "d38584cb-9017-45eb-c00f-f3c1d47df361",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total 1.9G\n",
            "drwx------ 2 root root 4.0K Jan  9 13:25 .\n",
            "drwx------ 3 root root 4.0K Jan  9 13:24 ..\n",
            "-rw------- 1 root root  200 Jan  9 13:25 checkpoint\n",
            "-rw------- 1 root root    5 Jan  9 13:24 counter\n",
            "-rw------- 1 root root 266K Jan  9 13:24 events.out.tfevents.1641722773.f2d20458afe9\n",
            "-rw------- 1 root root 475M Jan  9 13:24 model-1000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 13:24 model-1000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 13:24 model-1000.meta\n",
            "-rw------- 1 root root 475M Jan  9 13:25 model-2000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 13:25 model-2000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 13:24 model-2000.meta\n",
            "-rw------- 1 root root 475M Jan  9 13:25 model-3000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 13:25 model-3000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 13:25 model-3000.meta\n",
            "-rw------- 1 root root 475M Jan  9 13:25 model-4000.data-00000-of-00001\n",
            "-rw------- 1 root root 5.1K Jan  9 13:25 model-4000.index\n",
            "-rw------- 1 root root 2.6M Jan  9 13:25 model-4000.meta\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vS1RJJDFOPnb"
      },
      "source": [
        "Save our checkpoints to start training again later"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JretqG1zOXdi"
      },
      "source": [
        "# !cp -r /content/gpt-2/checkpoint/ /content/drive/My\\ Drive/"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6D-i7vERWbNS"
      },
      "source": [
        "Load your trained model for use in sampling below (117M or 345M)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VeETvWvrbKga"
      },
      "source": [
        "!cp -r /content/gpt-2/models/checkpoint/run1/* /content/gpt-2/models/117M/"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "np0r6qfXBeUX"
      },
      "source": [
        "# !cp -r /content/gpt-2/checkpoint/run1/* /content/gpt-2/models/345M/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GmnSrXqtfRbq"
      },
      "source": [
        "Generate conditional samples from the model given a prompt you provide -  change top-k hyperparameter if desired (default is 40),  if you're using 345M, add \"--model-name 345M\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "utJj-iY4gHwE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ccfc7539-ae28-40fa-9a5a-581a8d3f310b"
      },
      "source": [
        "!python3 src/interactive_conditional_samples.py --top_k 10 --model_name \"117M\" \\\n",
        "--temperature=0.7 --length=2 --nsamples=3"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-01-09 13:35:51.129279: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:60: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:1096: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.random.categorical` instead.\n",
            "Model prompt >>> This is a good day to\n",
            "======================================== SAMPLE 1 ========================================\n",
            " think big\n",
            "======================================== SAMPLE 2 ========================================\n",
            " think big\n",
            "======================================== SAMPLE 3 ========================================\n",
            " think big\n",
            "================================================================================\n",
            "Model prompt >>> The republicans are\n",
            "======================================== SAMPLE 1 ========================================\n",
            " going to\n",
            "======================================== SAMPLE 2 ========================================\n",
            " having a\n",
            "======================================== SAMPLE 3 ========================================\n",
            " going to\n",
            "================================================================================\n",
            "Model prompt >>> Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py\", line 5570, in get_controller\n",
            "    yield default\n",
            "  File \"src/interactive_conditional_samples.py\", line 73, in interact_model\n",
            "    raw_text = input(\"Model prompt >>> \")\n",
            "KeyboardInterrupt\n",
            "\n",
            "During handling of the above exception, another exception occurred:\n",
            "\n",
            "Traceback (most recent call last):\n",
            "  File \"src/interactive_conditional_samples.py\", line 91, in <module>\n",
            "    fire.Fire(interact_model)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/fire/core.py\", line 141, in Fire\n",
            "    component_trace = _Fire(component, args, parsed_flag_args, context, name)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/fire/core.py\", line 471, in _Fire\n",
            "    target=component.__name__)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/fire/core.py\", line 681, in _CallAndUpdateTrace\n",
            "    component = fn(*varargs, **kwargs)\n",
            "  File \"src/interactive_conditional_samples.py\", line 88, in interact_model\n",
            "    print(\"=\" * 80)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py\", line 1628, in __exit__\n",
            "    exec_tb)\n",
            "  File \"/usr/lib/python3.7/contextlib.py\", line 130, in __exit__\n",
            "    self.gen.throw(type, value, traceback)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py\", line 5570, in get_controller\n",
            "    yield default\n",
            "KeyboardInterrupt\n",
            "Exception ignored in: <generator object _DefaultGraphStack.get_controller at 0x7fdf60c5e150>\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py\", line 5770, in get_controller\n",
            "TypeError: 'NoneType' object is not callable\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# !cp /content/gpt-2/models/345M/vocab.bpe /content/gpt-2/models/4000M"
      ],
      "metadata": {
        "id": "xcQX-eBnoQ2K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LeDhY97XMDXn"
      },
      "source": [
        "To check flag descriptions, use:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBaj2L_KMAgb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5acaab42-c7e0-4235-ffd7-74cc456bb8b9"
      },
      "source": [
        "!python3 src/interactive_conditional_samples.py -- --help"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1mNAME\u001b[0m\n",
            "    interactive_conditional_samples.py - Interactively run the model :model_name=124M : String, which model to use :seed=None : Integer seed for random number generators, fix seed to reproduce results :nsamples=1 : Number of samples to return total :batch_size=1 : Number of batches (only affects speed/memory).  Must divide nsamples. :length=None : Number of tokens in generated text, if None (default), is determined by model hyperparameters :temperature=1 : Float value controlling randomness in boltzmann distribution. Lower temperature results in less random completions. As the temperature approaches zero, the model will become deterministic and repetitive. Higher temperature results in more random completions. :top_k=0 : Integer value controlling diversity. 1 means only 1 word is considered for each step (token), resulting in deterministic completions, while 40 means 40 words are considered at each step. 0 (default) is a special setting meaning no restrictions. 40 generally is a good value. :models_dir : path to parent folder containing model subfolders (i.e. contains the <model_name> folder)\n",
            "\n",
            "\u001b[1mSYNOPSIS\u001b[0m\n",
            "    interactive_conditional_samples.py <flags>\n",
            "\n",
            "\u001b[1mDESCRIPTION\u001b[0m\n",
            "    Interactively run the model :model_name=124M : String, which model to use :seed=None : Integer seed for random number generators, fix seed to reproduce results :nsamples=1 : Number of samples to return total :batch_size=1 : Number of batches (only affects speed/memory).  Must divide nsamples. :length=None : Number of tokens in generated text, if None (default), is determined by model hyperparameters :temperature=1 : Float value controlling randomness in boltzmann distribution. Lower temperature results in less random completions. As the temperature approaches zero, the model will become deterministic and repetitive. Higher temperature results in more random completions. :top_k=0 : Integer value controlling diversity. 1 means only 1 word is considered for each step (token), resulting in deterministic completions, while 40 means 40 words are considered at each step. 0 (default) is a special setting meaning no restrictions. 40 generally is a good value. :models_dir : path to parent folder containing model subfolders (i.e. contains the <model_name> folder)\n",
            "\n",
            "\u001b[1mFLAGS\u001b[0m\n",
            "    --model_name=\u001b[4mMODEL_NAME\u001b[0m\n",
            "        Default: '124M'\n",
            "    --seed=\u001b[4mSEED\u001b[0m\n",
            "        Type: Optional[]\n",
            "        Default: None\n",
            "    --nsamples=\u001b[4mNSAMPLES\u001b[0m\n",
            "        Default: 1\n",
            "    --batch_size=\u001b[4mBATCH_SIZE\u001b[0m\n",
            "        Default: 1\n",
            "    --length=\u001b[4mLENGTH\u001b[0m\n",
            "        Type: Optional[]\n",
            "        Default: None\n",
            "    --temperature=\u001b[4mTEMPERATURE\u001b[0m\n",
            "        Default: 1\n",
            "    --top_k=\u001b[4mTOP_K\u001b[0m\n",
            "        Default: 0\n",
            "    --top_p=\u001b[4mTOP_P\u001b[0m\n",
            "        Default: 1\n",
            "    --models_dir=\u001b[4mMODELS_DIR\u001b[0m\n",
            "        Default: 'models'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K8rSqkGxg5OK"
      },
      "source": [
        "Generate unconditional samples from the model,  if you're using 345M, add \"--model-name 345M\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LaQUEnRxWc3c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d0583392-21b1-4fbb-bfe0-8e0747592fd4"
      },
      "source": [
        "!python3 src/generate_unconditional_samples.py --model_name \"117M\" | tee /tmp/samples"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-01-09 11:50:15.936280: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "WARNING:tensorflow:From /content/gpt-2/src/sample.py:60: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:1096: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.random.categorical` instead.\n",
            "That is not what I \n",
            "======================================== SAMPLE 1 ========================================\n",
            "RT: Hey Sherry! Howdy is always online and helpful!\n",
            "i do some research for me and then try to find information I can agree to\n",
            "(Interrupting interferon over warning interferes with the clock in ms. High Signal Semiconductance Delay.)\n",
            "Thanks! I am aware of it.\n",
            "As a child my love, you were most likely extremely observant. With each passing month, the more we learn, the more we find we are, the lesser our time is.\n",
            "I could go for a high drama year, but I am extremely wary of moving forward if I don't learn to love.\n",
            "Lucille Inspirations is a phonemic circus.\n",
            "Auditory hallucinations are most often apparent when listening to music with voices in the background.\n",
            "See you on the show street!\n",
            "#Goodnight, & have a great night!\n",
            "Sincerely,\n",
            "\n",
            "Colton\n",
            "Colton! I'm so sorry for the inconvenience!\n",
            "of course Loudspeaker Boutique will not be held responsible if non-emergency shipments become public.\n",
            "My typo said 90 days - would like a reminder when new items are available.\n",
            ";\n",
            "My all time favorite Bon Appetit sticker - if I don't win a prize I don't care!\n",
            "Because.\n",
            "Go back to India.\n",
            "Greetings, Krishnamurth and all of Belgium.\n",
            "Loving new PHD scanner.\n",
            "Guess that ass is going to woo a lot of girls despite its looks -hips*.\n",
            "A girl will often tell you that they are embarrassed to admit it, but I never thought I would see such a amazing girl..\n",
            "i like the taste of skin of the man. i think i may tattoo his..\n",
            "oh well...and for the record I never said I was a pussy.\n",
            "being nice to a fault is a horrible mindset to me.\n",
            "I never abused anyone.\n",
            "David's Corner is a family-friendly night of soul food & good vibes. 8-9 $ and drink specials coming soon!\n",
            "If you have today pray for my family\n",
            "Loved that cigar.\n",
            "#FF Back at ya!\n",
            "There are different standards for artists and different dos and don'ts for artists.\n",
            "#FF There's a difference between being happy and being sad\n",
            "I never thought spending $225 - $250 on a home remedy hair care product would give me the power to stop you, really.\n",
            "$$\\'s a great way to get $$$ from our social network, but it's not the same. Make that know by following me on tumblr, tweeting @SCOT_PR\n",
            "sometimes peoples faces are in the made-for-TV-reality show business\n",
            "working towards my goal of getting a post-doc. trolling for extra credit\n",
            "The downfall of us all #adaytore!! Great job trying to find sponsors!\n",
            "I love my neighborhood!\n",
            "Everyone say cheers!\n",
            "Yes they can, and they're shooting them!\n",
            "THANKS HEAVENLY FATHER TO SEE ANOTHER DAY!\n",
            "I love talking to my daughters.\n",
            "Everybody say cheers!!\n",
            "That song still gives me chills!\n",
            "Just heard a mixtape featuring my guys, Attack the Block, and Ken Burns!\n",
            "and they're coming back.\n",
            "Enjoy!! Stay cool!!\n",
            "#word of mouth, mindset, and action.\n",
            "Being a free agent is going to be very interesting.\n",
            "#FAC chapter up. Was awesome!\n",
            "I'm learningin it.\n",
            "Good thing I took a shower last night. I am very dehydrated.\n",
            "after all, you die young.\n",
            "Not to be bored, not at all.\n",
            "privilege comes with it.\n",
            "Thanks. Thanks for the #FFthank you party.\n",
            "I mean, hello to= Sir Trevor, who is wise but kindhearted.\n",
            "I am very excited about the #Oscars right now. The first time I saw them was weirdly\n",
            "I have a horrible stomach.\n",
            "Not much, just a basic list of what I like and dislike about the various things I do.\n",
            "\n",
            "I'm not saying that everything I say is right. Sure, you can go back and repeat it a third time, but only if you know what you're saying.\n",
            "I just hate numbers.\n",
            "Well there might not be any positive reasons for doing so. There might be some benefits.\n",
            "But why am I watching this game?\n",
            "x\n",
            "iraga x\n",
            "thanks for getting back to us. you need to follow us when you can follow me\n",
            "I am so excited about the Coffee Ninja deal! We are auctioning off 5000 of these types of applications.\n",
            "I stinks of DSA...\n",
            "FAC can't possibly be a judge...\n",
            "Justened my 5th search for Francesco Crispio (pronounced \"poo-oh\" correct spelling?) ...on Twitter. Very cool!\n",
            "that bebbs. ;)\n",
            "The downfall of us all #ant\n",
            "======================================== SAMPLE 2 ========================================\n",
            "Commissioner Bailey: the officiating in this lockout-shortened, unadjusted officiating period should begin shortly. -- Tim\n",
            "We need to end this \"now is the wrong time\" stupidity to decide what kind of kids are *good* music and what isn't.\n",
            "Hey, could be better.\n",
            "Hahahahaha, thanks for the RT! :)\n",
            "Just heard about a iOS experiment that lets you make 360-degree 360-degree predictions about real-world events.\n",
            "cant wait till u rez all the way back to school?! been w/ Margaret at work all night\n",
            "is that a fire truck or are the police on pedestrians...?\n",
            " Continuing my fascination with the business of music.\n",
            "The perfume of avant-garde Vienna -- charming.\n",
            "keep going up access points for bird watching -- and you come up with interesting uses for landscapes\n",
            "lmfao! Your appointment?\n",
            "Surround yourself with people who are attractive, engaging and kind.\n",
            "like our page? please donating! you can help make your event more memorable!\n",
            "this is where I get crazy about things. it sometimes feels like I'm in some weird new country (:\n",
            "maybe we should've added \"p.S.. If you've been missing@ https://www.facebook.com/pokenientproducts/\n",
            "pod share is the first real easy plug-and-play way to buy nicotine.\n",
            "Ok whoever created the x (x) is hers.\n",
            "O.K.\n",
            "I'm not the biggest fan of addresses, do you?\n",
            "The forlorn monuments to the Second World War should be pretty empty without a monument to Parliament.\n",
            "holy shit ya someone DPzzzzz\n",
            "These are big days for Chris Raintree.\n",
            "Shakespeare should be in there.\n",
            "great quote: \"we have this great many conflicts but there is no peace without a document.\"-- J.R.R. Tolkien.\"\n",
            "I'm so glad RI is okay\n",
            "I guess so. Someone dose.\n",
            "Nobody Given My Name Is Any Good So No Addiction Or Murder. Everybody Gives One.\n",
            "Man I am in this moment alone.\n",
            "it just so happens that I have two moms\n",
            "Hi everyone, love is from the get-go; crazy not knowing what you're getting into, but I bet you wont take any\n",
            "how are you enjoying this newscast? What's next for NI?\n",
            "The great thing about high-fiving Charlie Sheen is that he can give you a dozen questions that you will not believe you asked him.\n",
            "go do something crazy and do it well.\n",
            "Would you consider an advertising opportunity with ? Not sure how, but would like to discuss.\n",
            "If they lose, we pay your tab! #Heat vs #Lakers. Gator ribs are the specials!\n",
            "thanks for reading! Have a great day!\n",
            "still haven't seen the #Cuts twitter feed, I entered tweet tag name into One Ratings System.\n",
            "MONEY IS ALL I KNOW\n",
            "#edutweet what are the best educational uses for twitter?\n",
            "a. Recommend a class from a book. bs nd I have what are you waiting for me to do something else :)\n",
            "Nice to see PHD trending!\n",
            "Simple City any day?\n",
            "oh shit I got an order for $60 go get it\n",
            "You guys are awesome:).\n",
            "Promote a good non sales Non Product line-up for I want it when it's ready. You assholes.\n",
            "Hey sage, your presentation was cool; I appreciate your perspective. What kind of professional is you?\n",
            "You nasty dog doesn't mind just calling you that.\n",
            "NOT IF YOU'RE WITH THE NY NY NY YAY\n",
            "cant wait till next year to see your film\n",
            "The Lakers and the Predators will be friday night and the hosts are the bidders. Discuss.\n",
            "Thanks for listening and have a great weekend!\n",
            "I am very excited about the Information Age thought police book, by Steven Covey (Not intended as a complete theory follows instructions).\n",
            "My network does a great job of avoiding the awkward moment and recognizing the point.\n",
            "I have a frosty beverage on the news and wonder what it is I'm drinking in it.\n",
            "Beyond the pelaing asshole bullshit...\n",
            "Congrats to Patti Groff, FR of California, who has taken on the enormous and challenging task of reaching all. !\n",
            "Maybe there's a cure for cancer, a place where there's hope and confidence, and not the optimism of chemotherapy.\n",
            "yeah I have a problem....\n",
            "relatively new phone cases reduce hospital admissions by 40%. :c\n",
            "MyTH: tow truck drivers. THX\n",
            " ballsfellow baggies!\n",
            "Just, I just got pregnant lookalng for your nation. you somet�it\n",
            "Tables provided by the University of Arkansas.\n",
            "BY THE DRY RACE BULK! race is my father baked in 1879\n",
            "as a teenager, I would\n",
            "======================================== SAMPLE 3 ========================================Traceback (most recent call last):\n",
            "  File \"src/generate_unconditional_samples.py\", line 79, in <module>\n",
            "    fire.Fire(sample_model)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/fire/core.py\", line 141, in Fire\n",
            "    component_trace = _Fire(component, args, parsed_flag_args, context, name)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/fire/core.py\", line 471, in _Fire\n",
            "    target=component.__name__)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/fire/core.py\", line 681, in _CallAndUpdateTrace\n",
            "    component = fn(*varargs, **kwargs)\n",
            "  File \"src/generate_unconditional_samples.py\", line 71, in sample_model\n",
            "    out = sess.run(output)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py\", line 971, in run\n",
            "    run_metadata_ptr)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py\", line 1194, in _run\n",
            "    feed_dict_tensor, options, run_metadata)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py\", line 1374, in _do_run\n",
            "    run_metadata)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py\", line 1380, in _do_call\n",
            "    return fn(*args)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py\", line 1364, in _run_fn\n",
            "    target_list, run_metadata)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py\", line 1458, in _call_tf_sessionrun\n",
            "    run_metadata)\n",
            "KeyboardInterrupt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VM1Hag-JL3Bt"
      },
      "source": [
        "To check flag descriptions, use:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sdxfye-SL66I"
      },
      "source": [
        "!python3 src/generate_unconditional_samples.py -- --help"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}