example-KG-COVID-19-usage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running KG-COVID-19 pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The KG-COVID-19 pipeline can be run on the command line or via this notebook. The goal here is to run the pipeline end-to-end.\n",
    "\n",
    "We will also demonstrate some ways that you can use the KG downstream, and show some other features of the framework.\n",
    "\n",
    "**Note:** This notebook assumes that you have already installed the required dependencies for KG-COVID-19. For more information refer to [Installation instructions](https://github.com/Knowledge-Graph-Hub/kg-covid-19/wiki#installation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading all required datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we start with downloading all required datasets as listed in [download.yaml](../download.yaml)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "!python run.py download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform all required datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then transform all the datasets and generate the files `nodes.tsv` and `edges.tsv` for each dataset.\n",
    "\n",
    "The files are located in `data/transformed/SOURCE_NAME` where `SOURCE_NAME` is the name of the data source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "!python run.py transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge all datasets into a single graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we create a merged graph by reading in the individual nodes.tsv and edges.tsv and merging them.\n",
    "The merge process is driven by the [merge.yaml](../merge.yaml)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python run.py merge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The merged graph should be available in `data/merged/` folder.\n",
    "\n",
    "This pipeline generates a graph in KGX TSV format here:\n",
    "`data/merged/merged-kg.tar.gz`\n",
    "\n",
    "Prebuilt graphs are also available here:\n",
    "https://kg-hub.berkeleybop.io/kg-covid-19/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other tooling/functionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make training data for machine learning use case\n",
    "\n",
    "KG-COVID-19 contains tooling to produce training data for machine learning. Briefly, a training graph is produced with 80% (by default, override with `-t` parameter) of edges. 20% of edges are removed such that they do not create new components. These graphs are emitted as KGX TSV files in the `data/holdouts/` folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract the generated graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the generated graph from `data/merged/merged-kg.tar.gz`.\n",
    "\n",
    "> You can use the graph generated in the previous step OR download the latest graph from https://kg-hub.berkeleybop.io/kg-covid-19/current/kg-covid-19.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xvzf data/merged/merged-kg.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the training/holdout data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then generate a training/holdout data which will be used in subsequent steps for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# this might take 10 minutes or so\n",
    "!python run.py holdouts -e merged-kg_edges.tsv -n merged-kg_nodes.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get some stats on our training data. We're tightly integrated with ensmallen_graph, so we'll use that package to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from ensmallen_graph import EnsmallenGraph\n",
    "\n",
    "training = EnsmallenGraph.from_unsorted_csv(\n",
    "    edge_path=\"data/holdouts/pos_train_edges.tsv\",\n",
    "    sources_column=\"subject\",\n",
    "    destinations_column=\"object\",\n",
    "    directed=False,\n",
    "    edge_types_column='label',\n",
    "    default_edge_type='biolink:Association',\n",
    "    node_path=\"data/holdouts/pos_train_nodes.tsv\",\n",
    "    nodes_column='id',\n",
    "    default_node_type='biolink:NamedThing',\n",
    "    node_types_column='category',\n",
    "    ignore_duplicated_edges=True,\n",
    "    ignore_duplicated_nodes=True,    \n",
    ")\n",
    "\n",
    "training.report()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stats for the original graph, for comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from ensmallen_graph import EnsmallenGraph\n",
    "\n",
    "graph = EnsmallenGraph.from_unsorted_csv(\n",
    "    edge_path=\"merged-kg_edges.tsv\",\n",
    "    sources_column=\"subject\",\n",
    "    destinations_column=\"object\",\n",
    "    directed=False,\n",
    "    edge_types_column='edge_label',\n",
    "    default_edge_type='biolink:Association',\n",
    "    node_path=\"merged-kg_nodes.tsv\",\n",
    "    nodes_column='id',\n",
    "    default_node_type='biolink:NamedThing',\n",
    "    node_types_column='category',\n",
    "    ignore_duplicated_edges=True,\n",
    "    ignore_duplicated_nodes=True,    \n",
    ")\n",
    "\n",
    "graph.report()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making embeddings for a KG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate embeddings from the KG you've created above, take a look at notebooks available at https://github.com/monarch-initiative/embiggen/blob/master/notebooks/\n",
    "\n",
    "There are notebooks to make embeddings using:\n",
    "- [Skipgram](https://github.com/monarch-initiative/embiggen/blob/master/notebooks/Graph%20embedding%20using%20SkipGram.ipynb)\n",
    "- [CBOW](https://github.com/monarch-initiative/embiggen/blob/master/notebooks/Graph%20embedding%20using%20CBOW.ipynb)\n",
    "- [GloVe](https://github.com/monarch-initiative/embiggen/blob/master/notebooks/Graph%20embedding%20using%20GloVe.ipynb)\n",
    "\n",
    "These embeddings can then be used to train MLP, random forest, decision tree, and logistic regression classifiers using [this notebook](https://github.com/monarch-initiative/embiggen/blob/master/notebooks/Classical%20Link%20Prediction.ipynb).\n",
    "\n",
    "**Note:** Consider running the code in above notebooks on a server with GPUs in order to complete in a reasonable amount of time. Currently on a server with 2 V100 GPUs, the creation of embeddings and the training of classifiers both take on the order of 1 day each to complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use SPARQL queries to query our Blazegraph endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "KG-COVID-19 has tooling to query our Blazegraph endpoint using templated SPARQL queries and emit the results as a TSV file. Different SPARQL queries on our endpoint or other endpoints can be used by creating a new YAML file and specifying this file with the `-y` flag.\n",
    "\n",
    "The following is a simple query that retrieves a summary of the types of entities in the current KG-COVID-19 knowledge graph loaded on Blazegraph endpoint. These are counted as Biolink Model categories, which are high level entities such as genes, proteins, publications, etc. You can read more about the Biolink Model [here](https://biolink.github.io/biolink-model/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "!python run.py query -y queries/query-01-bl-cat-counts.yaml # or make a new YAML file and write your own query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "with open('data/queries/query-01-bl-cat-counts.tsv', newline='') as tsv:\n",
    "    read_tsv = csv.reader(tsv, delimiter=\"\\t\")\n",
    "    for row in read_tsv:\n",
    "      print(row)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}