Jupyter Notebook for Literate Computing for Reproducible Infrastructure

Jupyter Notebook server which bundles a set of tools for Literate Computing for Reproducible Infrastructure. This bundle shows how to implement robust and reliable operational procedure rather than explorative procedure within Jupyter’s GUI.

The goals for Literate Computing tools are:

• Preventing miss-operation; once a cell has been executed, it “freezes” against unintended execution. Also you can “lock” cells for unintended modification.
• Adding a perspective into a notebook; markdown’s hierarchy is collapsible as document according to your focus. However, embedded cells underneath are represented as dots and run through with a click as a routine procedure.
• Enables collaborative annotation onto Jupyter Notebooks, where your ideas and communications stay in context and grow.

What it Gives You

• Jupyter Notebook 6.0.x
• Python3 and bash kernel with LC_wrapper https://github.com/NII-cloud-operation/Jupyter-LC_wrapper
• Extensions
  • multi_outputs https://github.com/NII-cloud-operation/Jupyter-multi_outputs
  • run_through https://github.com/NII-cloud-operation/Jupyter-LC_run_through
  • nblineage https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
  • index https://github.com/NII-cloud-operation/Jupyter-LC_index
  • sidestickies https://github.com/NII-cloud-operation/sidestickies
  • nbsearch https://github.com/NII-cloud-operation/nbsearch

Basic Use

You can start the Notebook server on port 8888 with the following command.

docker run -it --rm -p 8888:8888 niicloudoperation/notebook

You can login the Notebook server with the authentication token in the startup message.

If you would like to use NBSearch, use Solr container like the following:

# Launch Solr
git clone https://github.com/NII-cloud-operation/nbsearch /tmp/nbsearch
docker run -d --name nbsearch-solr -v /tmp/nbsearch:/tmp/nbsearch --rm solr:8 \
    bash -c "precreate-core jupyter-notebook /tmp/nbsearch/solr/jupyter-notebook/ && \
        precreate-core jupyter-cell /tmp/nbsearch/solr/jupyter-cell/ && \
        solr-foreground"

# Launch MinIO
docker run -d --rm -e MINIO_ACCESS_KEY=nbsearchak -e MINIO_SECRET_KEY=nbsearchsk \
    --name nbsearch-minio minio/minio:latest server /data --compat

# Launch Notebook
docker run -it --rm --link nbsearch-solr:solr --link nbsearch-minio:minio \
    -e NBSEARCHDB_SOLR_BASE_URL=http://solr:8983 -e NBSEARCHDB_S3_ENDPOINT_URL=http://minio:9000 \
    -e NBSEARCHDB_S3_ACCESS_KEY=nbsearchak -e NBSEARCHDB_S3_SECRET_KEY=nbsearchsk \
    -p 8888:8888 niicloudoperation/notebook

To enable the NBSearch extension, refer 03_Notebookの検索.ipynb in the container.

Docker Options

You may customize the execution of Docker container and the Notebook server contained with the following optional arguments.

• -e lc_wrapper_force=on - Force summarizing mode of lc_wrapper (via env)
• -e TZ=JST-9 - Specify the container timezone
• -e PASSWORD=MY_UNBREAKABLE_PASS - Set a initial password
• -v /some/host/folder/for/work:/home/jovyan - Mounts the host directory to the working directory in the container

Using sidestickies

You can use sidestickies by either of the following steps.

Using Scrapbox https://scrapbox.io/:

• Add -e SIDESTICKIES_SCRAPBOX_PROJECT_ID=value -e SIDESTICKIES_SCRAPBOX_COOKIE_CONNECT_SID=value to docker options - Specify Scrapbox account to sidestickies.
• Enable the sidestickies extension via the Nbextensions tab. Note: you need to enable both "Sidestickies for file tree" and "Sidestickies for notebook" nbextensions.

Using ep_weave(Etherpad) https://github.com/NII-cloud-operation/ep_weave :

• Add -e SIDESTICKIES_EP_WEAVE_URL=http://ep_weave:9001 -e SIDESTICKIES_EP_WEAVE_API_KEY=YOUR_ETHERPAD_APIKEY to docker options.
• Enable the sidestickies extension via the Nbextensions tab. Note: you need to enable both "Sidestickies for file tree" and "Sidestickies for notebook" nbextensions.

Using NBSearch

You can use NBSearch by enabling the NBSearch extension via the Nbextensions tab. After the page of Jupyter is reloaded, the NBSearch tab appears on the page of Jupyter.

NBSearch uses Solr and S3 Compatible Storage to store and search notebooks. You will be able to use NBSearch once you set up your Solr and S3 compatible storage configurations, set the following environment variables to the options.

• -e NBSEARCHDB_SOLR_BASE_URL=your_solr_url - The base URL of Solr(default: http://localhost:8983)
• -e NBSEARCHDB_SOLR_BASIC_AUTH_USERNAME=your_solr_username, -e NBSEARCHDB_SOLR_BASIC_AUTH_PASSWORD=your_solr_password - The username and password for Solr(if needed)
• -e NBSEARCHDB_S3_ENDPOINT_URL=your_database_name - The URL of S3(default: http://localhost:9000)
• -e NBSEARCHDB_S3_ACCESS_KEY=your_s3_access_key, -e NBSEARCHDB_S3_SECRET_KEY=your_s3_secret_key - The access key and secret key for S3(required)
• -e NBSEARCHDB_S3_REGION_NAME=your_s3_region_name - The region name of S3(if needed)
• -e NBSEARCHDB_S3_BUCKET_NAME=your_s3_bucket_name - The bucket on S3(required)
• -e NBSEARCHDB_SOLR_NOTEBOOK=your_solr_notebook_core - The core for notebooks on Solr(default: jupyter-notebook)
• -e NBSEARCHDB_SOLR_CELL=your_solr_cell_core - The core for cells on Solr(default: jupyter-cell)
• -e NBSEARCHDB_BASE_DIR=your_notebook_home_dir - Notebook directory to be searchable(default: /home/$NB_USER)
• -e NBSEARCHDB_MY_SERVER_URL=your_notebook_server_url - URL of my server, used to identify the notebooks on this server(default: http://localhost:8888/)
• -e NBSEARCHDB_AUTO_UPDATE=1 - Launch lsyncd process to update the index of Solr when local files are updated automatically
• -e NBSEARCHDB_UPDATE_INDEX_OPT - Options for the update-index NBSearch command invoked by the lsyncd process

Using NBWhisper

You can use NBWhisper by enabling the NBWhisper extension via the Nbextensions tab.

To use NBWhisper, you need an API token of the SkyWay (WebRTC) service. It can be specified from environment variables as follows. (For more information, see https://github.com/NII-cloud-operation/nbwhisper/ .)

• -e NBWHISPER_SKYWAY_API_TOKEN - An api token for SkyWay. This nbextension is not compatible with new SkyWay service, but old one. You can get an api token from your old SkyWay admin page. https://console-webrtc-free.ecl.ntt.com/users/login
• -e NBWHISPER_ROOM_MODE_FOR_WAITING_ROOM - Room mode for waiting room. Input "sfu" or "mesh" as waiting room mode you want to use.
• -e NBWHISPER_ROOM_MODE_FOR_TALKING_ROOM - Room mode for talking room. Input "sfu" or "mesh" as talking room mode you want to use.