Merge pull request #1 from Yolo1105/gh-pages

add new features in 1,2,3,5, and edit some attributes to nyu
nyuhpc · Oct 22, 2024 · 38d9018 · 38d9018
2 parents 6ed5fb9 + 8bdff10
commit 38d9018
Show file tree

Hide file tree

Showing 4 changed files with 200 additions and 31 deletions.
diff --git a/_episodes/01-connecting.md b/_episodes/01-connecting.md
@@ -10,12 +10,57 @@ objectives:
 - Connect to a remote HPC system.
 keypoints:
 - To connect to a remote HPC system using SSH and a password,
-  run `ssh [email protected]`.
+  run `ssh <NetID>@greene.hpc.nyu.edu`.
 - To connect to a remote HPC system using SSH and an SSH key,
-  run `ssh -i ~/.ssh/key_for_remote_computer [email protected]`.
+  run `ssh -i ~/.ssh/key_for_remote_computer <NetID>@greene.hpc.nyu.edu`.
 ---
 
+## Prerequisites
+To access the Greene HPC cluster, you must be connected to the NYU network. If you are physically on campus and connected via a wired connection in your office or through NYU's WiFi, you can directly SSH into the clusters without any additional steps. However, if you are off-campus or working remotely, connecting through the NYU VPN or using the gateway servers is required to establish a secure connection to the HPC systems.
+
+### Remote Connections with the NYU VPN & HPC Gateway Server
+If you are connecting from a remote location that is not on the NYU network (your home for example), you have two options: 
+
+1. VPN Option: set up your computer to use the NYU VPN. Once you've created a VPN connection, you can proceed as if you were connected to the NYU net.  
+
+2. Gateway Option: go through our gateway servers (example below). Gateways are designed to support only a very minimal set of commands and their only purpose is to let users connect HPC systems without needing to first connect to the VPN.
+
+### Log into the Greene Cluster
+NYU Campus: From within the NYU network, that is, from an on-campus location, or after you VPN inside NYU's network, you can login to the HPC clusters directly.
+
+Off-campus: The host name of Greene is 'greene.hpc.nyu.edu'. Logging in to Greene is the two-stage process. The HPC clusters (Greene) are not directly visible to the internet (outside the NYU Network). If you are outside NYU's Network (off-campus) you must first login to a bastion host named gw.hpc.nyu.edu.
+
+From within the NYU network, that is, from an on-campus location, or after you VPN inside NYU's network, you can log in to the HPC clusters directly. You do not need to log in to the bastion host.
+
+To log in to the HPC cluster (Greene), simply use:
+
+```bash
+ssh <NYUNetID>@greene.hpc.nyu.edu
+```
+
+For access from Windows stations using PuTTY, please [click here](#).
+
+To connect to VPN from Linux/MAC, please [click here](#).
+
+From an off-campus location (outside NYU-NET), logging in to the HPC clusters is a two-step process:
+
+1. First, log in to the bastion host, `gw.hpc.nyu.edu`. From a Mac or Linux workstation, this is a simple terminal command (replace `<NYUNetID>` with your NetID). Your password is the same password you use for NYU Home:
+
+```bash
+ssh <NYUNetID>@gw.hpc.nyu.edu
+```
+
+Windows users will need to use PuTTY, see [here](#) for instructions.
+
+2. Next, log in to the cluster. For Greene, this is done with:
+
+```bash
+ssh <NYUNetID>@greene.hpc.nyu.edu
+```
+
+
 ## Opening a Terminal
+Accessing the Greene HPC cluster is primarily done through the Command Line Interface (CLI). A CLI provides a text-based environment that allows users to manage files, run programs, and navigate directories via command input. On macOS, the built-in CLI tool is Terminal, while Windows 10 users can leverage the Windows Subsystem for Linux (WSL) for similar functionality. Additionally, a popular tool for connecting to Linux servers from Windows is PuTTY, a free SSH client.
 
 Connecting to an HPC system is most often done through a tool known as "SSH"
 (Secure SHell) and usually SSH is run through a terminal. So, to begin using an
@@ -37,6 +82,17 @@ then a quick search on the Internet for "how to open a terminal window in" with
 your particular Linux flavour appended to the end should quickly give you the
 directions you need.
 
+To connect to the gateway servers, simply open a terminal application and enter the following command:
+```bash
+ssh <NetID>@gw.hpc.nyu.edu
+```
+
+After typing in your password you will be logged in to the cluster. Once this connection is established, you can make one more hop and connect to one of the HPC clusters:
+```bash
+# this will connect you to Greene HPC cluster
+ssh <NetID>@greene.hpc.nyu.edu
+```
+
 ### Mac
 
 Macs have had a terminal built in since the first version of OS X since it is
@@ -48,6 +104,19 @@ of results (it will look like a tiny, black computer screen) and you will be
 presented with a terminal window. Alternatively, you can find Terminal under
 "Utilities" in the Applications menu.
 
+To connect to the gateway servers, simply open a terminal application and enter the following command:
+
+```bash
+ssh <NetID>@gw.hpc.nyu.edu
+```
+
+After typing in your password you will be logged in to the cluster. Once this connection is established, you can make one more hop and connect to one of the HPC clusters:
+
+```bash
+# this will connect you to Greene HPC cluster
+ssh <NetID>@greene.hpc.nyu.edu
+```
+
 ### Windows
 
 While Windows does have a command-line interface known as the "Command Prompt"
@@ -80,6 +149,8 @@ it is easy to add plug-ins or remove the software. The installer edition adds
 MobaXterm to your Windows installation and menu as any other program you might
 install. If you are not sure that you will continue to use MobaXterm in the
 future, the portable edition is likely the best choice for you.
+MobaKeyGen, see the MoabXterm 
+[documentation](https://mobaxterm.mobatek.net/documentation.html) 
 
 Download the version that you would like to use and install it as you would any
 other software on your Windows installation. Once the software is installed you
@@ -107,6 +178,8 @@ PuTTY is available for free download from
 Download the version that is correct for your operating system and install it 
 as you would other software on your Windows system. Once installed it will be 
 available through the start menu or similar.
+puttygen, see the Putty 
+[documentation](https://www.chiark.greenend.org.uk/~sgtatham/putty/docs.html)
 
 Running PuTTY will not initially produce a terminal but instead a window full
 of connection options. Putting the address of the remote system in the "Host
@@ -128,6 +201,47 @@ For those logging in with PuTTY it would likely be best to cover the terminal
 basics already mentioned above before moving on to navigating the remote
 system.
 
+### Open OnDemand (Web-based Graphical User Interface)
+Open OnDemand is an open source project funded by the National Science Foundation (NSF). Open OnDemand is designed to create easier access to users to interface with HPC systems. Originally developed by Ohio Supercomputer Center (OSC), used by many universities around the world, and now servicing the NYU Greene HPC cluster.
+
+Open OnDemand has a variety of convenient tools to manage files, access the command line, manage and monitor jobs, and launch interactive applications, such as Jupyter Notebooks, RStudio sessions, and even full Linux Desktops. 
+
+Features Include:
+
+Easy file management - upload and download files, view HTML and pictures without downloading
+
+Command-line shell access without any SSH client locally installed
+
+Job management and monitoring
+
+Full Linux desktop experience without X11
+
+Interactive Apps such as JupyterHub and RStudio without the need for port forwarding
+
+Open OnDemand (OOD) is accessible to all users with a valid NYU HPC account while on-campus network or through a VPN.
+
+To access OOD visit: https://ood.hpc.nyu.edu (VPN Required)
+
+#### Access the Shell
+
+Under the clusters menu you can select the Greene Shell Access option to access the Linux shell. No local SSH client is required.
+
+#### Interactive Applications
+
+GUI based applications are accessible without the need for port or X11 forwarding. Select the Interactive Apps menu, select the desired application, and submit the job based on required resources and options. 
+
+#### Troubleshooting Connections to Open OnDemand
+
+A common issue that can occur is receiving an error that the Open OnDemand page cannot be reached. Sometimes this can indicate that the service is down, but often this is an issue with the the local browser cache. You can test this by opening a private browser window and seeing if https://ood.hpc.nyu.edu will load. If it does, try deleting the cache for https://ood.hpc.nyu.edu in your browser history to resolve this issue.
+
+In Chrome, this can be done by navigating to this page in your settings:
+
+chrome://settings/content/all?searchSubpage=ood.hpc.nyu.edu&search=site+data
+
+The link above will automatically search for the Open OnDemand site data and cookies. You can then simply click on the trashcan icon to delete the site cache.
+
+Once done, try navigating again to https://ood.hpc.nyu.edu and the site should load. For other issues please email [email protected].
+
 ## Creating an SSH key
 
 SSH keys are an alternative method for authentication to obtain access to 
@@ -207,15 +321,6 @@ to provide.
 > [Secure Secure Shell](https://stribika.github.io/2015/01/04/secure-secure-shell.html).
 {: .callout}
 
-
-### Windows
-
-On Windows you can use
-- puttygen, see the Putty 
-[documentation](https://www.chiark.greenend.org.uk/~sgtatham/putty/docs.html)
-- MobaKeyGen, see the MoabXterm 
-[documentation](https://mobaxterm.mobatek.net/documentation.html) 
-
 ## Logging onto the system
 
 With all of this in mind, let's connect to a remote HPC system. In this
@@ -227,7 +332,7 @@ example computer, we will use SSH (if you are using PuTTY, see above).
 
 SSH allows us to connect to UNIX computers remotely, and use them as if they
 were our own. The general syntax of the connection command follows the format
-`ssh -i ~/.ssh/key_for_remote_computer [email protected]` 
+`ssh -i ~/.ssh/key_for_remote_computer <NetID>@greene.hpc.nyu.edu` 
 when using SSH keys and `ssh [email protected]` if only
 password access is available.  Let's attempt to connect to the HPC system
 now:

diff --git a/_episodes/02-navigation.md b/_episodes/02-navigation.md
@@ -23,6 +23,20 @@ happened yet, and we're not going to be able to do anything until we learn a
 few basic commands. By the end of this lesson, you will know how to "move
 around" the system and look at what's there.
 
+
+## System Architecture
+Files Systems for usage:
+
+The NYU HPC clusters have multiple file systems for user's files. Each file system is configured differently to serve a different purpose.
+
+| Space     | Environment Variable | Space Purpose                                                                 | Flushed                                         | Allocation (per user) |
+|-----------|----------------------|-------------------------------------------------------------------------------|-------------------------------------------------|-----------------------|
+| /home     | $HOME                 | Program development space; storing small files you want to keep long term, e.g. source code, scripts. | NO                                              | 20 GB                 |
+| /scratch  | $SCRATCH              | Computational workspace. Best suited to large, infrequent reads and writes.    | YES. Files not accessed for 60 days are deleted. | 5 TB                  |
+| /archive  | $ARCHIVE              | Long-term storage                                                             | NO                                              | 2 TB                  |
+| /beegfs   | $BEEGFS               | Computational workspace, workflows with many small files.                      | YES. Files not accessed for 60 days are deleted. | 2 TB                  |
+
+
 Right now, all we see is something that looks like this:
 
 ~~~
@@ -45,7 +59,7 @@ $ whoami
 ~~~
 {: .language-bash}
 ~~~
-yourUsername
+<NetID>
 ~~~
 {: .output}
 
@@ -61,7 +75,7 @@ stands for "print working directory"). ("Directory" is another word for
 "folder"). At any moment, our **current working directory** (where we are) is
 the directory that the computer assumes we want to run commands in unless we
 explicitly specify something else. Here, the computer's response is `{{
-site.workshop_host_homedir }}/yourUsername`, which is ``yourUsername`` **home
+site.workshop_host_homedir }}/<NetID>`, which is ``<NetID>`` **home
 directory**. Note that the location of your home directory may differ from
 system to system.
 
@@ -70,7 +84,7 @@ $ pwd
 ~~~
 {: .language-bash}
 ~~~
-{{ site.workshop_host_homedir }}/yourUsername
+{{ site.workshop_host_homedir }}/<NetID>
 ~~~
 {: .output}
 
@@ -135,7 +149,7 @@ $ pwd
 {: .output}
 
 What is the `~` character? When using the shell, `~` is a shortcut that
-represents `{{ site.workshop_host_homedir }}/yourUserName`.
+represents `{{ site.workshop_host_homedir }}/<NetID>`.
 
 Now that we know how to use `cd`, we can go anywhere. That's a lot of
 responsibility. What happens if we get "lost" and want to get back to where we
@@ -144,7 +158,7 @@ started?
 To go back to your home directory, the following three commands will work:
 
 ```
-$ cd {{ site.workshop_host_homedir }}/yourUserName
+$ cd {{ site.workshop_host_homedir }}/<NetID>
 $ cd ~
 $ cd
 ```
@@ -224,8 +238,8 @@ $ pwd
 {: .language-bash}
 
 ```
-{{ site.workshop_host_homedir }}/yourUserName/documents
-{{ site.workshop_host_homedir }}/yourUserName
+{{ site.workshop_host_homedir }}/<NetID>/documents
+{{ site.workshop_host_homedir }}/<NetID>
 ```
 {: .output}
 
@@ -259,7 +273,7 @@ $ ls -l
 ```
 {: .language-bash}
 ```
-drwxr-xr-x 2 yourUsername tc001 4096 Jan 14 17:31 documents
+drwxr-xr-x 2 <NetID> tc001 4096 Jan 14 17:31 documents
 ```
 {: .output}
 
@@ -277,15 +291,15 @@ $ ls -l -a
 ```
 {{ site.workshop_host_prompt }} ls -la
 total 36
-drwx--S--- 5 yourUsername tc001 4096 Nov 28 09:58 .
+drwx--S--- 5 <NetID> tc001 4096 Nov 28 09:58 .
 drwxr-x--- 3 root         tc001 4096 Nov 28 09:40 ..
--rw-r--r-- 1 yourUsername tc001   18 Dec  6  2016 .bash_logout
--rw-r--r-- 1 yourUsername tc001  193 Dec  6  2016 .bash_profile
--rw-r--r-- 1 yourUsername tc001  231 Dec  6  2016 .bashrc
-drwxr-sr-x 2 yourUsername tc001 4096 Nov 28 09:58 documents
--rw-r--r-- 1 yourUsername tc001  334 Mar  3  2017 .emacs
-drwxr-xr-x 4 yourUsername tc001 4096 Aug  2  2016 .mozilla
-drwx--S--- 2 yourUsername tc001 4096 Nov 28 09:58 .ssh
+-rw-r--r-- 1 <NetID> tc001   18 Dec  6  2016 .bash_logout
+-rw-r--r-- 1 <NetID> tc001  193 Dec  6  2016 .bash_profile
+-rw-r--r-- 1 <NetID> tc001  231 Dec  6  2016 .bashrc
+drwxr-sr-x 2 <NetID> tc001 4096 Nov 28 09:58 documents
+-rw-r--r-- 1 <NetID> tc001  334 Mar  3  2017 .emacs
+drwxr-xr-x 4 <NetID> tc001 4096 Aug  2  2016 .mozilla
+drwx--S--- 2 <NetID> tc001 4096 Nov 28 09:58 .ssh
 ```
 {: .output}
 
@@ -308,8 +322,8 @@ $ ls -l -a ~/documents
 {: .language-bash}
 
 ```
-drwxr-sr-x 2 yourUsername tc001 4096 Nov 28 09:58 .
-drwx--S--- 5 yourUsername tc001 4096 Nov 28 09:58 ..
+drwxr-sr-x 2 <NetID> tc001 4096 Nov 28 09:58 .
+drwx--S--- 5 <NetID> tc001 4096 Nov 28 09:58 ..
 ```
 {: .output}
 

diff --git a/_episodes/03-files.md b/_episodes/03-files.md
@@ -31,7 +31,6 @@ $ cd hpc-test
 ```
 {: .language-bash}
 
-
 ## Creating and Editing Text Files
 
 When working on an HPC system, we will frequently need to create or edit text
@@ -72,6 +71,13 @@ editor. Here are the shortcut keys for a few common actions:
 * <kbd>Ctrl</kbd>+<kbd>U</kbd> &mdash; paste the cut text line (or lines). This
   command can be repeated to paste the same text elsewhere.
 
+| Option       | Explanation             |
+|--------------|-------------------------|
+| Ctrl + O     | Save the changes         |
+| Ctrl + X     | Exit nano                |
+| Ctrl + K     | Cut single line          |
+| Ctrl + U     | Paste the text           |
+
 
 > ## Using `vim` as a text editor
 >
@@ -180,6 +186,12 @@ draft.txt
 ```
 {: .output}
 
+| Command                          | Explanation                                                                                                      |
+|----------------------------------|------------------------------------------------------------------------------------------------------------------|
+| mv dummy_file.txt test_file.txt   | Renames `dummy_file.txt` as `test_file.txt`.                                                                     |
+| mv subdir new_subdir              | Renames the directory "subdir" to a new directory "new_subdir".                                                  |
+
+
 **Renaming** &mdash; `draft.txt` isn't a very descriptive name. How do we go
 about changing it? It turns out that `mv` is also used to rename files and
 directories. Although this may not seem intuitive at first, think of it as
@@ -215,6 +227,12 @@ has two different uses that work in the same way as `mv`:
 - Copy to other directory (copied file retains original name): `cp file
   directory`
 
+| Command                      | Explanation                                                                                                                                      |
+|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| cp test_file1.txt test_file2.txt | Copies a duplicate copy of `test_file1.txt` with the new name `test_file2.txt`.                                                                  |
+| cp -r subdir subdir2          | Recursively copies the directory "subdir" to a new directory "subdir2". That is, a new directory "subdir2" is created, and each file and directory under "subdir" is replicated in "subdir2". |
+
+
 Let's try this out.
 
 ```
@@ -274,6 +292,15 @@ files
 ```
 {: .output}
 
+| Command                          | Explanation                                                                                                      |
+|----------------------------------|------------------------------------------------------------------------------------------------------------------|
+| rm dummy_file.txt                | Remove a file.                                                                                                   |
+| rm -i dummy_file.txt             | If you use `-i` you will be prompted for confirmation before each file is deleted.                               |
+| rm -f serious_file.txt           | Forcibly removes a file without asking, regardless of its permissions (provided you own the file).                |
+| rmdir subdir/                    | Removes "subdir" if it is already empty. Otherwise, the command fails.                                           |
+| rm -r subdir/                    | Recursively deletes the directory "subdir" and everything in it. Use it with care!                               |
+
+
 What happened? As it turns out, `rmdir` is unable to remove directories that
 have stuff in them. To delete a directory and everything inside it, we will use
 a special variant of `rm`, `rm -rf directory`. This is probably the scariest
@@ -440,3 +467,4 @@ out and see which ones you like best!
 
 Out of `cat`, `head`, `tail`, and `less`, which method of reading files is your
 favourite? Why?
+