Launching Jupyter notebooks

This page explains how to launch Jupyter Notebooks as a remote-server on compute node of Ibex and connect the Jupyter client in the local browser on your laptop/workstation.

Here are the step you need to do:

  1. The Jupyter server instance will be submitted to the SLURM scheduler as a jobscript requesting for resource allocation and launching Jupyter when the allocation is available for your job. Note that all the required modules should have been loaded in your jobscript before submitting.
  2. Reverse connect to the compute node(s) running the Jupyter server
  3. Launching the Jupyter client in your local browser
  4. Terminating/ending the session

Let’s first look at how to launch a Jupyter server on Ibex GPU node and connect to it. For example, the following is a jobscript requesting GPU resource on Ibex. Say our jobscript's name is jupyter_notebook_server.slurm:

 

#!/bin/bash -l 
#SBATCH --ntasks=1 
#SBATCH --cpus-per-task=4 
#SBATCH --gres=gpu:1 
#SBATCH --mem=32GB 
#SBATCH --time=01:00:00 
#SBATCH --partition=batch 

# Load environment which has Jupyter installed. It can be one of the following:
# - Machine Learning module installed on the system (module load machine_learning)
# - your own conda environment on Ibex
# - a singularity container with python environment (conda or otherwise)  

module load machine_learning 

# get tunneling info 
export XDG_RUNTIME_DIR="" node=$(hostname -s) 
user=$(whoami) 
submit_host=${SLURM_SUBMIT_HOST} 
port=10000
echo $node pinned to port $port 
# print tunneling instructions 

echo -e " 
To connect to the compute node ${node} on IBEX running your jupyter notebook server, you need to run following two commands in a terminal 1. 
Command to create ssh tunnel from you workstation/laptop to glogin: 

ssh -L ${port}:${node}:${port} ${user}@glogin.ibex.kaust.edu.sa 

Copy the link provided below by jupyter-server and replace the NODENAME with localhost before pasting it in your browser on your workstation/laptop " 

# Run Jupyter 
jupyter notebook --no-browser --port=${port} --port-retries=50 --ip=${node} 

In the jobscript above, machine_learing module is loaded. If you are interested in other software on Ibex and would like to use them in your notebooks, you need to load them in the jobscript. 

To submit the above jobscript (e.g jupyter_notebook_server.slurm) to the scheduler:

sbatch jupyter_notebook.slurm

Once the job starts, the slurm output file created in the directory you submitted the job from, will have the instructions on how to reverse connect.

The slurm output will look something like this:

gpu208-14 pinned to port 10000 

To connect to the compute node gpu208-14 on IBEX running your jupyter notebook server, you need to run following two commands in a terminal 

1. Command to create ssh tunnel from you workstation/laptop to glogin: 
ssh -L 10000:gpu208-14:10000 shaima0d@glogin.ibex.kaust.edu.sa 

Copy the link provided below by jupyter-server and replace the NODENAME with localhost before pasting it in your browser on your workstation/laptop 
.... 
To access the notebook, open this file in a browser: file:///home/shaima0d/.local/share/jupyter/runtime/nbserver-28992-open.html 
Or copy and paste one of these 
URLs: http://gpu208-14:10000/?token=0406639be400d9e7e23898d57d27572ca65898243a053355 
or http://127.0.0.1:10000/?token=0406639be400d9e7e23898d57d27572ca65898243a053355

 

  • Open a new terminal on your local machine and copy paste the ssh tunnel command from the slurm-xxxxx.out
ssh -L 10000:gpu208-14:10000 your-username@glogin.ibex.kaust.edu.sa

NOTE: Before running the ssh command, confirm that the port Jupyter as started is the port you had requested in the beginning. This can be done by inspecting the last line http://127.0.0.1:10000/?token... where 10000 is the port on the remote node.

  • This has now created an SSH tunnel between the compute node your Jupyter server is launched on Ibex, and your local machine on IP address localhost and port 10000. 
  • Now we are ready to launch our Jupyter client. Copy the line (the last line in the slurm output file) and paste it in your browser address bar:
http://127.0.0.1:10000/?token=0406639be400d9e7e23898d57d27572ca65898243a053355