How To: Set Up a Pluto.jl Server for Teaching Purposes

Oct. 24, 2020

I want to share what my colleague Dennis Ogiermann and me learnt while setting up a server that I’m going to use for a teaching adventure with Julia. I’ll teach exercises in the course “Basics of FEM” for mechanical engineers.

The problems

The mechanical engineering students have no or little experience with programming (and very likely no experience with a terminal)
There is no time to teach them in depth programming, terminal handling, etc.
FEM can be done with a pocket calculator but it is very annoying and inefficient, in fact, it is so annoying that Konrad Zuse invented a computer to get rid of similar computations
Theoretical and practical aspects need to be covered in the exercises
Because of the corona pandemic, everything has to happen digitally

Goals

We want to interactively teach them a computational method (FEM) and hence, need a computational environment
The computational environment needs to be simple, such that people who never coded before can use it immediately
There should be the possibility to integrate LaTeX formulas, YouTube videos and plots in the environment
The environment needs to be easily accessible by a browser, so that the technical barrier of entry for students is low

Approach

Therefore, we decided to use Pluto.jl because of its reactivity and nice user interface. However, there is no equivalent to JupyterHub yet for Pluto.jl, which can control multiple users on one server. So, we decided to set up a JupyterHub server with a Pluto.jl proxy.

This tutorial was done with an arch based system. Installation commands via pacman need to be translated to your preferred linux distribution. If you don’t use systemd, you also have to exchange the systemd commands.

Static IP setup

Your teaching server will stay in the same network and thus, will have the same network adresses. In order to access your server over time with the same IP, you need to set up a static ip address. This wiki article and links therein explain how to set up a static ip address for your arch based system. Basically, what you have to do is:

Create device to address mapping /etc/conf.d/net-conf-enp25s0

address= # some ip address 
netmask= # a netmask
broadcast= # some broadcast
gateway= # some gateway

Create network service file /etc/systemd/system/network@.service

[Unit]
Description=Network startup (%i)
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/conf.d/net-conf-%i
ExecStart=/sbin/ip link set dev %i up
ExecStart=/sbin/ip addr add ${address}/${netmask} broadcast ${broadcast} dev %i
ExecStart=/sbin/ip route add default via ${gateway}
ExecStop=/sbin/ip addr flush dev %i
ExecStop=/sbin/ip link set dev %i down

[Install]
WantedBy=multi-user.target

Bind the server IP at boot time by typing

systemctl enable network@eth0.service

Now, reboot.

Setting up environment variables needed by the server

Enforce correct environment variables by editing /etc/profile , appending the following variables. Never skip any setting with the JULIA_DEPOT_PATH variable. In this and following sections it is crucial to set it as proposed. Otherwise, Julia packages will be installed in /home/username/.julia.

[...]
export JULIA_DEPOT_PATH="/usr/share/juliapackages/:$JULIA_DEPOT_PATH"
export JUPYTER_DATA_DIR="/usr/share/jupyter/"
export PATH=$PATH:/opt/julia/bin/

Install Julia to /opt/julia which should be done by downloading the 64 bit archive from https://julialang.org/downloads/ and unzipping it into the folder

Install JupyterHub to /opt/jupyterhub as described in https://jupyterhub.readthedocs.io/en/stable/installation-guide-hard.html

sudo python3 -m venv /opt/jupyterhub/
sudo /opt/jupyterhub/bin/pip3 install wheel
sudo /opt/jupyterhub/bin/pip3 install jupyterhub jupyterlab
sudo /opt/jupyterhub/bin/pip3 install ipywidgets
sudo npm install -g configurable-http-proxy
sudo mkdir -p /opt/jupyterhub/etc/jupyterhub/
cd /opt/jupyterhub/etc/jupyterhub/
sudo /opt/jupyterhub/bin/jupyterhub --generate-config

Configure your JupyterHub Server

Customize the JupyterHub configuration /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py. In our first try a working config looked like this

## The public facing port of the proxy.
#  
#  This is the port on which the proxy will listen. This is the only port through
#  which JupyterHub should be accessed by users.
#  
#  .. deprecated: 0.9
#      Use JupyterHub.bind_url
c.JupyterHub.port = 'YOUR PORT AS AN INTEGER VARIABLE'

## The URL the single-user server should start in.
#  
#  `{username}` will be expanded to the user's username
#  
#  Example uses:
#  
#  - You can set `notebook_dir` to `/` and `default_url` to `/tree/home/{username}` to allow people to
#    navigate the whole filesystem from their notebook server, but still start in their home directory.
#  - Start with `/notebooks` instead of `/tree` if `default_url` points to a notebook instead of a directory.
#  - You can set this to `/lab` to have JupyterLab start by default, rather than Jupyter Notebook.
c.Spawner.default_url = '/lab'

## Whitelist of environment variables for the single-user server to inherit from
#  the JupyterHub process.
#  
#  This whitelist is used to ensure that sensitive information in the JupyterHub
#  process's environment (such as `CONFIGPROXY_AUTH_TOKEN`) is not passed to the
#  single-user server's process.
c.Spawner.env_keep = ['PATH', 'JUPYTERLAB_DIR', 'MEM_LIMIT', 'PYTHONPATH', 'JULIA_DEPOT_PATH', 'CONDA_ROOT', 'CONDA_DEFAULT_ENV', 'VIRTUAL_ENV', 'LANG', 'LC_ALL']

## Maximum number of bytes a single-user notebook server is allowed to use.
#  
#  Allows the following suffixes:
#    - K -> Kilobytes
#    - M -> Megabytes
#    - G -> Gigabytes
#    - T -> Terabytes
#  
#  If the single user server tries to allocate more memory than this, it will
#  fail. There is no guarantee that the single-user notebook server will be able
#  to allocate this much memory - only that it can not allocate more than this.
#  
#  **This is a configuration setting. Your spawner must implement support for the
#  limit to work.** The default spawner, `LocalProcessSpawner`, does **not**
#  implement this support. A custom spawner **must** add support for this setting
#  for it to be enforced.
c.Spawner.mem_limit = 1000000000

## Set of users that will have admin rights on this JupyterHub.
#  
#  Admin users have extra privileges:
#   - Use the admin panel to see list of users logged in
#   - Add / remove users in some authenticators
#   - Restart / halt the hub
#   - Start / stop users' single-user servers
#   - Can access each individual users' single-user server (if configured)
#  
#  Admin access should be treated the same way root access is.
#  
#  Defaults to an empty set, in which case no user has admin access.
c.Authenticator.admin_users = {"YOUR USER NAME AS STRING"}

Note that the memory limiter won’t work ootb, as stated in the comment lines above the c.Spawner.mem_limit definition. Next, set up JupyterHub systemd integration by creating the file /opt/jupyterhub/etc/systemd/jupyterhub.service with

[Unit]
Description=JupyterHub
After=syslog.target network.target

[Service]
User=root
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/jupyterhub/bin:/opt/julia/bin"
Environment="JULIA_DEPOT_PATH=/usr/share/juliapackages"
Environment="MEM_LIMIT=1000000000"
ExecStart=/opt/jupyterhub/bin/python3 -m jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

[Install]
WantedBy=multi-user.target

followed by

sudo ln -s /opt/jupyterhub/etc/systemd/jupyterhub.service /etc/systemd/system/jupyterhub.service
sudo systemctl daemon-reload
sudo systemctl enable jupyterhub.service
sudo systemctl start jupyterhub.service

Now, check if the startup was without errors by

sudo systemctl status jupyterhub.service

At this point reboot again. Finally we have to install the Julia Kernel. We have to use “-E” with sudo to properly transfer the environment variables.

sudo -E julia
]add IJulia
]add Pluto
using IJulia
using Pluto

At this point the JupyterHub environment is ready and can be accessed by simply typing the ip and port into the browser.

Next we can make the Julia package depot accessible for non-root accounts by setting the file mods

sudo groupadd juliausers
sudo chown -R root:juliausers /usr/share/juliapackages
sudo chmod -R 740 /usr/share/juliapackages

This way admin accounts can still add new packages and users in the juliausers group can use them.

Install Pluto.jl as Proxy

Up until now, we only have JupyterHub and the Julia kernel available when we log in to the server. In order to use Pluto.jl, we need to install a JupyterLab labextension and also a proxy service that will redirect the individual user to a Pluto.jl server that is started and hosted at the remote machine.

sudo /opt/jupyterhub/bin/jupyter labextension install @jupyterlab/server-proxy
sudo /opt/jupyterhub/bin/pip3 install git+https://github.com/fonsp/pluto-on-jupyterlab.git

Now restart the jupyterhub service via

sudo systemctl restart jupyterhub.service

and the installation is done.

Finally we are able to add student accounts via

sudo useradd -aG juliausers studentXX
sudo passwd passwd studentXX

Et voila

Pluto Server

Acknowledgement

Dennis Ogiermann for supporting me with his IT 🧙 skills
Pluto.jl Dev’s for Pluto and the reference repo

If you have any questions, don’t hesitate to ask me work@maximiliankoehler.de