Repository files navigation

Conjuring provides an easy way for students and teachers to use Python for acourse.

Technically, Conjuring can be viewed as a multi-user (jupyterhub) version ofrepo2docker (turngit repositories intoJupyter-enabledDocker Images)without requiring Python on the host. Don't worry if that sounds unintelligible.It's really cool. Trust us.

Specifically, Conjuring is designed withGeocomputing at KCL in mind;though is general enough to be used for any course.

We assume Conjuring will run on a single machine on a private network. Students(clients) on the same network will connect to the machine via their web browsersto access their Python workspace. The Python web interface is calledJupyter.

JupyterHub is used to manage multiple student accounts; each with their ownisolated workspace.

JupyterHub recommends "The Littlest JupyterHub (TLJH)" or "Zero to JupyterHub(Z2JH)" to install, neither of which are appropriate in our case. We thusmanually install it ourselves. It would be painful to provide installationinstructions for all operating systems. Instead, we provide a docker container(which can run on any supported OS). The container itself runs Ubuntu 18.04.

• A machine which is accessible by students(e.g. via Ethernet or even a WiFi hotspot)
• Docker CE (Community Edition)
• docker-compose

1. Download and if necessary customise (see below)
2. Rundocker-compose up --build -d

The Conjuring JupyterHub machine should be built and accessible via a browser onhttp://localhost:8989.

To shut down, rundocker-compose down.Student home directories will persist in thecustom/home/folder with the correct access permissions.

Configuration files are all found within thecustom/ directory.

• Define packages toapt install
  • Modifyapt.txt
• Define packages toconda install
  • Modifyenvironment.yml
• Define packages topip install
  • Modifyenvironment.yml orrequirements.txt
• Define alternative environments (kernels)
  • Createenvironment-<name>.yml files
• Define student usernames and passwords
  • Modifyusers.csv (first row is a header and is ignored)
• Define files which should be copied to each student's workspace
  • Add files tohome_default/
• Define read-only files which should be shared for all students
  • Add files toshared/
• Change the base image to something other thanubuntu:18.04
  • Modifybase.Dockerfile

A physical server can be configured to automatically start conjuring uponbootup, start a hotspot (or connect to a network),and monitor for external USB drives with additional configuration.

• Use theautoboot.sh script for this purpose
  • Runcrontab -e and add@reboot cd /path/to/conjuring && ./autoboot.sh
  • Runsudo apt-get install git openssh-server make sudo
  • Runsudo visudo and add the line%sudo ALL=(ALL) NOPASSWD:ALL
  • EnableSettings >Details >Users >Automatic Login

Notes:

• The path monitored for a configuration folder is/media/*/*/conjuring/custom
• The default network which is connected to is calledHotspot
  • This can be set up inSettings >Wi-Fi >(settings icon) >Turn On Wi-Fi Hotspot...,then running (in a terminal)nm-connection-editor to rename theconnection toHotspot and edit the password
  • Alternatively, choose a different default (e.g. external) network name bymodifyingautoboot.sh
  • The NUC itself and all clients need to be on the same network. This meansthat the NUC doesn't have to act as a Wi-Fi hotspot if there's apre-existing Wi-Fi network which everyone can connect to

If you are not familiar with docker, it may seem quite complicated.This overview (combined with theglossary below) might help.

docker-compose readsdocker-compose.yml (and if itexits,docker-compose.override.yml) in order to make the following happen:

1. docker downloads the latest version of theubuntu:18.04image
2. docker follows the instructions incustom/base.Dockerfile to (re)buildanimage calledconjuring/conjuring:base (based onubuntu:18.04)
3. docker follows the instructions in the first half ofDockerfile to (re)build animage calledconjuring/conjuring:core (based onconjuring/conjuring:base)
4. docker follows the instructions in the second half ofDockerfile to (re)build animage calledconjuring/conjuring:latest (based onconjuring/conjuring:core)
   • this references files from thesrc/ andcustom/directories to create users and Jupyter environments
5. docker creates acontainer calledconjuring(based onconjuring/conjuring:latest) which also does the following:
   • linkscustom/home/ toconjuring:/home/
   • linkscustom/home_default/ (read-only)
   • populates the container user home directories (conjuring:/home/*)
     • linkscustom/shared/ (read-only)
   • starts aJupyterHub server accessible on the host athttp://localhost:8989

All builds are "cached", i.e. unchanged lines from*Dockerfile will be notactually be re-run; saving time and bandwidth. Note that a line which referencesa changed file (fromsrc/ orcustom/) also counts as achanged line.

Rundocker system prune to clear unused cache.

• Base image: the starting point for building a Dockerimage
  • analogous to a clean OS (in this caseubuntu:18.04)
• Layer: a single (cached) build step
  • usually represented by a single line in aDockerfile(e.g.apt-get install git)
• Image: a sequence oflayers (applied on top of abase image)
  • analogous to a clean OS with things set up as specified incustom/(in this casetaggedconjuring/conjuring:latest)
• Container: a sandboxed workspace derived from animage
  • analogous to a running virtual machine (in this case namedconjuring)
  • easily stoppable, restartable, and disposable
  • can be thought of as end-user-createdlayers which would never beformally part of a redistributableimage
  • can share files, network connections, and devices with the host computer

Images arepulled orbuilt.Containers arecreated from them:

• Pull: typically refers to downloading animage from the internet (which someone elsebuilt)
  • usually only required when there is no source code available to allow forbuilding locally (e.g.ubuntu:18.04)
• Build: typically refers topulling abase image, thenbuilding all thelayers necessary to form animage
  • usually once-off
• Create: typically refers to making acontainer from animage
  • often recreated for a semi-clean slate - especially if data is shared with the host computer so that no data is lost on disposal

• Python: a programming language designed to be very human-readable

• Jupyter: an IDE (integrated development environment -- i.e. glorified texteditor) which runs in a web browser

• JupyterHub: a tool to manage multiple Jupyter servers for multiple users

• git: a code version control tool

• Docker: a virtual machine replacement tool

  • allows running e.g. isolated Ubuntu Linux JupyterHub containers on any host operating system

• GitHub: a website which hosts manygit repositories