Jupyter Notebook - problems with dotnet-try under IIS

I keep encountering things when setting up Jupyter! After I got it running under IIS I went to bed then picked it up the next evening, tried to install the C# and F# kernels from dotnet try. Of course I hit two snags:

Issue 1. C# and F# Kernels aren't available at all

I installed it as usual by opening up Windows Terminal, source-ing my venv, installing dotnet-try globally and installing the dotnet try kernel. I suspected I was gonna encounter something a little odd - I wasn't sure if the dotnet try would respect the venv when I installed jupyter. Turns out the kernel got installed in a weird location (under AppData/Roaming rather than in my venv dir). I did a bit of experimenting and this was how I ended up resolving the issue. First I updated the PATH so that it included both the venv directory and the dotnet tools directory:

    <aspNetCore processPath="python.exe" 
                arguments="launch.py"
                stdoutLogEnabled="true" 
                stdoutLogFile="logs\stdout">
      <environmentVariables>
        <environmentVariable name="PYTHONPATH" value="." />
        <environmentVariable name="PATH" value="%PATH%;C:\Users\sean\.dotnet\tools;.\venv\Scripts" />
      </environmentVariables>    
    </aspNetCore>

Then I opened the terminal and ran the kernel install steps once more:

Once that was in place I could create a new notebook and run F# code:

Issue 2. Notebooks weren't saving

I was getting a little "Method Not Allowed" notification when I made some changes to my notebooks - so things like saving, renaming and deleting notebooks didn't work. The issue was that IIS was preventing HTTP verbs other than GET and POST (so PUT, PATCH and DELETE were getting blocked). The solution for me was to remove WebDAV handler and module - which again involved adding a couple of lines to my Web.config. It's maybe easier to just show the final Web.config to see everything together:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      <remove name="WebDAV" />
    </handlers>

    <modules>
      <remove name="WebDAVModule" />
    </modules>
    
    <aspNetCore processPath=".\venv\Scripts\python.exe" 
                arguments="launch.py"
                stdoutLogEnabled="true" 
                stdoutLogFile="logs\stdout">
      <environmentVariables>
        <environmentVariable name="PYTHONPATH" value="." />
        <environmentVariable name="PATH" value="%PATH%;C:\Users\sean\.dotnet\tools;.\venv\Scripts" />
      </environmentVariables>    
    </aspNetCore>
  </system.webServer>
</configuration>
So yet another hack - I'm starting to wonder if I'm using it in an unintended or unexpected way. I've put it all up on a git repo for anyone who wants to quickly get it up and running without thinking too hard.


Jupyter Notebook - Running under IIS on Windows

I use Jupyter Notebooks quite a lot for personal development, and to do this I set up an instance on a DigitalOcean droplet. However I also wanted to do something similar for some tasks at work which can access resources only available in our internal network. Since we use Windows that means I need to do something a little different. I could just use Anaconda to install and load it, but I find it a little clunky, I don't like how it tries to manage everything for you, I wanted a slightly nicer url than https://localhost:8000 and more importantly I wanted it to just be there without having to start anything up. So I set out to get Jupyter Notebook running on IIS using the ASP.NET Core Module V2 on http://notebook.local.

Make sure Python, IIS and the ASP.NET Core Hosting Bundle are installed. Open up a Terminal (I'm using bash, Powershell is probably file but I don't like it), create an new directory where our notebook application will be served, setup a venv there, and run the activation script, upgrade pip so that it doesn't complain at us and then finally install Jupyter Notebook:

$ cd ~/source/repos
$ mkdir notebook
$ python -m venv venv
$ source venv/Scripts/activate
$ pip install --upgrade pip
$ pip install notebook
To start with let's just run it on its own to check that it works:

$ jupyter notebook
[I 20:39:42.322 NotebookApp] The port 8888 is already in use, trying another port.
[I 20:39:42.328 NotebookApp] Serving notebooks from local directory: C:\Users\sean\source\external
[I 20:39:42.329 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 20:39:42.329 NotebookApp] http://localhost:8889/?token=e27aa8d7754e1bb078dfdf48e7c55032ac3551360389fd65
[I 20:39:42.329 NotebookApp]  or http://127.0.0.1:8889/?token=e27aa8d7754e1bb078dfdf48e7c55032ac3551360389fd65
[I 20:39:42.329 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 20:39:42.476 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///C:/Users/sean/AppData/Roaming/jupyter/runtime/nbserver-260-open.html
    Or copy and paste one of these URLs:
        http://localhost:8889/?token=e27aa8d7754e1bb078dfdf48e7c55032ac3551360389fd65
     or http://127.0.0.1:8889/?token=e27aa8d7754e1bb078dfdf48e7c55032ac3551360389fd65

So here you can see why I'm doing all this - http://localhost:8888 isn't quite as nice as https://notebook.local. Next let's create a logs folder where our stdout logging will live and notebooks folder where we'll keep our notebook files.

$ mkdir logs notebooks

Next we'll create a Python script called launch.py that will launch Jupyter:

import os, sys
from IPython.terminal.ipapp import launch_new_instance
from IPython.lib import passwd
import warnings

# set the password you want to use
notebook_password = "test"

sys.argv.append("notebook")
sys.argv.append("--IPKernelApp.pylab='inline'")
sys.argv.append("--NotebookApp.ip='*'")
sys.argv.append("--NotebookApp.port=" + os.environ["ASPNETCORE_PORT"])
sys.argv.append("--NotebookApp.open_browser=False")
sys.argv.append("--NotebookApp.notebook_dir=./notebooks")
sys.argv.append("--NotebookApp.password=" + passwd(notebook_password))

launch_new_instance()

This script is kind of interesting, but we'll go into it later once we've got everything up and running. Next we will need to create a Web.config file that'll be used by IIS and ANCMV2 to call this script with the Python executable and libraries from our venv. So let's create the following:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
    </handlers>
    <aspNetCore processPath=".\venv\Scripts\python.exe" 
                arguments="launch.py"
                stdoutLogEnabled="true" 
                stdoutLogFile="logs\stdout">
      <environmentVariables>
        <environmentVariable name="PYTHONPATH" value="." />
      </environmentVariables>    
    </aspNetCore>
  </system.webServer>
</configuration>

Next we'll set up the new Site in IIS - so open up IIS Manager (hit Start and start typing "Internet Information Services" - it'll appear in the suggestions) then:

  1. in the left-hand menu right click the top-level node, hit "Add Website..."
  2. in the dialog that appears call the site whatever you like
  3. in the "Physical Path" section click the "..." button, and choose the directory your notebooks live in
  4. in the "Binding" section type "notebook.local" in the "Host name" section

Now nearly everything's in place we just need to modify our local DNS to resolve "notebook.local" to 127.0.0.1 so add the following line to C:/Windows/System32/drivers/etc/hosts (you'll need to be an Administrator to do so):

127.0.0.1    notebook.local

So now we have a Website bound to "notebook.local" running under IIS, which is configured to use ASP.NET Core Runtime Module V2 to launch a Python script that will run Jupyter Notebook. Let's fire up a browser and navigate to http://notebook.local and enter the password test:

It loads nicely so let's just create a simple Python 3 notebook to confirm everything works end-to-end:

Excellent. Ok so let's rewind a little bit and ask the question: why do we use a special script instead of having our Web.config file launch jupyter notebook directly? Well that would be ideal, but a bit of a hack required. Notice in launch.py that we grab the ASPNETCORE_PORT environment variable and use that as our port? Well that contains the port IIS wants to communicate with our app in - so we need to hook Jupyter up to listen on that port. While it is possible to call jupyter notebook --port 8080 to specify the port we want to use, it's actually not possible to use the ASPNETCORE_PORT variable in our Web.config. These environment variables are not expanded in our arguments="..." attribute, (see issue #117 in the aspnet/AspNetCoreModule repo on GitHub) - so we need to run a script that grabs this from the Environment and sets it up. It's a bit hacky, but it gets us where we need to be.

Jupyter - problems with dotnet-try

Since .NET Try supports F# officially I wanted to switch over my Jupyter notebook instance to use that instead of IFsharp. But I ran into a couple of frustrating issues that I wanted to document in case anyone else hit them and didn't know how to debug the issue.

Issue 1. dotnet-try installs but isn't usable

I followed Scott Hanselman's instructions to install dotnet-try, but when I tried to execute dotnet try jupyter install it seemed as though dotnet-try wasn't installed at all:

$ dotnet tool install dotnet-try --global
You can invoke the tool using the following command: dotnet-try
Tool 'dotnet-try' (version '1.0.19553.4') was successfully installed.

$ dotnet try jupyter install Could not execute because the specified command or file was not found. Possible reasons for this include: * You misspelled a built-in dotnet command. * You intended to execute a .NET Core program, but dotnet-try does not exist. * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.

After a lot of head scratching I dug into the actual docs and learned that both .NET Core 3.0 and .NET Core 2.1 SDKs should be installed, while I only had .NET Core 3.1's SDK So after a quick sudo apt install dotnet-sdk-3.0 dotnet-sdk-2.1 I was successfully able to install the kernel and list it:

$ jupyter kernelspec list
Available kernels:
  .net-csharp    /home/notebook/.local/share/jupyter/kernels/.net-csharp
  .net-fsharp    /home/notebook/.local/share/jupyter/kernels/.net-fsharp
  mit-scheme     /usr/local/share/jupyter/kernels/mit-scheme
  python3        /usr/local/share/jupyter/kernels/python3

Issue 2. Jupyter can't run dotnet-try

However even though it was installed, each time I tried to create a new F# notebook Jupyter would give an error saying that it was unable to connect to the kernel. After taking a quick look at my logs I saw the same error as before!

Feb 09 13:19:11 aviemore jupyter[837]: [I 13:19:11.169 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
Feb 09 13:19:11 aviemore jupyter[837]: Could not execute because the specified command or file was not found.
Feb 09 13:19:11 aviemore jupyter[837]: Possible reasons for this include:
Feb 09 13:19:11 aviemore jupyter[837]: * You misspelled a built-in dotnet command.
Feb 09 13:19:11 aviemore jupyter[837]: * You intended to execute a .NET Core program, but dotnet-try does not exist.
Feb 09 13:19:11 aviemore jupyter[837]: * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
Feb 09 13:19:14 aviemore jupyter[837]: [I 13:19:14.180 NotebookApp] KernelRestarter: restarting kernel (2/5), new random ports
Feb 09 13:19:14 aviemore jupyter[837]: Could not execute because the specified command or file was not found.
Feb 09 13:19:14 aviemore jupyter[837]: Possible reasons for this include:
Feb 09 13:19:14 aviemore jupyter[837]: * You misspelled a built-in dotnet command.
Feb 09 13:19:14 aviemore jupyter[837]: * You intended to execute a .NET Core program, but dotnet-try does not exist.
Feb 09 13:19:14 aviemore jupyter[837]: * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH. 
I restarted the service, the server, checked and it took a while to realise the root cause. What happened was, even though dotnet-try was in PATH when I switched over to my jupyter use, it wasn't the case when I ran it via systemd. It seems that /etc/profile - which runs all the scripts in /etc/profile.d, one of which adds the ~/.dotnet/tools dir to PATH - is not used when systemd starts the service. I don't know the correct way to address this, but I wrote a quick script to setup PATH correctly and added an EnvironmentFile to the [Service] section my notebook.service file to use it:
EnvironmentFile=/home/jupyter/config/notebook-config
After I set this up and restarted notebook.service it was able to access dotnet-try and spin up the F# kernel correctly.

Fedora - Jupyter on a Linux remote using systemd

When you want to do some experimentation or put together a simple code-based presentation Jupyter notebooks are a powerful tool to have at your disposal. But if you use a number of devices over a few locations it can be useful to have a single instance hosted somewhere central (Linode, Digital Ocean, wherever) that you can access from any device wherever you are. There are a handful of ways that you can achieve this:

  1. log in to your remote machine, set Jupyter up and run jupyter notebook (perhaps in a tmux session) then log out - do this whenever your machine reboots
  2. as above but using an existing docker image
  3. spin up an Azure notebook
  4. ... or we could do something like #1 - but have it setup under a separate user and administered via a systemd service

All four of the above are fine for different reasons and use-cases but here I'll talk about how I put #4 together in a little Linode instance running Fedora 25 - it's relatively simple, you can control over the kernels installed, and it's another excuse to get a bit more in-depth with another Linux subsystem (systemd).

Requirements

All you need is a Linux system which uses systemd (Fedora 15.0 or newer, Debian 8.x or newer, Ubuntu 15.04 or newer, for example) which you have sudoer level access on, and Python 3.x. It's probably pretty straight-forward to set this up on systems using the SysV init but I won't cover them here.

Install and Set Up Jupyter 

First thing we need to do is install Jupyter and set up the user context which the Jupyter will be run under - which is a user called "jupyter":

$ sudo python3 -m ensurepip
$ sudo pip install jupyter
$ sudo useradd jupyter
$ sudo passwd jupyter

Next we should switch to the new jupyter user, create the directory our notebooks will live in and generate the Jupyter config we'll mess around with:

$ su - jupyter
$ mkdir notebooks
$ jupyter notebook --generate-config

The last command will create a new file ~/.jupyter/jupyter_notebook_config.py which we'll do a little messing around with shortly, but before this we'll set up a password 

$ python3
Python 3.5.2 (default, Sep 14 2016, 11:28:32) 
[GCC 6.2.1 20160901 (Red Hat 6.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from notebook.auth import passwd
>>> passwd() # below I enter "password123"
Enter password: 
Verify password: 
'sha1:2eff88aac285:385c87867bd18fe852ee1d56b1010d4beed96969'

This will be used to log in to the application when its running. Open up the ~/.jupyter/jupyter_notebook_config.py file in a text editor and add/modify the following lines (using the SHA1 hash returned by the above):

c.NotebookApp.port = 8888
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.password = 'sha1:2eff88aac285:385c87867bd18fe852ee1d56b1010d4beed96969'

Setting up a Jupyter systemd service

Now we want to create a new systemd service so we can make sure our Jupyter notebook runs on startup, handles logging nicely and has all the other bells-and-whistles afforded to us by systemd. This is surprisingly simple - we want to create a new file jupyter.service in /usr/lib/systemd/system - this will tie together our newly installed Jupyter software and our newly setup jupyter user - using your favourite text editor create it so it looks like the below:

$ sudo cat /usr/lib/systemd/system/jupyter.service
[Unit]
Description=Jupyter

[Service]
Type=simple
PIDFile=/var/run/jupyter.pid
ExecStart=/usr/bin/jupyter notebook --no-browser
WorkingDirectory=/home/jupyter/notebooks
User=jupyter
Group=jupyter

[Install]
WantedBy=multi-user.target%

Now all that's left to do is cross our fingers, enable our services, kick them off and browse to our remote box and login with our password:

$ sudo systemctl daemon-reload
$ sudo systemctl enable jupyter
$ sudo systemctl start jupyter

And if you want you can stop here - bookmark your http://www.xxx.yyy.zzz:port address and you're all set!

Conclusion

This was initially just an experiment - an excuse to test out my ability to put together a systemd .service file and do something more with a mostly-idle linux server sitting in a facility somewhere in Amsterdam. However I have found that I really like using this setup. When I was first shown Jupyter (née IPython) I was unimpressed and didn't see the point. However over the last few days I've been working through Project Euler problems again while teaching myself F# (using the IfSharp kernel) and I have found that it lends itself very well to my problem solving workflow on Project Euler.