Jupyter Notebook - problems with dotnet-try under IIS

I keep encountering things when setting up Jupyter! After I got it running under IIS I went to bed then picked it up the next evening, tried to install the C# and F# kernels from dotnet try. Of course I hit two snags:

Issue 1. C# and F# Kernels aren't available at all

I installed it as usual by opening up Windows Terminal, source-ing my venv, installing dotnet-try globally and installing the dotnet try kernel. I suspected I was gonna encounter something a little odd - I wasn't sure if the dotnet try would respect the venv when I installed jupyter. Turns out the kernel got installed in a weird location (under AppData/Roaming rather than in my venv dir). I did a bit of experimenting and this was how I ended up resolving the issue. First I updated the PATH so that it included both the venv directory and the dotnet tools directory:

    <aspNetCore processPath="python.exe" 
                arguments="launch.py"
                stdoutLogEnabled="true" 
                stdoutLogFile="logs\stdout">
      <environmentVariables>
        <environmentVariable name="PYTHONPATH" value="." />
        <environmentVariable name="PATH" value="%PATH%;C:\Users\sean\.dotnet\tools;.\venv\Scripts" />
      </environmentVariables>    
    </aspNetCore>

Then I opened the terminal and ran the kernel install steps once more:

Once that was in place I could create a new notebook and run F# code:

Issue 2. Notebooks weren't saving

I was getting a little "Method Not Allowed" notification when I made some changes to my notebooks - so things like saving, renaming and deleting notebooks didn't work. The issue was that IIS was preventing HTTP verbs other than GET and POST (so PUT, PATCH and DELETE were getting blocked). The solution for me was to remove WebDAV handler and module - which again involved adding a couple of lines to my Web.config. It's maybe easier to just show the final Web.config to see everything together:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      <remove name="WebDAV" />
    </handlers>

    <modules>
      <remove name="WebDAVModule" />
    </modules>
    
    <aspNetCore processPath=".\venv\Scripts\python.exe" 
                arguments="launch.py"
                stdoutLogEnabled="true" 
                stdoutLogFile="logs\stdout">
      <environmentVariables>
        <environmentVariable name="PYTHONPATH" value="." />
        <environmentVariable name="PATH" value="%PATH%;C:\Users\sean\.dotnet\tools;.\venv\Scripts" />
      </environmentVariables>    
    </aspNetCore>
  </system.webServer>
</configuration>
So yet another hack - I'm starting to wonder if I'm using it in an unintended or unexpected way. I've put it all up on a git repo for anyone who wants to quickly get it up and running without thinking too hard.


Jupyter - problems with dotnet-try

Since .NET Try supports F# officially I wanted to switch over my Jupyter notebook instance to use that instead of IFsharp. But I ran into a couple of frustrating issues that I wanted to document in case anyone else hit them and didn't know how to debug the issue.

Issue 1. dotnet-try installs but isn't usable

I followed Scott Hanselman's instructions to install dotnet-try, but when I tried to execute dotnet try jupyter install it seemed as though dotnet-try wasn't installed at all:

$ dotnet tool install dotnet-try --global
You can invoke the tool using the following command: dotnet-try
Tool 'dotnet-try' (version '1.0.19553.4') was successfully installed.

$ dotnet try jupyter install Could not execute because the specified command or file was not found. Possible reasons for this include: * You misspelled a built-in dotnet command. * You intended to execute a .NET Core program, but dotnet-try does not exist. * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.

After a lot of head scratching I dug into the actual docs and learned that both .NET Core 3.0 and .NET Core 2.1 SDKs should be installed, while I only had .NET Core 3.1's SDK So after a quick sudo apt install dotnet-sdk-3.0 dotnet-sdk-2.1 I was successfully able to install the kernel and list it:

$ jupyter kernelspec list
Available kernels:
  .net-csharp    /home/notebook/.local/share/jupyter/kernels/.net-csharp
  .net-fsharp    /home/notebook/.local/share/jupyter/kernels/.net-fsharp
  mit-scheme     /usr/local/share/jupyter/kernels/mit-scheme
  python3        /usr/local/share/jupyter/kernels/python3

Issue 2. Jupyter can't run dotnet-try

However even though it was installed, each time I tried to create a new F# notebook Jupyter would give an error saying that it was unable to connect to the kernel. After taking a quick look at my logs I saw the same error as before!

Feb 09 13:19:11 aviemore jupyter[837]: [I 13:19:11.169 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
Feb 09 13:19:11 aviemore jupyter[837]: Could not execute because the specified command or file was not found.
Feb 09 13:19:11 aviemore jupyter[837]: Possible reasons for this include:
Feb 09 13:19:11 aviemore jupyter[837]: * You misspelled a built-in dotnet command.
Feb 09 13:19:11 aviemore jupyter[837]: * You intended to execute a .NET Core program, but dotnet-try does not exist.
Feb 09 13:19:11 aviemore jupyter[837]: * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
Feb 09 13:19:14 aviemore jupyter[837]: [I 13:19:14.180 NotebookApp] KernelRestarter: restarting kernel (2/5), new random ports
Feb 09 13:19:14 aviemore jupyter[837]: Could not execute because the specified command or file was not found.
Feb 09 13:19:14 aviemore jupyter[837]: Possible reasons for this include:
Feb 09 13:19:14 aviemore jupyter[837]: * You misspelled a built-in dotnet command.
Feb 09 13:19:14 aviemore jupyter[837]: * You intended to execute a .NET Core program, but dotnet-try does not exist.
Feb 09 13:19:14 aviemore jupyter[837]: * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH. 
I restarted the service, the server, checked and it took a while to realise the root cause. What happened was, even though dotnet-try was in PATH when I switched over to my jupyter use, it wasn't the case when I ran it via systemd. It seems that /etc/profile - which runs all the scripts in /etc/profile.d, one of which adds the ~/.dotnet/tools dir to PATH - is not used when systemd starts the service. I don't know the correct way to address this, but I wrote a quick script to setup PATH correctly and added an EnvironmentFile to the [Service] section my notebook.service file to use it:
EnvironmentFile=/home/jupyter/config/notebook-config
After I set this up and restarted notebook.service it was able to access dotnet-try and spin up the F# kernel correctly.

F# - Polymorphic parameter overflow

I was mucking around in F# over lunch and started thinking about how it assigns names to type variables (see "Automatic Generalization" of the Type Inference page of the F# docs for more info). By way of introduction, let's create a function foo that takes a single parameter without an explicitly defined type and see what its signature looks like:
> let foo x = x;;
val foo : x:'a -> 'a
So the parameter x is assigned the type variable 'a - which makes sense, the first unknown type gets named after the first letter of the alphabet. And of course it follows that if we have a function with two parameters the second type is called ...
> let foo x y = (x,y);;
val foo : x:'a -> y:'b -> 'a * 'b
... 'b! Ok now what happens when we've got a function with a ridiculous amount of parameters? We'll run out of lower-case letters eventually. When that happens do start using upper-case letters? Non-ASCII? Something else? I quickly hacked together a python script to generate functions with arbitrary parameters, created one with 40 and pasted it into the F# REPL and ...
> let foo x0 x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 x20 x21 x22 x23 x24 x25 x26 x27 x28 x29 x30 x31 x32 x33 x34 x35 x36 x37 x38 x39 = (x0,x1,x2,x- 3,x4,x5,x6,x7,x8,x9,x10,x11,x12,x13,x14,x15,x16,x17,x18,x19,x20,x21,x22,x23,x24,x25,x26,x27,x28,x29,x30,x31,x32,x33,x34,x35,x36,x37,x38,x39);;
val foo :
  x0:'a ->
    x1:'b ->
      x2:'c ->
        x3:'d ->
          x4:'e ->
            x5:'f ->
              x6:'g ->
                x7:'h ->
                  x8:'i ->
                    x9:'j ->
                      x10:'k ->
                        x11:'l ->
                          x12:'m ->
                            x13:'n ->
                              x14:'o ->
                                x15:'p ->
                                  x16:'q ->
                                    x17:'r ->
                                      x18:'s ->
                                        x19:'t ->
                                          x20:'a1 ->
                                            x21:'a2 ->
                                              x22:'a3 ->
                                                x23:'a4 ->
                                                  x24:'a5 ->
                                                    x25:'a6 ->
                                                      x26:'a7 ->
                                                        x27:'a8 ->
                                                          x28:'a9 ->
                                                            x29:'a10 ->
                                                              x30:'a11 ->
                                                                x31:'a12 ->
                                                                  x32:'a13 ->
                                                                    x33:'a14 ->
                                                                      x34:'a15 ->
                                                                        x35:'a16 ->
                                                                          x36:'a17 ->
                                                                            x37:'a18 ->
                                                                              x38:'a19 ->
                                                                                x39:'a20 ->
                                                                                  'a *
                                                                                  'b *
                                                                                  'c *
                                                                                  'd *
                                                                                  'e *
                                                                                  'f *
                                                                                  'g *
                                                                                  'h *
                                                                                  'i *
                                                                                  'j *
                                                                                  'k *
                                                                                  'l *
                                                                                  'm *
                                                                                  'n *
                                                                                  'o *
                                                                                  'p *
                                                                                  'q *
                                                                                  'r *
                                                                                  's *
                                                                                  't *
                                                                                  'a1 *
                                                                                  'a2 *
                                                                                  'a3 *
                                                                                  'a4 *
                                                                                  'a5 *
                                                                                  'a6 *
                                                                                  'a7 *
                                                                                  'a8 *
                                                                                  'a9 *
                                                                                  'a10 *
                                                                                  'a11 *
                                                                                  'a12 *
                                                                                  'a13 *
                                                                                  'a14 *
                                                                                  'a15 *
                                                                                  'a16 *
                                                                                  'a17 *
                                                                                  'a18 *
                                                                                  'a19 *
                                                                                  'a20
So 'a up to 't ... but then numbered 'as after that. I've no idea why it stops at 't, then just counts up from 'a. Interestingly this is the case with F# under Mono on Linux and under the official MS runtime on VS2017 on Windows. OCaml, which F# is heavily influenced by, does not exhibit this behaviour - it refers to the unknown types as a..z, a1..z1, a2...z2, etc.

That's reasonbaly interesting and all, but since I had already written a script to generate these F# files I figured I'd keep trying with more and more parameters to see what happens. After adding a hundred parameters at a time I soon hit an interesting warning between 400 and 500 params:

Warning: line too long, ignoring some characters
- 
It turned out we hit some line-length restriction in either the parser or the readline equivalent that fsi.exe uses. A bit of experimentation later I found that when the function had 429 parameters it parsed just fine ... but 430 parameters caused the warning.

429 parameters: 4093 characters
430 parameters: 4104 characters

Minor sidebar - I was going to just say we hit a restriction where 4096 bytes is the max line length, but obviously 1 character isn't necessarily 1 byte if we're using some Unicode representation, and a nice way to check that we're using Unicode is to punch in identifiers that would definitely be in different encodings in an ASCII representation - I used Georgian and Czech characters:
> let სამი = 3;;
val სამი : int = 3

> let čtyři = 4;;
val čtyři : int = 4

> სამი + čtyři;;
val it : int = 7
When I changed my program to start each parameter with the Georgian character "ა" (i.e. let bar ა0 ა1... etc) which cannot be represented in a single byte in UTF-8 we hit the error after only 162 parameters. So it's not a character limit, but a buffer of bytes that we filled - and the size of the buffer is ... uh 4096 bytes.

Since I wanted to pull this thread until I hit some logical conclusion I tweaked my program to split the insanely long lines so they wouldn't hit this error and continued to add more parameters. The next issue I encountered was after trying a function with a couple thousand parameters, where I encountered the following error:
  x1328,
  ^^^^^

/home/sean/stdin(6636,1): error FS0039: The value or constructor 'x1328' is not defined. Maybe you want one of the following:
   x132
   x138
   x1238
   x128
   x1321
So it seems we've bumped up into another limit - somehow 1327 parameters is ok but F# loses track of the 1328th one, thinking it doesn't exist. Is this a bug? Probably, there should maybe be a more helpful error message. Is it important? Probably not, if your code contains a function with upwards of a thousand parameters then this is the least of your problems.

IIS - Annoying error with .NET Core

I deployed a .NET Core application recently into an environment which previously had only run .NET Framework apps, but got a tricky HTTP Error 500.19 - Internal Server Error suggesting there was a problem with my Web.config:


What was also peculiar was that when I tried to enable stdout logging using the IIS UI I got an error "There was an error while performing this operation" with blank "Details" and "Error" but with the "Filename" set to the path to my Web.config. This was true for any of the configuration sections:


However after a lot of head scratching and googling turned up nothing my colleague realised that this environment was missing the Windows Server Hosting bundle, available from the .NET Core downloads section under the "Other Windows Downloads" section. Once this was installed everything worked as expected.