Collaborative Calculation and Data Science

facebook

github

google

twitter

sagemath

depaul

I have just released a Jupyter kernel pool optimization for all projects and the use of Jupyter notebooks. I wrote and deleted this 3 or 4 times before getting something that works robustly (I hope). The final version should not break anything or negatively impact any functionality, as it falls back to not using a pool in every subtle case where something could go wrong (e.g., environment variable customization changes, etc.).

<img width="169" alt="image" src="https://user-images.githubusercontent.com/1276278/232261594-f24853e4-fa57-4d40-ba8b-b9bb90a3f6fd.png">

- To benefit from this optimization, you need to restart your project (or start a new one). Refreshing your browser will not make any difference.
- The very first time you start your project and open a notebook, there will be no difference except for one extra kernel starting in the background (that's the pool). Moreover, a file ~/.config/cocalc-jupyter-pool will be created with the parameters of that kernel.
- If you then open _another_ notebook with the same kernel, running code should start MUCH more quickly. Additionally, "restart and run all" in a notebook should be significantly faster than before (e.g., less than a second instead of 10 seconds, say, for Sage!).
- If your project stops and you start it up again later, your first use of a kernel should be MUCH faster, assuming you're using the same kernel as before and you haven't changed any custom environment variables. Under the hood, whatever was stored in ~/.config/cocalc-jupyter-pool will be started next time.

That's it! One interesting thing I needed was [code to generate code](https://github.com/sagemathinc/cocalc/blob/master/src/packages/util/jupyter-api/setenv-commands.ts) for each language that would set custom environment variables on the fly. I [had something](https://github.com/sagemathinc/cocalc/blob/master/src/packages/util/jupyter-api/chdir-commands.ts) that would generate code to change directories, and I pointed GPT-3.5 at that code and said, "rewrite this to instead generate code for custom environment variables..." and it worked perfectly on the first try.