Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place.
Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place.
Path: blob/master/src/README.md
Views: 646
How to Build and Run CoCalc
Updated: Jan 2023
CoCalc is a pretty large and complicated project, and it will only work with the current standard LTS release of node.js ( at least 16.8.x) and a recent version of pnpm.
Node.js and NPM Version Requirements:
You must be using Node version 16.8.x or newer. CoCalc will definitely NOT work with any older version! In a CoCalc.com project, you can put this in
~/.bashrc
to get a valid node version:
Alternatively, if you are already using Node Version Manager, you can simply run the following command to use a version of Node guaranteed to work with this project:
Note that nvm install
is only required the first time you run this command or when CoCalc's Node version changes.
Make sure to install the newest version of pnpm as well; one way to do that is as follows:
Alternatively, if you do not wish to install pnpm
globally, you can run npm install
to install it as a dev dependency.
Python virtual environment
Some features of CoCalc (e.g., file creation) require local Python modules to be installed. To create a Python virtual environment from which to run these modules, run (from the src
directory):
To activate the virtual environment, run
To install required dependencies, run
You must have your virtual environment activated when running the CoCalc Hub (via pnpm hub
)! If, on the other hand, you prefer that development packages be installed globally, you can jump directly to the above pip install
command outside the context of a virtual environment.
Initial Build
Launch the install and build for doing development:
Note: If you installed pnpm
locally (instead of globally), simply run npm run
in place of pnpm
to execute these commands via NPM run scripts.
This will do pnpm install
for all packages, and also build the typescript/coffeescript, and anything else into a dist directory for each module. Once pnpm make
finishes successfully, you can start using CoCalc by starting the database and the backend hub in two separate terminals.
The hub will send minimal logging to stdout, and the rest to data/logs/log
.
To get real-time updates from, e.g., the packages/server
directory, you'll also need to run
If you need to do a production build instead:
The main (only?) difference is that static and next webpack builds are created in production mode, which takes much longer.
Starting All Over
If necessary, you can delete all the node_modules
and dist
directories in all packages and start over as follows:
Doing Development
The code of CoCalc is in NPM packages in the src/packages/
subdirectory. To do development you need to ensure each of the following two services are running, as explained above:
PostgreSQL database -- a postgres instance started via
pnpm database
Hub -- a nodejs instance started via
pnpm hub
Optionally, you may also need to type pnpm tsc
in packages that you're editing to watch for changes, compile using Typescript and show an errors.
1. More about Starting the Database
CoCalc stores all of its data in a PostgreSQL database. Start your PostreSQL database server as follows:
The database runs in the foreground and logs basic information. It serves via a "Unix domain socket", i.e., something that looks like a file. If you set the environment variables PGUSER
and PGHOST
as follows, you can use psql
to connect to the database:
You can also just type pnpm psql
:
NOTE: As of Jan 2023, CoCalc should fully work with any version of PostgreSQL from version 10.x onward. However, obviously at some point we will stop supporting PostgreSQL v 10.
2. More about Starting the Hub
The Hub is CoCalc's backend node.js server. You can start it from its package directory as follows:
That will ensure the latest version of the hub Typescript and Coffeescript gets compiled, and start a new hub running in the foreground logging what is happening to the console and also logging to files in cocalc/src/data/logs/hub
. Hit Control+C to terminate this server. If you change any code in packages/hub
, you have to stop the hub, then start it again as above in order for the changes to take effect.
The hub itself is running two copies of webpack along with two separate "Hot Module Replacement" servers, etc. One is for the /static
endpoint (see packages/static and packages/frontend) and the other is for the nextjs server (see packages/next).
3. Building only what has changed
The command pnpm build (or build-dev)
, when run from the src directory, caches the fact that there was a successful build by touching a file src/packages/package-name/.successful_build
. This caching only does anything if you explicitly use the pnpm build
command from the src/ directory, and is ignored when directly building in a subdirectory. You can do pnpm build --exclude=static
periodically to rebuild precisely what needs to be built, except what is built using webpack (e.g., via pnpm static
as explained above):
This is useful if you pull in a git branch or switch to a different git branch, and have no idea which packages have changed. That said, it's always much safer to just do the following instead of relying on this:
In particular, this will make sure exactly the right packages are installed and everything is built before doing pnpm static
and pnpm hub
. The simplest way to do this is
which installs exactly the right packages, and builds the code.
Other
Environment Variables
See packages/backend/data.ts
. In particular, you can set BASE_PATH, DATA, PGHOST, PGDATA, PROJECTS, SECRETS to override the defaults. Data is stored in cocalc/src/data/
by default.
File System Build Caching
There are two types of file system build caching. These greatly improve the time to compile typescript or start webpack between runs. However, in rare cases bugs may lead to weird broken behavior. Here's where the caches are, so you know how to clear them to check if this is the source of trouble. As of now, I'm not aware of any bugs in file system caching.
In the
dist/
subdirectory of a package, there's a filetsconfig.tsbuildinfo
that caches incremental typescript builds, so runningtsc
is much faster. This is enabled by settingincremental: true
intsconfig.json
. I've never actually seen a case where caching of this file caused a problem (those typescript developers are careful).Webpack caches its builds in
/tmp/webpack
. This is configured inpackages/static/webpack.config.js
, and we use/tmp
since random access file system performance is critical for this large GB+ cache -- otherwise, it's almost slower than no cache. (I also benchmarked tsc, and it works fine on a potentially slow local file system.) I did sees bugs with this cache when I had some useless antd tree shaking plugin enabled, but I have never seen any problems with it since I got rid of that.
Creating an admin user
It is handy to have a user with admin rights in your dev cocalc server. With the database running you can make a [email protected]
an admin as follows:
Admin users have an extra tab inside the main cocalc app labeled "Admin" where they can configure many aspects of the server, search for users, etc.
IDE file autosave settings
During development, Next.js and Webpack will eagerly compile your code changes and hot-reload the page you're working with. If your IDE automatically saves files very quickly, this will be very intensive, causing a lot of CPU and disk usage. Hence we recommend to tame or disable autosaving files.
Regarding VS Code, the relevant settings can be found by searching for "autosave" – or to be more precise, this is recommended:
Packages on NPMJS.com (DEPRECATED)
There's some @cocalc/
packages at NPMJS.com. However, we're no longer using them in any way, and don't plan to publish anything new unless there is a compelling use case.