Salvus

Distributed Scalable Online Mathematical Software

William Stein

University of Washington

Screencast of talk (part 1 of 2)

Screencast of talk (part 2 of 2)

Project Goals

• Make Sage (and other math software) very robustly available over the internet to a large number of users
• Generate substantial revenue to support core sage development: working with UW C4C on license agreement, payment system, etc. They get 20% of revenue and the other 80% goes in the Sage Foundation account.
• If successful, will spin off as a separate company in a few years.
• If successful, will hire elite developers to work fulltime on the core research-level mathematical algorithms in Sage.

Related Projects

• Programming, but no math
  • http://c9.io -- "Your code anywhere, anytime... Write, run, and debug your code with our powerful and flexible cloud IDE."
  • http://www.heroku.com/ -- "cloud application platform"
  • https://www.appsoma.com/ -- "Code & run in the cloud"
• Math, but no programming or scalable computation
  • http://www.wolframalpha.com/
• Sage/Ipython
  • http://sagenb.org -- online notebooks
  • http://aleph.sagemath.org -- single embeddable sage snippet
  • ipython notebook
• Cloud hosting: can do everything, but expensive and complicated...
  • http://aws.amazon.com/ -- Amazon Web Services
  • http://www.linode.com/ -- "Deploy and Manage Linux Virtual Servers in the Linode Cloud."

sage-support mailing list a few days ago

On Wed, Sep 26, 2012 at 9:54 AM, Chris Seberino  wrote:
> I have a Xen Ubuntu Linux 1 Gb RAM virtual machine  from a provider 
> called Linode.
>

Linode charges $39.95/month for your 1GB machine.  For a 4GB machine,
which would be more likely to work well, you would have to pay
$159.95/month.

My plan with http://salv.us/ -- which I'm working hard on right now --
is to buy a bunch of hardware (mostly done), host it at UW, write much
better more efficient software to make Sage ("highly") available
through it, and provide enough guaranteed resources for a class like
you have... for *substantially* less than $159.95/month, since the
monthly hosting and network fees I have to pay at UW are super-cheap.

Stay tuned...

 -- William

Longterm Programing Goals (part 1)

• Simple starter page: where you can type Sage code in a box (no login required) with tab completion and interact support. If you login, then you can easily browse/search/share all past computations you've ever done. Support for mobile browsers from the start.
• Projects: A project is defined by a git repository, which salvus stores. Many users can share the same project. It can be linked to github (or google code, or bitbucket, etc.) Thus a project provides a directory tree of files, and there is a revision history.
• A command line: alternative way to browse a project, and will be like the bash command line. Can type "sage" to run Sage here, and "git" to run git, providing the full power of git.

Longterm Programing Goals (part 2)

• Editing documents: Salvus will also provide browser-based editing of several file types, starting with .sws files (for compatibility with sage), but also supporting many other file types, and ways of working with documents and data. Multiple users can simultaneously edit a document; changes are broadcast and updated on all clients. Also, editors for .py files, .pyx files, csv files, presentations, etc. Special support for the Sage library as a project.
• Publishing: this is like Heroku or Google App Engine; it will support making a persistent Sage-based application available to many users, either with a specific URL, or embedded in another web page, like with aleph.sagemath.org, ee Rob Beezer's page. Paying users can allocate dedicated hosting resources.
• Chat/cross-user messaging: useful for teaching, online classes, tutorials, etc.

Approach to Salvus

• Take my time and do things right: With the sage notebook, I wrote something that looks and feels a lot like the current version in a sleep-deprived month.
• Service-based architecture: Numerous independent components running on dozens of machines communicating securely via messages over a VPN.
• NO SPOF! absolutely no single points of failure anywhere in the design of salvus.
• Concise: research existing solutions to make new code shorter and clearer, instead of just implementing everything from scratch.

Physical Hardware and Remote VM's

• 4 3Ghz 16-core 64GB RAM, 1TB disk, 1U servers in Padelford basement
• 1 8-core 32GB RAM machine in my office running OS X
• 3 3Ghz 16-core 64GB RAM, 6TB disk, 1U servers at UW tower
• 2 servers at servedby.net (a cloud hosting started by UW undergrad)
• Google Compute Engine: "8 single CPU instances with 100GB of persistent disk space"
• Google App Engine: I received $60,000 via a research grant program -- not sure how to use it yet!

Fixed monthly costs

• Current monthly fixed cost for hardware above: $60/month (for UW tower hosting)
• Amazon (in multiple ways) also told me they would give me free credits, but didn't deliver. So far, too big, confused and confusing.
• This should be enough resources to support several thousand paying customers (and many free customers). Do the math.
• Google and Servedby resources are free right now, because they want my longterm business

Revenue Model

• Competitive advantage: better software and hardware resources than other players. Better understanding of end user needs in this space, because of Sage.
• Pricing: probably around $10/month per user for the first tier of non-free account, which competes with linnode + sagenb ($150/month?).
• Goal for March 2013: Have 1,500 paying users at $10/month, which is what the above hardware should be able to support, done right. $1500*10/month = $15,000/month

Other Longterm Revenue

• Purchase higher levels of dedicated resources, e.g., 30 students in a class.
• Site licenses with service level and special features for an entire university or company.
• Software license for the complete Salvus implementation, so a company with strong security restrictions can deploy their own internal server.
• Pay more for access to commercial software via Salvus (e.g., Mathematica, Maple, Matlab, Magma).

The Salvus Stack

• Infrastructure: private cloud (libvirt + kvm); public cloud
• VPN: tinc
• Database: Apache Cassandra
• SSL: stunnel
• Load balancing: HAproxy
• Static HTTP: Nginx
• Dynamic HTTP: Tornado
• Math/compute: forking tcp-based Sage server
• Management: Paramiko + much new code

Architecture Diagram

   Client    Client    Client   Client   Client  Client
     /|\
      |       (sockjs)
   Stunnels ... (https://salv.us)
      |     
      |
     \|/ 
 HAProxy Load Balancers  ...  Nginx static servers ...
 /|\       /|\      /|\      /|\
  |         |        |        |           [tinc vpn]
  |https1.1 |        |        |                                     
 \|/       \|/      \|/      \|/                                      
Tornado  Tornado  Tornado  Tornado    <------------------>   Cassandra   ...
           /|\      /|\                                         /|\
            |        |        -----------------------------------|
   ---------|        |        |                                         
   |                 |        |
   |                 |        |
  \|/               \|/      \|/
 SageServer   SageServer  SageServer   SageServer ...

Client/server communication

• SockJS: websocket connection between browser and tornado, when supported; otherwise, falls back to other long-polling approaches
• TCP + SSL: standard UNIX Socket connection for non-browser clients, e.g, command line.
• HTTP: old-fashioned POST for simple HTTP clients

Virtualization: libvirt + kvm

• Virtualization: absolutely essential due to allowing users to remotely execute arbitrary code (ok, with sagenb.org nothing is virtualized, but that is just a ticking bomb, which keeps exploding in my face!!)
• Public clouds: many companies (famously Amazon.com) now rent virtual machines to anybody
• Private clouds: on your own hardware, start up many virtual machines based on a template with certain parameters, start services running on them. When the machine shuts down, it leaves no trace (except persistent sparse disk images).
• The only persistent data: in all of Salvus is in the distributed database. This will sit in a disk partition that is encrypted using the Linux ecryptfs filesystem and a passphrase that I type in (exactly once from my management interface) when launching the database.

Private Cloud Distributions

• OpenStack, CloudStack, OpenNebula, Ganeti, etc. -- war of the private cloud software distributions.
• I spent quite a bit of time investigating all this, and it's not useful yet, at least for what I'm doing -- basically a lot of companies and people are making noise about implementing something open source kind of like what Amazon and VMware already did, but with a lot of the truly hard real work being doing by libvirt. (This will likely be useful for something in a year or two, and now if you pay companies money, but now it is worse than useless...)
• The real work is done by libvirt, kvm (or xen). Evidently, Redhat put a ton of serious money/development into supporting the creation of high quality virtualization support -- KVM -- that is built into the Linux kernel now. This is what I'm using directly... in the end, to do everything I want nicely, I wrote a few hundred lines of code (vm.py) and learned about how to get around bugs.

VPN: Motivation

• VPN = Virtual Private Network, in public, using encryption
• By using a VPN, Salvus can easily combine cloud computers, my own computers, etc., and don't have to worry about ip addresses changing.
• We don't have to worry about securing communication between the different services or vm's in salvus.
• For a long time, I was worrying about things such as implementing SSL + TCP communication between the Tornado web server and backend Sage process; I threw all that code away, and instead just use a VPN and have a nice clean class A address space as a bonus.

VPN I chose: tinc

• I'm NOT using openVPN, even though it is enormously popular... because all traffic goes through a master VPN server, a single point of failure!
• tinc is "a Virtual Private Network (VPN) daemon that uses tunnelling and encryption to create a secure private network between hosts on the Internet." Key features:
  • Public-facing machines on the VPN also acts as servers, so clients can connect to VPN as long as at least one public-facing machine is up.
  • It's P2P -- traffic does not all go through one master node, which could be very inefficient.
  • All traffic is encrypted.
  • Easy/sensible; to add a machine to the VPN, you just generate a public and private key, then add the public key to at least one public facing machine.
  • Small C program that builds in seconds.

Database: the memory of Salvus

• PostgreSQL is very nice, as is MySQL and even SQLite. I really wanted to use one of them, and tried very, very hard to do so (and wrote a lot of code that I threw away). Then I started thinking seriously about the amount of data I will have to manage with the number of users I plan to have in the long run. And I read books about how to scale website using mysql. What they do is very difficult and ugly, e.g., sharding -- a bunch of different databases based on usernames. (I also read a lot about SQLalchemy, used PostgreSQL a lot, etc.)
• Then the Padelford server room blew up for the n-th time, and I firmly decided on no SPOF's as an Axiom.

Database: noSQL

• NO SPOFS!: Relational databases like PostgreSQL become even more difficult to use with this constraint. You have to have a master database, at least one other slave databases replicating it, and some sort of automatic failover that is activated when your software decides the master is not working (which is not an easy thing to decide!).
• "I'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents." http://news.ycombinator.com/item?id=4524340
• One of the motivations for the recent wave of "NoSQL databases" is creation of databases that have no single point of failure. They scale up by just adding more small machines, and one can configure how redundant they are. I had learned a lot about MongoDB (a noSQL database) a few years ago, and after getting LMFDB.org to use it a lot (and using it for my own database work), I think it sucks.... for me at least. "Does everyone hate MongoDB?"

Database: Apache Cassandra

• NoSQL: I read about and tried out every noSQL database I could find. (I even wrote one: http://code.google.com/p/nosqlite/) For my particular use case, by far the best one is Apache Cassandra (cassandra.apache.org).
• Cassandra is an open source database with copyright owned by Apache, and also commercial support from a company. It is written in Java, which I'm no fan of, but still I really like Cassandra. Unlike many options, Cassandra is pretty heavily used by serious companies. E.g, Netflix makes enormous use of Cassandra, as does Disney. Cassandra was originally written by Facebook, who open sourced it, and it gained traction in the open source world.
• Cassandra seems to be the only database that satisfied my constraints (mature, distributed, scalable, support for multiple data centers). I've been using it for a while now, and I'm impressed. I started with version 1.1 using CQL3, which looks like a restricted subset of SQL, with NO JOINS or relations at all between tables!

SSL: stunnel

• One of the main motivations for rewriting the sage notebook "back in the day" to use twisted was to support SSL, i.e., encrypted communcation between the user's browser and sagenb.org. This worked, but we never properly signed the cert, so users got a terrifying warning when they visited sagenb.org, and it was deamed easier to just remove the security than deal with the cert stuff.
• Can easily make any webapp using SSL encryption without having to touch the code that defines that website -- just use the program stunnel!
• Getting a properly signed SSL certificate setup isn't impossible. (I even bought my cert for $12.99 by some trick... though UW actually offers free sigs.)
• With salvus we run a couple of stunnel's and put them all in DNS, so the browser chooses one. When an stunnel fails, the browser will just load a different one. No SPOF.

Load balancing: haproxy

• Haproxy is pretty much the canonical open source load balancer program.
• Haproxy monitors a bunch of web servers (for salvus, that means nginx servers and tornado servers), makes sure they are working, and balances incoming connections between them.
• Static content requests go to nginx.
• Dynamic websocket connections go to tornado servers.

Static HTTP: nginx

• nginx seems to be the canonical choice for serving a huge number of static webpages efficiently. It was easy to setup and use. It's lightweight and easy to build from source.
• nginx is single threaded (but fixed number of multiple server processes). This is absolutely key to its success in serving huge numbers of requests per second. It's far more efficient than the old apache approach, which would spawn a thread or process per request.

Dynamic HTTP: Tornado

• http://www.tornadoweb.org/ -- an open source version of the scalable, non-blocking web server
• Tornado is basically a clean self-contained nicely coded webserver written in Python that works a lot like twisted (i.e., it is asynchronous), but is much simpler and more targeted.
• "While Tornado is similar to existing Web-frameworks in Python (Django, Google's webapp, web.py), it focuses on speed and handling large amounts of simultaneous traffic."
• Very complete and self-contained and self-consistent, e.g., has its own template system built in
• It was written by some company called Friendfeed founded by a bunch of ex-Googlers that Facebook bought; Facebook then open sourced it.

More about Tornado

• Ipython's notebook also uses Tornado
• Tornado does well in benchmarks.
• Tornado is event driven. This is a basic axiom in the design of Salvus.
• One can read all the documenation for tornado in a few hours, and you really know all you need to know, except what you'll learn from the excellent source code.
• One concern I have is that right now the way Tornado talks to cassandra is not asynchronous -- I may have to write a better driver. Database queries are typically super fast though, by the nature of cassandra slow queries simply aren't allowed by the language.