Lecture 9: Modules, packages and SageMath core development

References:

• Python modules and packages

• Python packaging user guide

• SageMath developer's guide

Summary:
In this lecture we discuss how (and why) to put code into Python modules and Python packages, and how to install the latter on your system. In the second part, I show a practical example of the steps necessary to contribute to the code of SageMath itself.

Python modules and packages

In the previous lecture we saw that one way to share code is to save it to a .sage-file, and run it using load. This is quick to set up and works reasonably in many use cases, but it has some drawbacks (that we discuss below). These can be avoided using modules and packages in Python.

Python modules

One disadvantage when using load is that it transfers all variable and function names from the loaded file into the current session (of Python or SageMath). In particular, this overwrites existing functions, which can create problems e.g. if you want to use multiple code files using the same function name.

To avoid this, you can instead store code inside a file with ending .py (which defines a Python module), and then import this file. Instead of loading all function names into the current session, this will keep them bundled up inside a module, and you can then access them as shown below.

Exercise

• Create a file basics.py in the folder of the current notebook, and save the following text inside (similar to the .sage files we had before). If you feel lazy, you can also download the file here.

def is_prime(n):
    for m in range(2, n):
        if n % m == 0:
            return False
    return True

def one_third():
    return 1/3

• Import it as a module using

import basics

• Run the function is_prime in an example of your choice, accessing it via basics.is_prime.

Some remarks about the example above:

• First, note that the is_prime function from the modulie basics has not overwritten the default is_prime function in SageMath.

is_prime == basics.is_prime

• If you do want to import some function into the namespace of the current session, you can do it as follows:

from basics import is_prime

is_prime == basics.is_prime

• Finally, it is both possible to import a function (or class) giving it a custom name, or to import all names from a given module.

from basics import is_prime as is_p

is_p(7)

from basics import *

One issue to be aware of is that code stored in a .py file is interpreted as Python code. A first consequence is that e.g. numbers you type in this code will be interpreted as Python int or float values. Therefore, the function one_third above, which returns 1/3, will actually return int(1)/int(3) which gives a float:

one_third()

Similarly, you need to use e.g. ** instead of ^ for exponentiation. Moreover, functions (like factorial) that are part of SageMath will not automatically be available and need to be imported inside the new Python module. To get the line of code that you need to add for this (traditionally at the beginning of the .py file) you can use the function import_statements in your SageMath session:

import_statements(factorial)

Exercise

Say we want to have a function computing 1/3 as a Rational. The following code is a first attempt:

def better_one_third():
    return Integer(1)/Integer(3)

• Add the code above to the file basics.py. Restart the kernel of the notebook (via Kernel -> Restart in the menu) and import basics again. Check that when trying to call this function, we obtain an error message since Integer is not known.
  Remark: Note that restarting the kernel is really necessary: the import will not overwrite a module that was imported before, even if its code changed in the meantime.

• Find the correct line for importing the missing name Integer and add it to basics.py. Then restart the kernel, import the module and check that now the function better_one_third works.

While it is a bit cumbersome having to import the relevant SageMath-functions as above, it is (as far as I know) the only way to use the strengths of Python modules (and the Python packages below) in SageMath. One tool making it easier is to use the automatic preparser of SageMath. This means, you can create a file basics.sage and write normal Sage-code (that would work inside a SageMath notebook). Then, open a terminal (not a SageMath session!) and type

sage --preparse basics.sage

The resulting code is correct, but has the disadvantage of containing some inconvenient abbreviations (e.g. for constants).

# This file was *autogenerated* from the file basics_for_preparsing.sage
from sage.all_cmdline import *   # import sage library

_sage_const_2 = Integer(2); _sage_const_0 = Integer(0); _sage_const_1 = Integer(1); _sage_const_3 = Integer(3)
def is_prime(n):
    for m in range(_sage_const_2 , n):
        if n % m == _sage_const_0 :
            return False
    return True

def one_third():
    return _sage_const_1 /_sage_const_3 

For more information see this guide.

Python packages

For bigger projects, it is often convenient to be able to split the code in multiple files (containing different parts of the project). To bundle these, one can combine them into a Python package.

To get our hands on an example, you can download this zip file and unpack it (in all common operating systems: do a right-click and select "Extract all" or something similar). You should get a folder structure as follows:

testpackage/
├── LICENSE
├── README.md
├── setup.py
└── testpack
    ├── __init__.py
    ├── advanced.py
    └── basics.py

Ignoring the files LICENSE etc for now, the actual Python package is testpack. In general, such a package is essentially a folder containing

• a file __init__.py

• some Python modules, which can either be .py-files or further sub-packages

Here the __init__.py-file mostly has a dummy function (allowing the folder to be recognized as a package). However, it can contain some Python code, which is executed when testpack is first imported. Let's try it out and make some comments.

Make sure that the folder testpackage is contained in the same folder as the current notebook. Then we can load the package testpack as follows:

import testpackage.testpack as testpack

type(testpack)

When the package is first imported, all code contained in the __init__.py file is run (inside the module). In our case, the __init__.py file contains the following lines:

from .advanced import my_favorite_function as myfavfunc
from .basics import scalar_product

Note that above we do a relative import: the . stands for the current directory of __init__.py, and so .advanced stands for the file advanced.py located in the same folder as __init__.py.

Typing testpack. + Tab you can check that the module testpack now contains

• the sub-modules advanced and basic, and

• the functions myfavfunc and scalar_product which were imported in the __init__.py file.

testpack.

So in practice, you can use the __init__.pyfile to determine which functions will be loaded into the session when the user types

from testpack import *

Exercise

Look at the code contained in advanced and basics, test some of the functions below and try to understand what each line does.

Installing and distributing packages

Getting the package ready for installation

For the Python package testpack above, there is one drawback: to import it, it is most convenient to run the SageMath console or notebook inside the folder containing testpack (otherwise the command import testpack will give an error). This can be solved by installing the package, effectively adding it permanently to the list of "known modules" of your SageMath installation.

To make this possible, we put the folder testpack into a directory testpackage with several other files. In more detail, these are:

• setup.py: It contains instructions for the setuptools module of Python, which takes care of the installation. In the file we provide some basic information about the package (such as name, author, a short description etc). An important variable is packages which includes the path (relative to setup.py) of the actual Python package we want to install.

• README.md: For a longer description, it is customary to provide such a readme file. Here .md stands for Markdown, the same language we use for the text/formulas/pictures in Jupyter notebooks. This description is also input in the setup instructions above via a short script in setup.py loading its text into the parameter long_description.

• LICENSE: This file contains the text of a possible software license that you want to use when distributing your code (see below).

The Python package manager pip

With these preparations in place, we can install testpack via the Python package manager pip. For this, navigate with a terminal into the folder testpackage (not testpack!) and type

sage -pip install .

Here we use sage -pip to use the version of pip associated with the Python installation used by SageMath, and . again stands for the current directory (containing setup.py).

Exercise

• Install the package testpack as described above.

• Navigate with your terminal into a folder not containing the testpack folder.

• Open a SageMath session, run the command import testpack and execute one of the functions from that package.

Some variants of the command above:

sage -pip install -e .

Installs the package in editable mode. This means, if you change the code inside testpack it will change the behaviour of the module (with code sage -pip install . above you save a copy of the module at the time of installation, which is not changed when you modify the original code). This editable option is particularly useful if you are still actively working on the code of the package.

sage -pip install . --upgrade

If you already have a previous version installed, you can use this to upgrade to the new one.

Revisiting doctests

Once you installed the package, it is also possible to run any of the doctests that you included. You should run the sage -t command on the folder testpack containing the code files.

sage -t testpackage/testpack

Note that to make this work, your doctests should always start by importing the relevant functions from the package, like the following doctest of scalar_product:

    EXAMPLES::
    
        sage: from testpack.basics import scalar_product
        sage: scalar_product([1,2], [3,4])
        11

The Python package indey PyPI

One final great feature about pip is that it is by default connected to the Python package index (PyPI). This is an online database of Python packages, and to install a package foo from there, you just need to type

sage -pip install foo

For our beloved test package above, I did not upload it to PyPI itself, but (following these instructions) I uploaded it to a test-verion of PyPI. Thus, from a console anywhere you can install it using the command

sage -pip install -i https://test.pypi.org/simple/ testpackmath007

Remark: As you see I had to change the name to testpackmat007 since, maybe not unsurprisingly, the very creative name testpack was already taken ...

Summary: the right format for your code

Here we provide again an overview over the strengths and weaknesses of various ways of storing your code:

• Jupyter notebooks are nice for short pieces of code with some additional explanatory text and examples

• Sage files are great for projects of medium size, where the user wants to load all relevant functions into their session

• Python files allow to create a separate namespace, avoiding collisions with user-defined functions, but they require that you write pure Python code and import relevant SageMath functions by hand

• Python packages are the professional standard for projects of greater complexity and can use the Python package index for easy distribution and installation

A panorama of further tools for development

• When editing your code files, you can use the versioning software git to

  • keep a record of previous versions of your code

  • synchronize your local files with some online repository (like gitlab) to allow multiple people to work on them

• On platforms like gitlab, you also have access to tools for continuous integration, e.g. a script which runs all doctests of your code whenever you create a new version. In addition, there are tools like airspeed velocity which automatically check the speed of the code on a set of example computations, so that you see whether your changes made things faster or slower.

• You can use sphinx to create nice documentation pages for your code (such as this one). They are created by putting the docstrings of your functions into some nice, readable format.

SageMath core development - a practical example (a.k.a. showing my homework)

The task

Recall from the first lecture the following disappointing performance of SageMath:

M = matrix([[17,2,-4,9],[-13,2,8,3],[6,1,-1,4],[5,5,-2,2]])
P = M.characteristic_polynomial()
P

S = ZZ[x].quotient_ring(P); S

S.is_integral_domain()

The problem was that the class of S did not have a good implementation of the function is_integral_domain. I gave myself the exercise to write a suitable version of the is_integral_domain function, and get it added to SageMath.

My solution

You can see the results of my effort under the following trac ticket of SageMath. To see the actual code changes I proposed, you can click on the link to the branch which I created.

Below is the relevant function. Note that in this case self will be the ring in question, which is of the form $$R[x]/(f(x))$$ where $R$ (=self.base_ring()) is any commutative ring with $1$ and $f$ (=self.modulus()) is a nonzero element of $R[x]$ with leading coefficient which is a unit in $R$.

After executing the cell below, the rings will know this better function is_integral_domain, so when you run the cell above again, it will give the correct result.

def is_integral_domain(self, proof = True):
    """
    Return whether or not this quotient ring is an integral domain.

    EXAMPLES::

        sage: R.<z> = PolynomialRing(ZZ)
        sage: S = R.quotient(z^2 - z)
        sage: S.is_integral_domain()
        False
        sage: T = R.quotient(z^2 + 1)
        sage: T.is_integral_domain()
        True
        sage: U = R.quotient(-1)
        sage: U.is_integral_domain()
        False
        sage: R2.<y> = PolynomialRing(R)
        sage: S2 = R2.quotient(z^2 - y^3)
        sage: S2.is_integral_domain()
        True
        sage: S3 = R2.quotient(z^2 - 2*y*z + y^2)
        sage: S3.is_integral_domain()
        False

        sage: R.<z> = PolynomialRing(ZZ.quotient(4))
        sage: S = R.quotient(z-1)
        sage: S.is_integral_domain()
        False

    TESTS:

    Here is an example of a quotient ring which is not an integral
    domain, even though the base ring is integral and the modulus is
    irreducible::

        sage: B = ZZ.extension(x^2 - 5, 'a')
        sage: R.<y> = PolynomialRing(B)
        sage: S = R.quotient(y^2 - y - 1)
        sage: S.is_integral_domain()
        Traceback (most recent call last):
        ...
        NotImplementedError
        sage: S.is_integral_domain(proof = False)
        False

    The reason that the modulus y^2 - y -1 is not prime is that it
    divides the product
    (2*y-(1+a))*(2*y-(1-a)) = 4*y^2 - 4*y - 4.

    Unfortunately, the program above is already unable to determine
    that the modulus is irreducible.
    """
    from sage.categories.all import IntegralDomains
    if self.category().is_subcategory(IntegralDomains()):
        return True
    ret = self.base_ring().is_integral_domain(proof)
    if ret:
        try:
            irr = self.modulus().is_irreducible()
            if not irr:
                # since the modulus is nonzero, the condition of the base ring being an
                # integral domain and the modulus being irreducible are
                # necessary but not sufficient
                ret = False
            else:
                from sage.categories.gcd_domains import GcdDomains
                if self.base_ring() in GcdDomains():
                    # if the base ring is a GCD domain, the conditions are sufficient
                    ret = True
                else:
                    raise NotImplementedError
        except NotImplementedError:
            if proof:
                raise
            else:
                ret = False

    if ret:
        self._refine_category_(IntegralDomains())
    return ret

# Adding this function to existing class via monkey patching
from sage.rings.polynomial.polynomial_quotient_ring import PolynomialQuotientRing_generic
PolynomialQuotientRing_generic.is_integral_domain = is_integral_domain

R.<z> = PolynomialRing(ZZ)
S = R.quotient(z^2 - z)
S.is_integral_domain()

T = R.quotient(z^2 + 1)
T.is_integral_domain()

Solution steps

The abstract process for contributing code to SageMath is described in the official developer guide. Below I give a summary of the concrete steps I had to take:

• To change the source code of SageMath, one needs to first download this code, and compile SageMath from it. Many of the usual installations will give you access to a pre-compiled version, which is not enough to do SageMath development. For me, working on Windows, the easiest way to do this compilation was to work with the Windows subsystem for Linux, which is an installation of Linux running inside Windows.

• I made changes to the source code (mostly in the file /src/sage/rings/polynomial/polynomial_quotient_ring.py), adding the function is_integral_domain above. Note that the function has a documentation with several examples.

• Using this updated version of SageMath, I ran all doctests in the source files of SageMath itself, and encountered an error! It turns out that my changes broke some expected behaviour in an obscure little corner of the software dealing with splitting algebras (whatever that is). This was fixed using the code

    from sage.categories.all import IntegralDomains
    if self.category().is_subcategory(IntegralDomains()):
        return True

Note that I would not have found this in a million years without doctests!

• I pushed my changes to a new branch on the trac server of SageMath where the development happens. For this I needed to create an account there, and install the git-trac software on my computer.

• I got some helpful feedback from Vincent Delecroix, one of the main developers of SageMath, and I changed the code accordingly.

• Once this review was finished, the code was marked ready to be merged into the main (development) version of SageMath (this happened a few days later).

• This means that starting from the next version of SageMath, which is scheduled to release sometime later this year, it will be possible to use the function above.

Assignments

Exercise

Recall that in a previous lecture we had some code for a class Rectangle implementing rectangles with sides parallel to the $x$ and $y$-axes. Similarly, one could write a class Circle implementing circles in the plane (e.g. with given center and radius).

• Write a small Python package, containing files rectangles.py and circles.py which contain classes for such rectangles and circles. Add methods so that for any rectangle, one can compute the circumscribed circle and so that for any circle one can compute the (unique) "insquare", i.e. the unique square with corners on the circle (and sides parallel to $x,y$-directions as before). Make sure that at least some of your functions contain documentation with doctests.

• Install the package in your system.

• Run the doctests.