Lectures on scientific computing with python, as IPython notebooks, by J. R. Johansson
Using Fortran and C code with Python
J.R. Johansson (jrjohansson at gmail.com)
The latest version of this IPython notebook lecture is available at http://github.com/jrjohansson/scientific-python-lectures.
The other notebooks in this lecture series are indexed at http://jrjohansson.github.io.
The advantage of Python is that it is flexible and easy to program. The time it takes to setup a new calulation is therefore short. But for certain types of calculations Python (and any other interpreted language) can be very slow. It is particularly iterations over large arrays that is difficult to do efficiently.
Such calculations may be implemented in a compiled language such as C or Fortran. In Python it is relatively easy to call out to libraries with compiled C or Fortran code. In this lecture we will look at how to do that.
But before we go ahead and work on optimizing anything, it is always worthwhile to ask....
Fortran
F2PY
F2PY is a program that (almost) automatically wraps fortran code for use in Python: By using the f2py
program we can compile fortran code into a module that we can import in a Python program.
F2PY is a part of NumPy, but you will also need to have a fortran compiler to run the examples below.
Example 0: scalar input, no output
Generate a python module using f2py
:
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "hellofortran" sources
f2py options: []
f2py:> /tmp/tmpz2IPjB/src.linux-x86_64-2.7/hellofortranmodule.c
creating /tmp/tmpz2IPjB/src.linux-x86_64-2.7
Reading fortran codes...
Reading file 'hellofortran.f' (format:fix,strict)
Post-processing...
Block: hellofortran
Block: hellofortran
Post-processing (stage 2)...
Building modules...
Building module "hellofortran"...
Constructing wrapper function "hellofortran"...
hellofortran(n)
Wrote C/API module "hellofortran" to file "/tmp/tmpz2IPjB/src.linux-x86_64-2.7/hellofortranmodule.c"
adding '/tmp/tmpz2IPjB/src.linux-x86_64-2.7/fortranobject.c' to sources.
adding '/tmp/tmpz2IPjB/src.linux-x86_64-2.7' to include_dirs.
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpz2IPjB/src.linux-x86_64-2.7
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpz2IPjB/src.linux-x86_64-2.7
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'hellofortran' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC
creating /tmp/tmpz2IPjB/tmp
creating /tmp/tmpz2IPjB/tmp/tmpz2IPjB
creating /tmp/tmpz2IPjB/tmp/tmpz2IPjB/src.linux-x86_64-2.7
compile options: '-I/tmp/tmpz2IPjB/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
x86_64-linux-gnu-gcc: /tmp/tmpz2IPjB/src.linux-x86_64-2.7/hellofortranmodule.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpz2IPjB/src.linux-x86_64-2.7/fortranobject.h:13,
from /tmp/tmpz2IPjB/src.linux-x86_64-2.7/hellofortranmodule.c:17:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
x86_64-linux-gnu-gcc: /tmp/tmpz2IPjB/src.linux-x86_64-2.7/fortranobject.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpz2IPjB/src.linux-x86_64-2.7/fortranobject.h:13,
from /tmp/tmpz2IPjB/src.linux-x86_64-2.7/fortranobject.c:2:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I/tmp/tmpz2IPjB/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gfortran:f77: hellofortran.f
/usr/bin/gfortran -Wall -Wall -shared /tmp/tmpz2IPjB/tmp/tmpz2IPjB/src.linux-x86_64-2.7/hellofortranmodule.o /tmp/tmpz2IPjB/tmp/tmpz2IPjB/src.linux-x86_64-2.7/fortranobject.o /tmp/tmpz2IPjB/hellofortran.o -lgfortran -o ./hellofortran.so
Removing build directory /tmp/tmpz2IPjB
Example of a python script that use the module:
Fortran says hello
Fortran says hello
Fortran says hello
Fortran says hello
Fortran says hello
Fortran says hello
Example 1: vector input and scalar output
Reading fortran codes...
Reading file 'dprod.f' (format:fix,strict)
Post-processing...
Block: dprod
{}
In: :dprod:dprod.f:dprod
vars2fortran: No typespec for argument "n".
Block: dprod
Post-processing (stage 2)...
Saving signatures to file "./dprod.pyf"
The f2py
program generated a module declaration file called dsum.pyf
. Let's look what's in it:
The module does not know what Fortran subroutine arguments is input and output, so we need to manually edit the module declaration files and mark output variables with intent(out)
and input variable with intent(in)
:
Compile the fortran code into a module that can be included in python:
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "dprod" sources
creating /tmp/tmpWyCvx1/src.linux-x86_64-2.7
f2py options: []
f2py: dprod.pyf
Reading fortran codes...
Reading file 'dprod.pyf' (format:free)
Post-processing...
Block: dprod
Block: dprod
Post-processing (stage 2)...
Building modules...
Building module "dprod"...
Constructing wrapper function "dprod"...
y = dprod(x,[n])
Wrote C/API module "dprod" to file "/tmp/tmpWyCvx1/src.linux-x86_64-2.7/dprodmodule.c"
adding '/tmp/tmpWyCvx1/src.linux-x86_64-2.7/fortranobject.c' to sources.
adding '/tmp/tmpWyCvx1/src.linux-x86_64-2.7' to include_dirs.
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpWyCvx1/src.linux-x86_64-2.7
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpWyCvx1/src.linux-x86_64-2.7
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'dprod' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC
creating /tmp/tmpWyCvx1/tmp
creating /tmp/tmpWyCvx1/tmp/tmpWyCvx1
creating /tmp/tmpWyCvx1/tmp/tmpWyCvx1/src.linux-x86_64-2.7
compile options: '-I/tmp/tmpWyCvx1/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
x86_64-linux-gnu-gcc: /tmp/tmpWyCvx1/src.linux-x86_64-2.7/dprodmodule.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpWyCvx1/src.linux-x86_64-2.7/fortranobject.h:13,
from /tmp/tmpWyCvx1/src.linux-x86_64-2.7/dprodmodule.c:18:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
/tmp/tmpWyCvx1/src.linux-x86_64-2.7/dprodmodule.c:111:12: warning: ‘f2py_size’ defined but not used [-Wunused-function]
static int f2py_size(PyArrayObject* var, ...)
^
x86_64-linux-gnu-gcc: /tmp/tmpWyCvx1/src.linux-x86_64-2.7/fortranobject.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpWyCvx1/src.linux-x86_64-2.7/fortranobject.h:13,
from /tmp/tmpWyCvx1/src.linux-x86_64-2.7/fortranobject.c:2:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I/tmp/tmpWyCvx1/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gfortran:f77: dprod.f
/usr/bin/gfortran -Wall -Wall -shared /tmp/tmpWyCvx1/tmp/tmpWyCvx1/src.linux-x86_64-2.7/dprodmodule.o /tmp/tmpWyCvx1/tmp/tmpWyCvx1/src.linux-x86_64-2.7/fortranobject.o /tmp/tmpWyCvx1/dprod.o -lgfortran -o ./dprod.so
Removing build directory /tmp/tmpWyCvx1
Using the module from Python
Compare performance:
Example 2: cummulative sum, vector input and vector output
The cummulative sum function for an array of data is a good example of a loop intense algorithm: Loop through a vector and store the cummulative sum in another vector.
Fortran subroutine for the same thing: here we have added the intent(in)
and intent(out)
as comment lines in the original fortran code, so we do not need to manually edit the fortran module declaration file generated by f2py
.
We can directly compile the fortran code to a python module:
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "dcumsum" sources
f2py options: []
f2py:> /tmp/tmpfvrMl6/src.linux-x86_64-2.7/dcumsummodule.c
creating /tmp/tmpfvrMl6/src.linux-x86_64-2.7
Reading fortran codes...
Reading file 'dcumsum.f' (format:fix,strict)
Post-processing...
Block: dcumsum
Block: dcumsum
Post-processing (stage 2)...
Building modules...
Building module "dcumsum"...
Constructing wrapper function "dcumsum"...
b = dcumsum(a)
Wrote C/API module "dcumsum" to file "/tmp/tmpfvrMl6/src.linux-x86_64-2.7/dcumsummodule.c"
adding '/tmp/tmpfvrMl6/src.linux-x86_64-2.7/fortranobject.c' to sources.
adding '/tmp/tmpfvrMl6/src.linux-x86_64-2.7' to include_dirs.
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpfvrMl6/src.linux-x86_64-2.7
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpfvrMl6/src.linux-x86_64-2.7
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'dcumsum' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC
creating /tmp/tmpfvrMl6/tmp
creating /tmp/tmpfvrMl6/tmp/tmpfvrMl6
creating /tmp/tmpfvrMl6/tmp/tmpfvrMl6/src.linux-x86_64-2.7
compile options: '-I/tmp/tmpfvrMl6/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
x86_64-linux-gnu-gcc: /tmp/tmpfvrMl6/src.linux-x86_64-2.7/dcumsummodule.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpfvrMl6/src.linux-x86_64-2.7/fortranobject.h:13,
from /tmp/tmpfvrMl6/src.linux-x86_64-2.7/dcumsummodule.c:18:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
/tmp/tmpfvrMl6/src.linux-x86_64-2.7/dcumsummodule.c:111:12: warning: ‘f2py_size’ defined but not used [-Wunused-function]
static int f2py_size(PyArrayObject* var, ...)
^
x86_64-linux-gnu-gcc: /tmp/tmpfvrMl6/src.linux-x86_64-2.7/fortranobject.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/tmpfvrMl6/src.linux-x86_64-2.7/fortranobject.h:13,
from /tmp/tmpfvrMl6/src.linux-x86_64-2.7/fortranobject.c:2:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I/tmp/tmpfvrMl6/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gfortran:f77: dcumsum.f
/usr/bin/gfortran -Wall -Wall -shared /tmp/tmpfvrMl6/tmp/tmpfvrMl6/src.linux-x86_64-2.7/dcumsummodule.o /tmp/tmpfvrMl6/tmp/tmpfvrMl6/src.linux-x86_64-2.7/fortranobject.o /tmp/tmpfvrMl6/dcumsum.o -lgfortran -o ./dcumsum.so
Removing build directory /tmp/tmpfvrMl6
Benchmark the different implementations:
Further reading
C
ctypes
ctypes is a Python library for calling out to C code. It is not as automatic as f2py
, and we manually need to load the library and set properties such as the functions return and argument types. On the otherhand we do not need to touch the C code at all.
Compile the C file into a shared library:
The result is a compiled shared library libfunctions.so
:
libfunctions.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d68173ae6a804f703472af96f413b81a189db4b8, not stripped
Now we need to write wrapper functions to access the C library: To load the library we use the ctypes package, which included in the Python standard library (with extensions from numpy for passing arrays to C). Then we manually set the types of the argument and return values (no automatic code inspection here!).
C says hello
C says hello
C says hello
Product function:
Cummulative sum:
Simple benchmark
Further reading
Cython
A hybrid between python and C that can be compiled: Basically Python code with type declarations.
A build file for generating C code and compiling it into a Python module.
Cython in the IPython notebook
When working with the IPython (especially in the notebook), there is a more convenient way of compiling and loading Cython code. Using the %%cython
IPython magic (command to IPython), we can simply type the Cython code in a code cell and let IPython take care of the conversion to C code, compilation and loading of the function. To be able to use the %%cython
magic, we first need to load the extension cythonmagic
:
Further reading
Versions
Software | Version |
---|---|
Python | 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] |
IPython | 1.1.0 |
OS | posix [linux2] |
ctypes | 1.1.0 |
Cython | 0.20.2 |
Tue Aug 26 23:37:29 2014 JST |