building_the_docs.rst

.. _documentation:

Documentation
=============

.. _writing_documentation:

How to Write Documentation
--------------------------

Writing documentation is one of the most important but often overlooked tasks
for increasing yt's impact in the community.  It is the way in which the
world will understand how to use our code, so it needs to be done concisely
and understandably.  Typically, when a developer submits some piece of code
with new functionality, the developer should also include documentation on how
to use that functionality (as per :ref:`requirements-for-code-submission`).
Depending on the nature of the code addition, this could be a new narrative
docs section describing how the new code works and how to use it, it could
include a recipe in the cookbook section, or it could simply be adding a note
in the relevant docs text somewhere.

The documentation exists in the main code repository for yt in the
``doc`` directory (i.e. ``$YT_GIT/doc/source`` where ``$YT_GIT`` is the path of
the yt git repository).  It is organized hierarchically into the main
categories of:

* Visualizing
* Analyzing
* Analysis Modules
* Examining
* Cookbook
* Quickstart
* Developing
* Reference
* FAQ
* Help

You will have to figure out where your new/modified doc fits into this, but
browsing through the existing documentation is a good way to sort that out.

All the source for the documentation is written in
`Sphinx <http://www.sphinx-doc.org/en/master/>`_, which uses ReST for markup.  ReST is very
straightforward to markup in a text editor, and if you are new to it, we
recommend just using other .rst files in the existing yt documentation as
templates or checking out the
`ReST reference documentation <http://www.sphinx-doc.org/en/master/usage/restructuredtext/>`_.

New cookbook recipes (see :ref:`cookbook`) are very helpful for the community
as they provide simple annotated recipes on how to use specific functionality.
To add one, create a concise Python script which demonstrates some
functionality and pare it down to its minimum.  Add some comment lines to
describe what it is that you're doing along the way.  Place this ``.py`` file
in the ``source/cookbook/`` directory, and then link to it explicitly in one
of the relevant ``.rst`` files in that directory (e.g. ``complex_plots.rst``,
etc.), and add some description of what the script
actually does.  We recommend that you use one of the
`sample data sets <https://yt-project.org/data>`_ in your recipe.  When the full
docs are built, each of the cookbook recipes is executed dynamically on
a system which has access to all of the sample datasets.  Any output images
generated by your script will then be attached inline in the built documentation
directly following your script.

After you have made your modifications to the docs, you will want to make sure
that they render the way you expect them to render.  For more information on
this, see the section on :ref:`docs_build`.  Unless you're contributing cookbook
recipes or notebooks which require a dynamic build, you can probably get away
with just doing a 'quick' docs build.

When you have completed your documentation additions, commit your changes
to your repository and make a pull request in the same way you would contribute
a change to the codebase, as described in the section on :ref:`sharing-changes`.

.. _docs_build:

Building the Documentation
--------------------------

The yt documentation makes heavy use of the Sphinx documentation automation
suite.  Sphinx, written in Python, was originally created for the documentation
of the Python project and has many nice capabilities for managing the
documentation of Python code.

While much of the yt documentation is static text, we make heavy use of
cross-referencing with API documentation that is automatically generated at
build time by Sphinx.  We also use Sphinx to run code snippets (e.g. the
cookbook and the notebooks) and embed resulting images and example data.

Essential tools for building the docs can be installed alongside yt itself. From
the top level of a local copy, run

.. code-block:: bash

   $ python -m pip install -e . -r requirements/docs.txt

Quick versus Full Documentation Builds
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Building the entire set of yt documentation is a laborious task, since you
need to have a large number of packages in order to successfully execute
and render all of the notebooks and yt recipes drawing from every corner
of the yt source.  As a quick alternative, one can do a ``quick`` build
of the documentation, which eschews the need for downloading all of these
dependencies, but it only produces the static docs.  The static docs do
not include the cookbook outputs and the notebooks, but this is good
enough for most cases of people testing out whether or not their documentation
contributions look OK before submitting them to the yt repository.

If you want to create the full documentation locally, then you'll need
to follow the instructions for building the ``full`` docs, so that you can
dynamically execute and render the cookbook recipes, the notebooks, etc.

Building the Docs (Quick)
^^^^^^^^^^^^^^^^^^^^^^^^^

In order to tell Sphinx not to do all of the dynamic building, you must set the
``$READTHEDOCS`` environment variable to be ``True`` by run from the command
line (using bash syntax for example), as

.. code-block:: bash

   export READTHEDOCS=True

This variable is set for automated builds on the free ReadTheDocs service but
can be used by anyone to force a quick, minimal build.

Now all you need to do is execute Sphinx on the yt doc source.  Go to the
documentation directory and build the docs:

.. code-block:: bash

   cd $YT_GIT/doc
   make html

This will produce an html version of the documentation locally in the
``$YT_GIT/doc/build/html`` directory.  You can now go there and open
up ``index.html`` or whatever file you wish in your web browser.

Building the Docs (Full)
^^^^^^^^^^^^^^^^^^^^^^^^

As alluded to earlier, building the full documentation is a bit more involved
than simply building the static documentation.

The full documentation makes heavy use of custom Sphinx extensions to transform
recipes, notebooks, and inline code snippets into Python scripts, IPython_
notebooks, or notebook cells that are executed when the docs are built.

To do this, we use Jupyter's nbconvert module to transform notebooks into
HTML. to simplify versioning of the notebook JSON format, we store notebooks in
an unevaluated state.

To build the full documentation, you will need yt, jupyter, and all dependencies
needed for yt's analysis modules installed. The following dependencies were
used to generate the yt documentation during the release of yt 3.2 in 2015.

* Sphinx_ 1.3.1
* Jupyter 1.0.0
* RunNotebook 0.1
* pandoc_ 1.13.2
* Rockstar halo finder 0.99.6
* SZpack_ 1.1.1
* ffmpeg_ 2.7.1 (compiled with libvpx support)
* Astropy_ 0.4.4

.. _SZpack: http://www.jb.man.ac.uk/~jchluba/Science/SZpack/SZpack.html
.. _Astropy: https://www.astropy.org/
.. _Sphinx: http://www.sphinx-doc.org/en/master/
.. _pandoc: https://pandoc.org/
.. _ffmpeg: http://www.ffmpeg.org/
.. _IPython: https://ipython.org/

You will also need the full yt suite of `yt test data
<https://yt-project.org/data/>`_, including the larger datasets that are not used
in the answer tests.

You will need to ensure that your testing configuration is properly
configured and that all of the yt test data is in the testing directory.  See
:ref:`run_answer_testing` for more details on how to set up the testing
configuration.

Now that you have everything set up properly, go to the documentation directory
and build it using Sphinx:

.. code-block:: bash

   cd $YT_GIT/doc
   make html

If all of the dependencies are installed and all of the test data is in the
testing directory, this should churn away for a while (several hours) and
eventually generate a docs build.  We suggest setting
:code:`suppress_stream_logging = True` in your yt configuration (See
:ref:`configuration-file`) to suppress large amounts of debug output from
yt.

To clean the docs build, use :code:`make clean`.

Building the Docs (Hybrid)
^^^^^^^^^^^^^^^^^^^^^^^^^^

It's also possible to create a custom Sphinx build that builds a restricted set
of notebooks or scripts.  This can be accomplished by editing the Sphinx
:code:`conf.py` file included in the :code:`source` directory at the top level
of the docs.  The extensions included in the build are contained in the
:code:`extensions` list.  To disable an extension, simply remove it from the
list.  Doing so will raise a warning when Sphinx encounters the directive in the
docs and will prevent Sphinx from evaluating the directive.

As a concrete example, if one wanted to include the :code:`notebook`, and
:code:`notebook-cell` directives, but not the :code:`python-script` or
:code:`autosummary` directives, one would just need to comment out the lines
that append these extensions to the :code:`extensions` list. The resulting docs
build will be significantly quicker since it would avoid executing the lengthy
API autodocumentation as well as a large number of Python script snippets in
the narrative docs.