How to Do a Release ------------------- Periodically, the yt development community issues new releases. yt loosely follows `semantic versioning <https://semver.org/>`_. The type of release can be read off from the version number used. Version numbers should follow the scheme ``MAJOR.MINOR.PATCH``. There are three kinds of possible releases: * Bugfix releases These releases should contain only fixes for bugs discovered in earlier releases and should not contain new features or API changes. Bugfix releases only increment the ``PATCH`` version number. Bugfix releases should *not* be generated by merging from the ``main`` branch, instead bugfix pull requests should be backported to a dedicated branch. See :ref:`doing-a-bugfix-release`. Version ``3.2.2`` is a bugfix release. * Minor releases These releases happen when new features are deemed ready to be merged into the ``stable`` branch and should not happen on a regular schedule. Minor releases can also include fixes for bugs if the fix is determined to be too invasive for a bugfix release. Minor releases should *not* include backwards-incompatible changes and should not change APIs. If an API change is deemed to be necessary, the old API should continue to function but might trigger deprecation warnings. Minor releases should happen by merging the ``main`` branch into the ``stable`` branch. Minor releases should increment the ``MINOR`` version number and reset the ``PATCH`` version number to zero. Version ``3.3.0`` is a minor release. * Major releases These releases happen when the development community decides to make major backwards-incompatible changes intentionally. In principle a major version release could include arbitrary changes to the library. Major version releases should only happen after extensive discussion and vetting among the developer and user community. Like minor releases, a major release should happen by merging the ``main`` branch into the ``stable`` branch. Major releases should increment the ``MAJOR`` version number and reset the ``MINOR`` and ``PATCH`` version numbers to zero. If it ever happens, version ``4.0.0`` will be a major release. The job of doing a release differs depending on the kind of release. Below, we describe the necessary steps for each kind of release in detail. .. _doing-a-bugfix-release: Doing a Bugfix Release ~~~~~~~~~~~~~~~~~~~~~~ As described above, bugfix releases are regularly scheduled updates for minor releases to ensure fixes for bugs make their way out to users in a timely manner. Since bugfix releases should not include new features, we do not issue bugfix releases by simply merging from the development ``main`` branch into the ``stable`` branch. Instead, commits are cherry-picked from the ``main`` branch to a backport branch, which is itself merged into ``stable`` when a release happens. Backport branches are named after the minor version then descend from, followed by an ``x``. For instance, ``yt-4.0.x`` is the backport branch for all releases in the 4.0 series. Backporting bugfixes can be done automatically using the `MeeseeksBox bot <https://meeseeksbox.github.io>`_. This necessitates having a Github milestone dedicated to the release, configured with a comment in its description such as ``on-merge: backport to yt-4.0.x``. Then, every PR that was triaged into the milestone will be replicated as a backport PR by the bot when it's merged into main. Some backports are non-trivial and require human attention; if conflicts occur, the bot will provide detailed instructions to perfom the task manually. In short, a manual backport consist of 4 steps - checking out the backport branch locally - create a new branch from there - cherry-picking the merge commit from the original PR with ``git cherry-pick -m1 <commit sha>`` - opening a PR to the backport branch Doing a Minor or Major Release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is much simpler than a bugfix release. First, make sure that every deprecated features targeted for removal in the new release are removed from the ``main`` branch, ideally in a single PR. Such a PR can be issued at any point between the previous minor or major release and the new one. Then, all that needs to happen is the ``main`` branch must get merged into the ``stable`` branch, and any conflicts that happen must be resolved, almost always in favor of the state of the code on the ``main`` branch. Incrementing Version Numbers and Tagging a Release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before creating the tag for the release, you must increment the version numbers that are hard-coded in a few files in the yt source so that version metadata for the code is generated correctly. This includes things like ``yt.__version__`` and the version that gets read by the Python Package Index (PyPI) infrastructure. The paths relative to the root of the repository for the three files that need to be edited are: * ``doc/source/conf.py`` The ``version`` and ``release`` variables need to be updated. * ``setup.py`` The ``VERSION`` variable needs to be updated * ``yt/__init__.py`` The ``__version__`` variable must be updated. Once these files have been updated, commit these updates. This is the commit we will tag for the release. To actually create the tag, issue the following command from the ``stable`` branch: .. code-block:: bash git tag <tag-name> Where ``<tag-name>`` follows the project's naming scheme for tags (e.g. ``yt-3.2.1``). Once you are done, you will need to push the tag to github:: git push origin --tag This assumes that you have configured the remote ``origin`` to point at the main yt git repository. If you are doing a minor or major version number release, you will also need to update back to the development branch and update the development version numbers in the same files. Uploading to yt-project.org ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Before uploading the release to the Python Package Index (pypi.org) we will first upload the package to yt-project.org. This facilitates building binary wheels for pypi and binary conda packages on conda-forge before doing the "official" release. This also ensures that there isn't a period of time when users do ``pip install yt`` and end up downloading the source distribution instead of one of the binary wheels. To create the source distribution, issue the following command in the root of the yt repository:: $ python setup.py sdist This will generate a tarball in a ``dist/`` directory located in the root of the repository. Access to yt-project.org mediated via SSH login. Please contact one of the current yt developers for access to the webserver running yt-project.org if you do not already have it. You will need a copy of your SSH public key so that your key can be added to the list of authorized keys. Once you login, use e.g. ``scp`` to upload a copy of the source distribution tarball to https://yt-project.org/sdist, like so:: $ scp dist/yt-3.5.1.tar.gz [email protected]:yt-project.org/sdist You may find it helpful to set up an ssh config for dickenson to make this command a bit easier to execute. Publishing ~~~~~~~~~~ We distribute yt on two main channels: PyPI.org and conda-forge, in this order. PyPI ++++ The publication process for PyPI is automated for the most part, via Github actions, using ``.github/workflows/wheels.yaml``. Specifically, a release is pushed to PyPI when a new git tag starting with ``yt-`` is pushed to the main repo. Let's review the details here. PyPI releases contain the source code (as a tarball), and wheels. Wheels are compiled distributions of the source code. They are OS specific as well as Python-version specific. Producing wheels for every supported combination of OS and Python versions is done with `cibuildwheels <https://cibuildwheel.readthedocs.org>`_ Upload to PyPI is automated via Github Actions `upload-artifact <https://github.com/actions/upload-artifact>`_ and `download-artifact <https://github.com/actions/upload-artifact>`_. Note that automated uploads are currently perfomed using Matt Turk's credentials. If that worked, you can skip to the next section. Otherwise, upload can be perfomed manually by first downloading the artifacts ``wheels`` and ``tarball`` from the workflow webpage, then at the command line (make sure that the ``dist`` directory doesn't exist or is empty) .. code-block:: bash unzip tarball.zip -d dist unzip wheels.zip -d dist python -m pip install --upgrade twine twine upload dist/* You will be prompted for your PyPI credentials and then the package should upload. Note that for this to complete successfully, you will need an account on PyPI and that account will need to be registered as an "owner" or "maintainer" of the yt package. ``conda-forge`` +++++++++++++++ Conda-forge packages for yt are managed via the yt feedstock, located at https://github.com/conda-forge/yt-feedstock. When a release is pushed to PyPI a bot should detect a new version and issue a PR to the feedstock with the new version automatically. When this feedstock is updated, make sure that the SHA256 hash of the tarball matches the one you uploaded to PyPI and that the version number matches the one that is being released. In case the automated PR fails CI, feedstock maintainers are allowed to push to the bot's branch with any fixes required. Should you need to update the feedstock manually, you will need to update the ``meta.yaml`` file located in the ``recipe`` folder in the root of the feedstock repository. Most likely you will only need to update the version number and the SHA256 hash of the tarball. If yt's dependencies change you may also need to update the recipe. Once you have updated the recipe, propose a pull request on github and merge it once all builds pass. Announcing ~~~~~~~~~~ After the release is uploaded to `PyPI <https://pypi.org/project/yt/#files>`_ and `conda-forge <https://anaconda.org/conda-forge/yt>`_, you should send out an announcement e-mail to the yt mailing lists as well as other possibly interested mailing lists for all but bugfix releases. Creating a Github release attached to the tag also offers a couple advantages. Auto-generated release notes can be a good starting point, though it's best to edit out PRs that not directly affecting users, and these notes can be edited before (draft mode) and after the release, so errors can be corrected after the fact.
