Releasing a version#

How to Prepare a Release#

This file gives an overview of what is necessary to build binary releases for NumPy.

Current build and release info#

The current info on building and releasing NumPy and SciPy is scattered in several places. It should be summarized in one place, updated, and where necessary described in more detail. The sections below list all places where useful info can be found.

Source tree#

INSTALL.rst.txt
pavement.py

Supported platforms and versions#

NEP 29 outlines which Python versions are supported; For the first half of 2020, this will be Python >= 3.6. We test NumPy against all these versions every time we merge code to main. Binary installers may be available for a subset of these versions (see below).

OS X#

OS X versions >= 10.9 are supported, for Python version support see NEP 29. We build binary wheels for OSX that are compatible with Python.org Python, system Python, homebrew and macports - see this OSX wheel building summary for details.

Windows#

We build 32- and 64-bit wheels on Windows. Windows 7, 8 and 10 are supported. We build NumPy using the mingw-w64 toolchain on Appveyor.

Linux#

We build and ship manylinux1 wheels for NumPy. Many Linux distributions include their own binary builds of NumPy.

BSD / Solaris#

No binaries are provided, but successful builds on Solaris and BSD have been reported.

Tool chain#

We build all our wheels on cloud infrastructure - so this list of compilers is for information and debugging builds locally. See the .travis.yml script in the numpy wheels repo for the definitive source of the build recipes. Packages that are available using pip are noted.

Compilers#

The same gcc version is used as the one with which Python itself is built on each platform. At the moment this means:

OS X builds on travis currently use clang. It appears that binary wheels for OSX >= 10.6 can be safely built from the travis-ci OSX 10.9 VMs when building against the Python from the Python.org installers;
Windows builds use the mingw-w64 toolchain;
Manylinux1 wheels use the gcc provided on the Manylinux docker images.

You will need Cython for building the binaries. Cython compiles the .pyx files in the NumPy distribution to .c files.

OpenBLAS#

All the wheels link to a version of OpenBLAS supplied via the openblas-libs repo. The shared object (or DLL) is shipped with in the wheel, renamed to prevent name collisions with other OpenBLAS shared objects that may exist in the filesystem.

Building source archives and wheels#

You will need write permission for numpy-wheels in order to trigger wheel builds.

Python(s) from python.org or linux distro.
cython (pip)
virtualenv (pip)
Paver (pip)
pandoc pandoc.org or linux distro.
numpy-wheels https://github.com/MacPython/numpy-wheels (clone)

Building docs#

Building the documents requires a number of latex .sty files. Install them all to avoid aggravation.

Sphinx (pip)
numpydoc (pip)
Matplotlib
Texlive (or MikTeX on Windows)

Uploading to PyPI#

terryfy https://github.com/MacPython/terryfy (clone).
beautifulsoup4 (pip)
delocate (pip)
auditwheel (pip)
twine (pip)

Generating author/pr lists#

You will need a personal access token https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/ so that scripts can access the github NumPy repository.

gitpython (pip)
pygithub (pip)

Virtualenv#

Virtualenv is a very useful tool to keep several versions of packages around. It is also used in the Paver script to build the docs.

What is released#

Wheels#

We currently support Python 3.8-3.10 on Windows, OSX, and Linux

Windows: 32-bit and 64-bit wheels built using Appveyor;
OSX: x64_86 OSX wheels built using travis-ci;
Linux: 32-bit and 64-bit Manylinux1 wheels built using travis-ci.

See the numpy wheels building repository for more detail.

Other#

Release Notes
Changelog

Source distribution#

We build source releases in both .zip and .tar.gz formats.

Release process#

Agree on a release schedule#

A typical release schedule is one beta, two release candidates and a final release. It’s best to discuss the timing on the mailing list first, in order for people to get their commits in on time, get doc wiki edits merged, etc. After a date is set, create a new maintenance/x.y.z branch, add new empty release notes for the next version in the main branch and update the Trac Milestones.

Make sure current branch builds a package correctly#

git clean -fxd
python setup.py bdist_wheel
python setup.py sdist

For details of the build process itself, it is best to read the pavement.py script.

Note

The following steps are repeated for the beta(s), release candidates(s) and the final release.

Check deprecations#

Before the release branch is made, it should be checked that all deprecated code that should be removed is actually removed, and all new deprecations say in the docstring or deprecation warning at what version the code will be removed.

Check the C API version number#

The C API version needs to be tracked in three places

numpy/core/setup_common.py
numpy/core/code_generators/cversions.txt
numpy/core/include/numpy/numpyconfig.h

There are three steps to the process.

If the API has changed, increment the C_API_VERSION in setup_common.py. The API is unchanged only if any code compiled against the current API will be backward compatible with the last released NumPy version. Any changes to C structures or additions to the public interface will make the new API not backward compatible.
If the C_API_VERSION in the first step has changed, or if the hash of the API has changed, the cversions.txt file needs to be updated. To check the hash, run the script numpy/core/cversions.py and note the API hash that is printed. If that hash does not match the last hash in numpy/core/code_generators/cversions.txt the hash has changed. Using both the appropriate C_API_VERSION and hash, add a new entry to cversions.txt. If the API version was not changed, but the hash differs, you will need to comment out the previous entry for that API version. For instance, in NumPy 1.9 annotations were added, which changed the hash, but the API was the same as in 1.8. The hash serves as a check for API changes, but it is not definitive.

If steps 1 and 2 are done correctly, compiling the release should not give a warning “API mismatch detect at the beginning of the build”.
The numpy/core/include/numpy/numpyconfig.h will need a new NPY_X_Y_API_VERSION macro, where X and Y are the major and minor version numbers of the release. The value given to that macro only needs to be increased from the previous version if some of the functions or macros in the include files were deprecated.

The C ABI version number in numpy/core/setup_common.py should only be updated for a major release.

Check the release notes#

Use towncrier to build the release note and commit the changes. This will remove all the fragments from doc/release/upcoming_changes and add doc/release/<version>-note.rst.

towncrier build –version “<version>” git commit -m”Create release note”

Check that the release notes are up-to-date.

Update the release notes with a Highlights section. Mention some of the following:

major new features

deprecated and removed features

supported Python versions

for SciPy, supported NumPy version(s)

outlook for the near future

Update the release status and create a release “tag”#

Identify the commit hash of the release, e.g. 1b2e1d63ff:

git co 1b2e1d63ff # gives warning about detached head

First, change/check the following variables in pavement.py depending on the release version:

RELEASE_NOTES = 'doc/release/1.7.0-notes.rst'
LOG_START = 'v1.6.0'
LOG_END = 'maintenance/1.7.x'

Do any other changes. When you are ready to release, do the following changes:

diff --git a/setup.py b/setup.py
index b1f53e3..8b36dbe 100755
--- a/setup.py
+++ b/setup.py
@@ -57,7 +57,7 @@ PLATFORMS           = ["Windows", "Linux", "Solaris", "Mac OS-
 MAJOR               = 1
 MINOR               = 7
 MICRO               = 0
-ISRELEASED          = False
+ISRELEASED          = True
 VERSION             = '%d.%d.%drc1' % (MAJOR, MINOR, MICRO)

 # Return the git revision as a string

And make sure the VERSION variable is set properly.

Now you can make the release commit and tag. We recommend you don’t push the commit or tag immediately, just in case you need to do more cleanup. We prefer to defer the push of the tag until we’re confident this is the exact form of the released code (see: Push the release tag and commit):

git commit -s -m “REL: Release.” setup.py git tag -s <version>

The -s flag makes a PGP (usually GPG) signed tag. Please do sign the release tags.

The release tag should have the release number in the annotation (tag message). Unfortunately, the name of a tag can be changed without breaking the signature, the contents of the message cannot.

See: https://github.com/scipy/scipy/issues/4919 for a discussion of signing release tags, and https://keyring.debian.org/creating-key.html for instructions on creating a GPG key if you do not have one.

To make your key more readily identifiable as you, consider sending your key to public keyservers, with a command such as:

gpg --send-keys <yourkeyid>

Update the version of the main branch#

Increment the release number in setup.py. Release candidates should have “rc1” (or “rc2”, “rcN”) appended to the X.Y.Z format.

Also create a new version hash in cversions.txt and a corresponding version define NPY_x_y_API_VERSION in numpyconfig.h

Trigger the wheel builds#

See the numpy wheels repository.

In that repository edit the files:

azure/posix.yml
azure/windows.yml.

In both cases, set the BUILD_COMMIT variable to the current release tag - e.g. v1.19.0:

$ gvim azure/posix.yml azure/windows.yml
$ git commit -a
$ git push upstream HEAD

Make sure that the release tag has been pushed.

Trigger a build by pushing a commit of your edits to the repository. Note that you can do this on a branch, but it must be pushed upstream to the MacPython/numpy-wheels repository to trigger uploads since only that repo has the appropriate tokens to allow uploads.

The wheels, once built, appear at https://anaconda.org/multibuild-wheels-staging/numpy

Make the release#

Build the changelog and notes for upload with:

paver write_release

Build and archive documentation#

Do:

cd doc/
make dist

to check that the documentation is in a buildable state. Then, after tagging, create an archive of the documentation in the numpy/doc repo:

# This checks out github.com/numpy/doc and adds (``git add``) the
# documentation to the checked out repo.
make merge-doc
# Now edit the ``index.html`` file in the repo to reflect the new content.
# If the documentation is for a non-patch release (e.g. 1.19 -> 1.20),
# make sure to update the ``stable`` symlink to point to the new directory.
ln -sfn <latest_stable_directory> stable
# Commit the changes
git -C build/merge commit -am "Add documentation for <version>"
# Push to numpy/doc repo
git -C build/merge push

Update PyPI#

The wheels and source should be uploaded to PyPI.

You should upload the wheels first, and the source formats last, to make sure that pip users don’t accidentally get a source install when they were expecting a binary wheel.

You can do this automatically using the wheel-uploader script from https://github.com/MacPython/terryfy. Here is the recommended incantation for downloading all the Windows, Manylinux, OSX wheels and uploading to PyPI.

NPY_WHLS=~/wheelhouse   # local directory to cache wheel downloads
CDN_URL=https://anaconda.org/multibuild-wheels-staging/numpy/files
wheel-uploader -u $CDN_URL -w $NPY_WHLS -v -s -t win numpy 1.11.1rc1
wheel-uploader -u $CDN_URL -w warehouse -v -s -t macosx numpy 1.11.1rc1
wheel-uploader -u $CDN_URL -w warehouse -v -s -t manylinux1 numpy 1.11.1rc1

The -v flag gives verbose feedback, -s causes the script to sign the wheels with your GPG key before upload. Don’t forget to upload the wheels before the source tarball, so there is no period for which people switch from an expected binary install to a source install from PyPI.

There are two ways to update the source release on PyPI, the first one is:

$ git clean -fxd  # to be safe
$ python setup.py sdist --formats=gztar,zip  # to check
# python setup.py sdist --formats=gztar,zip upload --sign

This will ask for your key PGP passphrase, in order to sign the built source packages.

The second way is to upload the PKG_INFO file inside the sdist dir in the web interface of PyPI. The source tarball can also be uploaded through this interface.

Push the release tag and commit#

Finally, now you are confident this tag correctly defines the source code that you released you can push the tag and release commit up to github:

git push  # Push release commit
git push upstream <version>  # Push tag named <version>

where upstream points to the main https://github.com/numpy/numpy.git repository.

Update scipy.org#

A release announcement with a link to the download site should be placed in the sidebar of the front page of scipy.org.

The scipy.org should be a PR at https://github.com/scipy/scipy.org. The file that needs modification is www/index.rst. Search for News.

Update oldest-supported-numpy#

If this release is the first one to support a new Python version, or the first to provide wheels for a new platform or PyPy version, the version pinnings in https://github.com/scipy/oldest-supported-numpy should be updated. Either submit a PR with changes to setup.cfg there, or open an issue with info on needed changes.

Announce to the lists#

The release should be announced on the mailing lists of NumPy and SciPy, to python-announce, and possibly also those of Matplotlib, IPython and/or Pygame.

During the beta/RC phase, an explicit request for testing the binaries with several other libraries (SciPy/Matplotlib/Pygame) should be posted on the mailing list.

Announce to Linux Weekly News#

Email the editor of LWN to let them know of the release. Directions at: https://lwn.net/op/FAQ.lwn#contact

After the final release#

After the final release is announced, a few administrative tasks are left to be done:

Forward port changes in the release branch to release notes and release scripts, if any, to main branch.

Update the Milestones in Trac.

Step-by-Step Directions#

This file contains a walkthrough of the NumPy 1.21.0 release on Linux, modified for building on azure and uploading to anaconda.org The commands can be copied into the command line, but be sure to replace 1.21.0 by the correct version. This should be read together with the general directions in releasing.

Facility Preparation#

Before beginning to make a release, use the *_requirements.txt files to ensure that you have the needed software. Most software can be installed with pip, but some will require apt-get, dnf, or whatever your system uses for software. Note that at this time the documentation cannot be built with Python 3.10, for that use 3.8-3.9 instead. You will also need a GitHub personal access token (PAT) to push the documentation. There are a few ways to streamline things.

Git can be set up to use a keyring to store your GitHub personal access token. Search online for the details.
You can use the keyring app to store the PyPI password for twine. See the online twine documentation for details.

Release Preparation#

Backport Pull Requests#

Changes that have been marked for this release must be backported to the maintenance/1.21.x branch.

Update Release documentation#

Four documents usually need to be updated or created before making a release:

The changelog
The release-notes
The .mailmap file
The doc/source/release.rst file

These changes should be made as an ordinary PR against the maintenance branch. After release all files except doc/source/release.rst will need to be forward ported to the main branch.

Generate the changelog#

The changelog is generated using the changelog tool:

$ python tools/changelog.py $GITHUB v1.20.0..maintenance/1.21.x > doc/changelog/1.21.0-changelog.rst

where GITHUB contains your GitHub access token. The text will need to be checked for non-standard contributor names and dependabot entries removed. It is also a good idea to remove any links that may be present in the PR titles as they don’t translate well to markdown, replace them with monospaced text. The non-standard contributor names should be fixed by updating the .mailmap file, which is a lot of work. It is best to make several trial runs before reaching this point and ping the malefactors using a GitHub issue to get the needed information.

Finish the release notes#

If this is the first release in a series the release note is generated, see the release note in doc/release/upcoming_changes/README.rst to see how to do this. Generating the release notes will also delete all the news fragment files in doc/release/upcoming_changes/.

The generated release note will always need some fixups, the introduction will need to be written, and significant changes should be called out. For patch releases the changelog text may also be appended, but not for the initial release as it is too long. Check previous release notes to see how this is done. Note that the :orphan: markup at the top, if present, will need changing to .. currentmodule:: numpy and the doc/source/release.rst index file will need updating.

Check the pavement.py file#

Check that the pavement.py file points to the correct release notes. It should have been updated after the last release, but if not, fix it now:

$gvim pavement.py

Release Walkthrough#

Note that in the code snippets below, upstream refers to the root repository on GitHub and origin to its fork in your personal GitHub repositories. You may need to make adjustments if you have not forked the repository but simply cloned it locally. You can also edit .git/config and add upstream if it isn’t already present.

Prepare the release commit#

Checkout the branch for the release, make sure it is up to date, and clean the repository:

$ git checkout maintenance/1.21.x
$ git pull upstream maintenance/1.21.x
$ git submodule update
$ git clean -xdfq

Sanity check:

$ python3 runtests.py -m "full"

Tag the release and push the tag. This requires write permission for the numpy repository:

$ git tag -a -s v1.21.0 -m"NumPy 1.21.0 release"
$ git push upstream v1.21.0

Build source releases#

Paver is used to build the source releases. It will create the release and release/installers directories and put the *.zip and *.tar.gz source releases in the latter.

$ paver sdist  # sdist will do a git clean -xdfq, so we omit that

Build wheels via MacPython/numpy-wheels#

Trigger the wheels build by pointing the numpy-wheels repository at this commit. This can take up to an hour. The numpy-wheels repository is cloned from https://github.com/MacPython/numpy-wheels. If this is the first release in a series, start with a pull as the repo may have been accessed and changed by someone else, then create a new branch for the series. If the branch already exists skip this:

$ cd ../numpy-wheels
$ git checkout main
$ git pull upstream main
$ git branch v1.21.x

Checkout the new branch and edit the azure-pipelines.yml and .travis.yml files to make sure they have the correct version, and put in the commit hash for the REL commit created above for BUILD_COMMIT variable. The azure/posix.yml and .travis.yml files may also need the Cython versions updated to keep up with Python releases, but generally just do:

$ git checkout v1.21.x
$ gvim azure-pipelines.yml .travis.yml
$ git commit -a -m"NumPy 1.21.0 release."
$ git push upstream HEAD

Now wait. If you get nervous at the amount of time taken – the builds can take a while – you can check the build progress by following the links provided at https://github.com/MacPython/numpy-wheels to check the build status. Check if all the needed wheels have been built and uploaded to the staging repository before proceeding.

Note that sometimes builds, like tests, fail for unrelated reasons and you will need to rerun them. You will need to be logged in under ‘numpy’ to do this on azure.

Build wheels via cibuildwheel#

Tagging the build at the beginning of this process will trigger a wheel build via cibuildwheel and upload wheels and an sdist to the staging area. The CI run on github actions (for all x86-based and macOS arm64 wheels) takes about 1 1/4 hours. The CI run on travis (for aarch64) takes less time.

If you wish to manually trigger a wheel build, you can do so:

On github actions -> Wheel builder there is a “Run workflow” button, click on it and choose the tag to build
On travis there is a “More Options” button, click on it and choose a branch to build. There does not appear to be an option to build a tag.

Download wheels#

When the wheels have all been successfully built and staged, download them from the Anaconda staging directory using the tools/download-wheels.py script:

$ cd ../numpy
$ python3 tools/download-wheels.py 1.21.0

Generate the README files#

This needs to be done after all installers are downloaded, but before the pavement file is updated for continued development:

$ paver write_release

Reset the maintenance branch into a development state (skip for prereleases)#

Create release notes for next release and edit them to set the version. These notes will be a skeleton and have little content:

$ cp doc/source/release/template.rst doc/source/release/1.21.1-notes.rst
$ gvim doc/source/release/1.21.1-notes.rst
$ git add doc/source/release/1.21.1-notes.rst

Add new release notes to the documentation release list and update the RELEASE_NOTES variable in pavement.py.

$ gvim doc/source/release.rst pavement.py

Commit the result:

$ git commit -a -m"REL: prepare 1.21.x for further development"
$ git push upstream HEAD

Upload to PyPI#

Upload to PyPI using twine. A recent version of twine of is needed after recent PyPI changes, version 3.4.1 was used here:

$ cd ../numpy
$ twine upload release/installers/*.whl
$ twine upload release/installers/numpy-1.21.0.zip  # Upload last.

If one of the commands breaks in the middle, you may need to selectively upload the remaining files because PyPI does not allow the same file to be uploaded twice. The source file should be uploaded last to avoid synchronization problems that might occur if pip users access the files while this is in process, causing pip to build from source rather than downloading a binary wheel. PyPI only allows a single source distribution, here we have chosen the zip archive.

Upload files to github#

Go to https://github.com/numpy/numpy/releases, there should be a v1.21.0 tag, click on it and hit the edit button for that tag. There are two ways to add files, using an editable text window and as binary uploads. Start by editing the release/README.md that is translated from the rst version using pandoc. Things that will need fixing: PR lines from the changelog, if included, are wrapped and need unwrapping, links should be changed to monospaced text. Then copy the contents to the clipboard and paste them into the text window. It may take several tries to get it look right. Then

Upload release/installers/numpy-1.21.0.tar.gz as a binary file.
Upload release/installers/numpy-1.21.0.zip as a binary file.
Upload release/README.rst as a binary file.
Upload doc/changelog/1.21.0-changelog.rst as a binary file.
Check the pre-release button if this is a pre-releases.
Hit the {Publish,Update} release button at the bottom.

Upload documents to numpy.org (skip for prereleases)#

Note

You will need a GitHub personal access token to push the update.

This step is only needed for final releases and can be skipped for pre-releases and most patch releases. make merge-doc clones the numpy/doc repo into doc/build/merge and updates it with the new documentation. If you already have a numpy installed, you need to locally install the new NumPy version so that document generation will use the correct NumPy. This is because make dist does not correctly set up the path. Note that Python 3.10 cannot be used for generating the docs as it has no easy_install, use 3.9 or 3.8 instead:

$ pushd doc
$ make dist
$ make merge-doc
$ pushd build/merge

If the release series is a new one, you will need to add a new section to the doc/build/merge/index.html front page just after the “insert here” comment:

$ gvim index.html +/'insert here'

Further, update the version-switcher json file to add the new release and update the version marked (stable):

$ gvim _static/versions.json

Otherwise, only the zip and pdf links should be updated with the new tag name:

$ gvim index.html +/'tag v1.21'

You can “test run” the new documentation in a browser to make sure the links work:

$ firefox index.html  # or google-chrome, etc.

Update the stable link and update:

$ ln -sfn 1.21 stable
$ ls -l  # check the link

Once everything seems satisfactory, update, commit and upload the changes:

$ python3 update.py
$ git commit -a -m"Add documentation for v1.21.0"
$ git push
$ popd
$ popd

Announce the release on numpy.org (skip for prereleases)#

This assumes that you have forked https://github.com/numpy/numpy.org:

$ cd ../numpy.org
$ git checkout master
$ git pull upstream master
$ git checkout -b announce-numpy-1.21.0
$ gvim content/en/news.md

For all releases, go to the bottom of the page and add a one line link. Look to the previous links for example.
For the *.0 release in a cycle, add a new section at the top with a short description of the new features and point the news link to it.

commit and push:

$ git commit -a -m"announce the NumPy 1.21.0 release"
$ git push origin HEAD

Go to your Github fork and make a pull request.

Announce to mailing lists#

The release should be announced on the numpy-discussion, scipy-devel, scipy-user, and python-announce-list mailing lists. Look at previous announcements for the basic template. The contributor and PR lists are the same as generated for the release notes above. If you crosspost, make sure that python-announce-list is BCC so that replies will not be sent to that list.

Post-Release Tasks (skip for prereleases)#

Checkout main and forward port the documentation changes:

$ git checkout -b post-1.21.0-release-update
$ git checkout maintenance/1.21.x doc/source/release/1.21.0-notes.rst
$ git checkout maintenance/1.21.x doc/changelog/1.21.0-changelog.rst
$ git checkout maintenance/1.21.x .mailmap  # only if updated for release.
$ gvim doc/source/release.rst  # Add link to new notes
$ git add doc/changelog/1.21.0-changelog.rst doc/source/release/1.21.0-notes.rst
$ git status  # check status before commit
$ git commit -a -m"REL: Update main after 1.21.0 release."
$ git push origin HEAD

Go to GitHub and make a PR.

NumPy benchmarks

NumPy governance

Releasing a version#

How to Prepare a Release#

Current build and release info#

Source tree#

NumPy Docs#

SciPy.org wiki#

Release Scripts#

Supported platforms and versions#

OS X#

Windows#

Linux#

BSD / Solaris#

Tool chain#

Compilers#

OpenBLAS#

Building source archives and wheels#

Building docs#

Uploading to PyPI#

Generating author/pr lists#

Virtualenv#

What is released#

Wheels#

Other#

Source distribution#

Release process#

Agree on a release schedule#

Make sure current branch builds a package correctly#

Check deprecations#

Check the C API version number#

Check the release notes#

Update the release status and create a release “tag”#

Update the version of the main branch#

Trigger the wheel builds#

Make the release#

Build and archive documentation#

Update PyPI#

Push the release tag and commit#

Update scipy.org#

Update oldest-supported-numpy#

Announce to the lists#

Announce to Linux Weekly News#

After the final release#

Step-by-Step Directions#

Facility Preparation#

Release Preparation#

Backport Pull Requests#

Update Release documentation#

Generate the changelog#

Finish the release notes#

Check the pavement.py file#

Release Walkthrough#

Prepare the release commit#

Build source releases#

Build wheels via MacPython/numpy-wheels#

Build wheels via cibuildwheel#

Download wheels#

Generate the README files#

Reset the maintenance branch into a development state (skip for prereleases)#

Upload to PyPI#

Upload files to github#

Upload documents to numpy.org (skip for prereleases)#

Announce the release on numpy.org (skip for prereleases)#

Announce to mailing lists#

Post-Release Tasks (skip for prereleases)#