This guide is intended for developers or administrators who want to contribute a new package, feature, or bugfix to Spack. It assumes that you have at least some familiarity with Git VCS and Github. The guide will show a few examples of contributing workflows and discuss the granularity of pull-requests (PRs). It will also discuss the tests your PR must pass in order to be accepted into Spack.
First, what is a PR? Quoting Bitbucket’s tutorials:
Pull requests are a mechanism for a developer to notify team members that they have completed a feature. The pull request is more than just a notification—it’s a dedicated forum for discussing the proposed feature.
Important is completed feature. The changes one proposes in a PR should correspond to one feature/bugfix/extension/etc. One can create PRs with changes relevant to different ideas, however reviewing such PRs becomes tedious and error prone. If possible, try to follow the one-PR-one-package/feature rule.
develop branch has the latest contributions. Nearly all pull
requests should start from
develop and target
There is a branch for each major release series. Release branches
develop and have tags for each point release in the
series. For example,
releases/v0.14 has tags for
0.14.2, etc. versions of Spack. We backport important bug
fixes to these branches, but we do not advance the package versions or
make other changes that would change the way Spack concretizes
dependencies. Currently, the maintainers manage these branches by
develop. See Releases for more
Spack uses Github Actions for Continuous Integration testing. This means that every time you submit a pull request, a series of tests will be run to make sure you didn’t accidentally introduce any bugs into Spack. Your PR will not be accepted until it passes all of these tests. While you can certainly wait for the results of these tests after submitting a PR, we recommend that you run them locally to speed up the review process.
Oftentimes, CI will fail for reasons other than a problem with your PR. For example, apt-get, pip, or homebrew will fail to download one of the dependencies for the test suite, or a transient bug will cause the unit tests to timeout. If any job fails, click the “Details” link and click on the test(s) that is failing. If it doesn’t look like it is failing for reasons related to your PR, you have two options. If you have write permissions for the Spack repository, you should see a “Restart workflow” button on the right-hand side. If not, you can close and reopen your PR to rerun all of the tests. If the same test keeps failing, there may be a problem with your PR. If you notice that every recent PR is failing with the same error message, it may be that an issue occurred with the CI infrastructure or one of Spack’s dependencies put out a new release that is causing problems. If this is the case, please file an issue.
We currently test against Python 2.7 and 3.6-3.10 on both macOS and Linux and perform 3 types of tests:
Unit tests ensure that core Spack features like fetching or spec resolution are working as expected. If your PR only adds new packages or modifies existing ones, there’s very little chance that your changes could cause the unit tests to fail. However, if you make changes to Spack’s core libraries, you should run the unit tests to make sure you didn’t break anything.
Since they test things like fetching from VCS repos, the unit tests require
and subversion to run. Make sure these are
installed on your system and can be found in your
PATH. All of these can be
installed with Spack or with your system package manager.
To run all of the unit tests, use:
$ spack unit-test
These tests may take several minutes to complete. If you know you are
only modifying a single Spack feature, you can run subsets of tests at a
time. For example, this would run all the tests in
$ spack unit-test lib/spack/spack/test/architecture.py
And this would run the
test_platform test from that file:
$ spack unit-test lib/spack/spack/test/architecture.py::test_platform
This allows you to develop iteratively: make a change, test that change,
make another change, test that change, etc. We use pytest as our tests framework, and these types of
arguments are just passed to the
pytest command underneath. See the
for more details on test selection syntax.
spack unit-test has a few special options that can help you
understand what tests are available. To get a list of all available
unit test files, run:
$ spack unit-test --list ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spec.json ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/patchelf-0.16.1/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spack ==> Installing "email@example.comfirstname.lastname@example.org ldflags="-static-libstdc++ -static-libgcc" build_system=autotools arch=linux-centos7-x86_64" from a buildcache ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-axtxtavfahxuazw2wueu3tjwwu6ttdfo.spec.json ==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-axtxtavfahxuazw2wueu3tjwwu6ttdfo.spack ...
To see a more detailed list of available unit tests, use
$ spack unit-test --list-long lib/spack/spack/test/abi.py:: test_architecture_compatibility test_compatibility test_compiler_compatibility lib/spack/spack/test/architecture.py:: test_arch_spec_container_semantic test_optimization_flags_with_custom_versions test_concretize_target_ranges test_platform test_cray_platform_detection test_satisfy_strict_constraint_when_not_concrete test_default_os_and_target test_target_container_semantic test_operating_system_conversion_to_dict test_user_input_combination test_optimization_flags ...
And to see the fully qualified names of all tests, use
$ spack unit-test --list-names lib/spack/spack/test/abi.py::test_architecture_compatibility lib/spack/spack/test/abi.py::test_compatibility lib/spack/spack/test/abi.py::test_compiler_compatibility lib/spack/spack/test/architecture.py::test_arch_spec_container_semantic lib/spack/spack/test/architecture.py::test_concretize_target_ranges ...
You can combine these with
pytest arguments to restrict which tests
you want to know about. For example, to see just the tests in
$ spack unit-test --list-long lib/spack/spack/test/architecture.py lib/spack/spack/test/architecture.py:: test_arch_spec_container_semantic test_optimization_flags_with_custom_versions test_concretize_target_ranges test_platform test_cray_platform_detection test_satisfy_strict_constraint_when_not_concrete test_default_os_and_target test_target_container_semantic test_operating_system_conversion_to_dict test_user_input_combination test_optimization_flags
You can also combine any of these options with a
search. See the pytest usage docs
for more details on test selection syntax. For example, to see the names of all tests that have “spec”
or “concretize” somewhere in their names:
$ spack unit-test --list-names -k "spec and concretize" lib/spack/spack/test/cmd/spec.py::test_spec_concretizer_args lib/spack/spack/test/concretize.py::TestConcretize::test_best_effort_coconcretize lib/spack/spack/test/concretize.py::TestConcretize::test_best_effort_coconcretize_preferences lib/spack/spack/test/concretize.py::TestConcretize::test_concrete_specs_are_not_modified_on_reuse lib/spack/spack/test/concretize.py::TestConcretize::test_conflicts_in_spec lib/spack/spack/test/concretize.py::TestConcretize::test_external_package_versions lib/spack/spack/test/concretize.py::TestConcretize::test_find_spec_children lib/spack/spack/test/concretize.py::TestConcretize::test_find_spec_none lib/spack/spack/test/concretize.py::TestConcretize::test_find_spec_parents lib/spack/spack/test/concretize.py::TestConcretize::test_find_spec_self lib/spack/spack/test/concretize.py::TestConcretize::test_find_spec_sibling lib/spack/spack/test/concretize.py::TestConcretize::test_git_ref_version_is_equivalent_to_specified_version lib/spack/spack/test/concretize.py::TestConcretize::test_installed_specs_disregard_conflicts lib/spack/spack/test/concretize.py::TestConcretize::test_mv_variants_disjoint_sets_from_spec lib/spack/spack/test/concretize.py::TestConcretize::test_no_conflict_in_external_specs lib/spack/spack/test/concretize.py::TestConcretize::test_no_matching_compiler_specs lib/spack/spack/test/concretize.py::TestConcretize::test_result_specs_is_not_empty lib/spack/spack/test/concretize.py::TestConcretize::test_reuse_does_not_overwrite_dev_specs lib/spack/spack/test/concretize.py::TestConcretize::test_simultaneous_concretization_of_specs lib/spack/spack/test/concretize_requirements.py::test_default_and_package_specific_requirements lib/spack/spack/test/concretize_requirements.py::test_multiple_packages_requirements_are_respected lib/spack/spack/test/env.py::test_concretize_user_specs_together lib/spack/spack/test/env.py::test_virtual_spec_concretize_together lib/spack/spack/test/mirror.py::TestMirrorCreate::test_all_specs_with_all_versions_dont_concretize lib/spack/spack/test/spec_dag.py::TestSpecDag::test_concretize_deptypes lib/spack/spack/test/spec_dag.py::TestSpecDag::test_copy_concretized lib/spack/spack/test/spec_semantics.py::test_concretize_partial_old_dag_hash_spec lib/spack/spack/test/spec_semantics.py::test_intersects_and_satisfies_on_concretized_spec
pytest captures the output of all unit tests, and it will
print any captured output for failed tests. Sometimes it’s helpful to see
your output interactively, while the tests run (e.g., if you add print
statements to a unit tests). To see the output live, use the
$ spack unit-test -s --list-long lib/spack/spack/test/architecture.py::test_platform
Unit tests are crucial to making sure bugs aren’t introduced into Spack. If you are modifying core Spack libraries or adding new functionality, please add new unit tests for your feature, and consider strengthening existing tests. You will likely be asked to do this if you submit a pull request to the Spack project on GitHub. Check out the pytest docs and feel free to ask for guidance on how to write tests!
You may notice the
share/spack/qa/run-unit-tests script in the
repository. This script is designed for CI. It runs the unit
tests and reports coverage statistics back to Codecov. If you want to
run the unit tests yourself, we suggest you use
Spack uses Flake8 to test for PEP 8 conformance and mypy <https://mypy.readthedocs.io/en/stable/> for type checking. PEP 8 is a series of style guides for Python that provide suggestions for everything from variable naming to indentation. In order to limit the number of PRs that were mostly style changes, we decided to enforce PEP 8 conformance. Your PR needs to comply with PEP 8 in order to be accepted, and if it modifies the spack library it needs to successfully type-check with mypy as well.
Testing for compliance with spack’s style is easy. Simply run the
$ spack style
spack style has a couple advantages over running the tools by hand:
It only tests files that you have modified since branching off of
It works regardless of what directory you are in.
It automatically adds approved exemptions from the
flake8checks. For example, URLs are often longer than 80 characters, so we exempt them from line length checks. We also exempt lines that start with “homepage”, “url”, “version”, “variant”, “depends_on”, and “extends” in
package.pyfiles. This is now also possible when directly running flake8 if you can use the
spackformatter plugin included with spack.
More approved flake8 exemptions can be found here.
If all is well, you’ll see something like this:
$ run-flake8-tests Dependencies found. ======================================================= flake8: running flake8 code checks on spack. Modified files: var/spack/repos/builtin/packages/hdf5/package.py var/spack/repos/builtin/packages/hdf/package.py var/spack/repos/builtin/packages/netcdf/package.py ======================================================= Flake8 checks were clean.
However, if you aren’t compliant with PEP 8, flake8 will complain:
var/spack/repos/builtin/packages/netcdf/package.py:26: [F401] 'os' imported but unused var/spack/repos/builtin/packages/netcdf/package.py:61: [E303] too many blank lines (2) var/spack/repos/builtin/packages/netcdf/package.py:106: [E501] line too long (92 > 79 characters) Flake8 found errors.
Most of the error messages are straightforward, but if you don’t understand what
they mean, just ask questions about them when you submit your PR. The line numbers
will change if you add or delete lines, so simply run
spack style again
to update them.
Try fixing flake8 errors in reverse order. This eliminates the need for
multiple runs of
spack style just to re-compute line numbers and
makes it much easier to fix errors directly off of the CI output.
Spack uses Sphinx to build its documentation. In order to prevent things like broken links and missing imports, we added documentation tests that build the documentation and fail if there are any warning or error messages.
Building the documentation requires several dependencies:
All of these can be installed with Spack, e.g.
$ spack install py-sphinx py-sphinxcontrib-programoutput py-sphinx-rtd-theme graphviz git mercurial subversion
Sphinx has several required dependencies.
If you’re using a
python from Spack and you installed
py-sphinx and friends, you need to make them available to your
python. The easiest way to do this is to run:
$ spack load py-sphinx py-sphinx-rtd-theme py-sphinxcontrib-programoutput
so that all of the dependencies are added to PYTHONPATH. If you see an error message like:
Extension error: Could not import extension sphinxcontrib.programoutput (exception: No module named sphinxcontrib.programoutput) make: *** [html] Error 1
that means Sphinx couldn’t find
py-sphinxcontrib-programoutput in your
Once all of the dependencies are installed, you can try building the documentation:
$ cd path/to/spack/lib/spack/docs/ $ make clean $ make
If you see any warning or error messages, you will have to correct those before your PR is accepted.
If you are editing the documentation, you should obviously be running the documentation tests. But even if you are simply adding a new package, your changes could cause the documentation tests to fail:
package_list.rst:8745: WARNING: Block quote ends without a blank line; unexpected unindent.
At first, this error message will mean nothing to you, since you didn’t edit that file. Until you look at line 8745 of the file in question:
Description: NetCDF is a set of software libraries and self-describing, machine- independent data formats that support the creation, access, and sharing of array-oriented scientific data.
Our documentation includes a list of all Spack packages. If you add a new package, its docstring is added to this page. The problem in this case was that the docstring looked like:
class Netcdf(Package): """ NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. """
Docstrings cannot start with a newline character, or else Sphinx will complain. Instead, they should look like:
class Netcdf(Package): """NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data."""
Documentation changes can result in much more obfuscated warning messages. If you don’t understand what they mean, feel free to ask when you submit your PR.
Spack uses Codecov to generate and report unit test coverage. This helps us tell what percentage of lines of code in Spack are covered by unit tests. Although code covered by unit tests can still contain bugs, it is much less error prone than code that is not covered by unit tests.
Codecov provides browser extensions for Google Chrome and Firefox. These extensions integrate with GitHub and allow you to see coverage line-by-line when viewing the Spack repository. If you are new to Spack, a great way to get started is to write unit tests to increase coverage!
Unlike with CI on Github Actions Codecov tests are not required to pass in order for your PR to be merged. If you modify core Spack libraries, we would greatly appreciate unit tests that cover these changed lines. Otherwise, we have no way of knowing whether or not your changes introduce a bug. If you make substantial changes to the core, we may request unit tests to increase coverage.
If the only files you modified are package files, we do not care about coverage on your PR. You may notice that the Codecov tests fail even though you didn’t modify any core files. This means that Spack’s overall coverage has increased since you branched off of develop. This is a good thing! If you really want to get the Codecov tests to pass, you can rebase off of the latest develop, but again, this is not required.
Spack is still in the beta stages of development. Most of our users run off of the develop branch, and fixes and new features are constantly being merged. So how do you keep up-to-date with upstream while maintaining your own local differences and contributing PRs to Spack?
The easiest way to contribute a pull request is to make all of your changes on
new branches. Make sure your
develop is up-to-date and create a new branch
off of it:
$ git checkout develop $ git pull upstream develop $ git branch <descriptive_branch_name> $ git checkout <descriptive_branch_name>
Here we assume that the local
develop branch tracks the upstream develop
branch of Spack. This is not a requirement and you could also do the same with
remote branches. But for some it is more convenient to have a local branch that
Normally we prefer that commits pertaining to a package
<package-name>: descriptive message. It is important to add
descriptive message so that others, who might be looking at your changes later
(in a year or maybe two), would understand the rationale behind them.
Now, you can make your changes while keeping the
develop branch pure.
Edit a few files and commit them by running:
$ git add <files_to_be_part_of_the_commit> $ git commit --message <descriptive_message_of_this_particular_commit>
Next, push it to your remote fork and create a PR:
$ git push origin <descriptive_branch_name> --set-upstream
GitHub provides a tutorial
on how to file a pull request. When you send the request, make
If you need this change immediately and don’t have time to wait for your PR to be merged, you can always work on this branch. But if you have multiple PRs, another option is to maintain a Frankenstein branch that combines all of your other branches:
$ git co develop $ git branch <your_modified_develop_branch> $ git checkout <your_modified_develop_branch> $ git merge <descriptive_branch_name>
This can be done with each new PR you submit. Just make sure to keep this local
branch up-to-date with upstream
What if you made some changes to your local modified develop branch and already committed them, but later decided to contribute them to Spack? You can use cherry-picking to create a new branch with only these commits.
First, check out your local modified develop branch:
$ git checkout <your_modified_develop_branch>
Now, get the hashes of the commits you want from the output of:
$ git log
Next, create a new branch off of upstream
develop and copy the commits
that you want in your PR:
$ git checkout develop $ git pull upstream develop $ git branch <descriptive_branch_name> $ git checkout <descriptive_branch_name> $ git cherry-pick <hash> $ git push origin <descriptive_branch_name> --set-upstream
Now you can create a PR from the web-interface of GitHub. The net result is as follows:
You patched your local version of Spack and can use it further.
You “cherry-picked” these changes in a stand-alone branch and submitted it as a PR upstream.
Should you have several commits to contribute, you could follow the same procedure by getting hashes of all of them and cherry-picking to the PR branch.
It is important that whenever you change something that might be of importance upstream, create a pull request as soon as possible. Do not wait for weeks/months to do this, because:
you might forget why you modified certain files
it could get difficult to isolate this change into a stand-alone clean PR.
Other developers are constantly making contributions to Spack, possibly on the
same files that your PR changed. If their PR is merged before yours, it can
create a merge conflict. This means that your PR can no longer be automatically
merged without a chance of breaking your changes. In this case, you will be
asked to rebase on top of the latest upstream
First, make sure your develop branch is up-to-date:
$ git checkout develop $ git pull upstream develop
Now, we need to switch to the branch you submitted for your PR and rebase it on top of develop:
$ git checkout <descriptive_branch_name> $ git rebase develop
Git will likely ask you to resolve conflicts. Edit the file that it says can’t be merged automatically and resolve the conflict. Then, run:
$ git add <file_that_could_not_be_merged> $ git rebase --continue
You may have to repeat this process multiple times until all conflicts are resolved. Once this is done, simply force push your rebased branch to your remote fork:
$ git push --force origin <descriptive_branch_name>
Rebasing with cherry-pick¶
You can also perform a rebase using
cherry-pick. First, create a temporary
$ git checkout <descriptive_branch_name> $ git branch tmp
If anything goes wrong, you can always go back to your
Now, look at the logs and save the hashes of any commits you would like to keep:
$ git log
Next, go back to the original branch and reset it to
Before doing so, make sure that you local
develop branch is up-to-date
$ git checkout develop $ git pull upstream develop $ git checkout <descriptive_branch_name> $ git reset --hard develop
Now you can cherry-pick relevant commits:
$ git cherry-pick <hash1> $ git cherry-pick <hash2>
Push the modified branch to your fork:
$ git push --force origin <descriptive_branch_name>
If everything looks good, delete the backup branch:
$ git branch --delete --force tmp
Sometimes you may end up on a branch that has diverged so much from develop that it cannot easily be rebased. If the current commits history is more of an experimental nature and only the net result is important, you may rewrite the history.
First, merge upstream
develop and reset you branch to it. On the branch
in question, run:
$ git merge develop $ git reset develop
At this point your branch will point to the same commit as develop and thereby the two are indistinguishable. However, all the files that were previously modified will stay as such. In other words, you do not lose the changes you made. Changes can be reviewed by looking at diffs:
$ git status $ git diff
The next step is to rewrite the history by adding files and creating commits:
$ git add <files_to_be_part_of_commit> $ git commit --message <descriptive_message>
After all changed files are committed, you can push the branch to your fork and create a PR:
$ git push origin --set-upstream