The process of using Spack involves building packages, running binaries from those packages, and developing software that depends on those packages. For example, one might use Spack to build the netcdf package, use spack load to run the ncdump binary, and finally, write a small C program to read/write a particular NetCDF file.

Spack supports a variety of workflows to suit a variety of situations and user preferences, there is no single way to do all these things. This chapter demonstrates different workflows that have been developed, pointing out the pros and cons of them.


First some basic definitions.

Package, Concrete Spec, Installed Package

In Spack, a package is an abstract recipe to build one piece of software. Spack packages may be used to build, in principle, any version of that software with any set of variants. Examples of packages include curl and zlib.

A package may be instantiated to produce a concrete spec; one possible realization of a particular package, out of combinatorially many other realizations. For example, here is a concrete spec instantiated from curl:

$ spack spec curl
Input spec

curl@7.79.0%gcc@7.5.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 tls=openssl arch=linux-ubuntu18.04-skylake_avx512
    ^openssl@1.1.1l%gcc@7.5.0~docs certs=system arch=linux-ubuntu18.04-skylake_avx512
        ^perl@5.34.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-skylake_avx512
            ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-ubuntu18.04-skylake_avx512
            ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-skylake_avx512
                ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-skylake_avx512
                    ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-skylake_avx512
            ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-skylake_avx512
                ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-skylake_avx512
                    ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-skylake_avx512
                        ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-skylake_avx512
            ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-skylake_avx512

Spack’s core concretization algorithm generates concrete specs by instantiating packages from its repo, based on a set of “hints”, including user input and the packages.yaml file. This algorithm may be accessed at any time with the spack spec command.

Every time Spack installs a package, that installation corresponds to a concrete spec. Only a vanishingly small fraction of possible concrete specs will be installed at any one Spack site.

Consistent Sets

A set of Spack specs is said to be consistent if each package is only instantiated one way within it — that is, if two specs in the set have the same package, then they must also have the same version, variant, compiler, etc. For example, the following set is consistent:

curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
    ^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64
    ^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64

The following set is not consistent:

curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
    ^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64
    ^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
zlib@1.2.7%gcc@5.3.0 arch=linux-SuSE11-x86_64

The compatibility of a set of installed packages determines what may be done with it. It is always possible to spack load any set of installed packages, whether or not they are consistent, and run their binaries from the command line. However, a set of installed packages can only be linked together in one binary if it is consistent.

If the user produces a series of spack spec or spack load commands, in general there is no guarantee of consistency between them. Spack’s concretization procedure guarantees that the results of any single spack spec call will be consistent. Therefore, the best way to ensure a consistent set of specs is to create a Spack package with dependencies, and then instantiate that package. We will use this technique below.

Building Packages

Suppose you are tasked with installing a set of software packages on a system in order to support one application – both a core application program, plus software to prepare input and analyze output. The required software might be summed up as a series of spack install commands placed in a script. If needed, this script can always be run again in the future. For example:

spack install modele-utils
spack install emacs
spack install ncview
spack install nco
spack install modele-control
spack install py-numpy

In most cases, this script will not correctly install software according to your specific needs: choices need to be made for variants, versions and virtual dependency choices may be needed. It is possible to specify these choices by extending specs on the command line; however, the same choices must be specified repeatedly. For example, if you wish to use openmpi to satisfy the mpi dependency, then ^openmpi will have to appear on every spack install line that uses MPI. It can get repetitive fast.

Customizing Spack installation options is easier to do in the ~/.spack/packages.yaml file. In this file, you can specify preferred versions and variants to use for packages. For example:

        version: [3.5.1]
        version: [cmake]

        version: [develop]
        variants: ~suitesparse
        variants: +mpi

        compiler: [gcc@5.3.0]
            mpi: [openmpi]
            blas: [openblas]
            lapack: [openblas]

This approach will work as long as you are building packages for just one application.

Multiple Applications

Suppose instead you’re building multiple inconsistent applications. For example, users want package A to be built with openmpi and package B with mpich — but still share many other lower-level dependencies. In this case, a single packages.yaml file will not work. Plans are to implement per-project packages.yaml files. In the meantime, one could write shell scripts to switch packages.yaml between multiple versions as needed, using symlinks.

Combinatorial Sets of Installs

Suppose that you are now tasked with systematically building many incompatible versions of packages. For example, you need to build petsc 9 times for 3 different MPI implementations on 3 different compilers, in order to support user needs. In this case, you will need to either create 9 different packages.yaml files; or more likely, create 9 different spack install command lines with the correct options in the spec. Here is a real-life example of this kind of usage:




for compiler in "${compilers[@]}"
    # Serial installs
    spack install szip           $compiler
    spack install hdf            $compiler
    spack install hdf5           $compiler
    spack install netcdf         $compiler
    spack install netcdf-fortran $compiler
    spack install ncview         $compiler

    # Parallel installs
    for mpi in "${mpis[@]}"
        spack install $mpi            $compiler
        spack install hdf5~cxx+mpi    $compiler ^$mpi
        spack install parallel-netcdf $compiler ^$mpi

Running Binaries from Packages

Once Spack packages have been built, the next step is to use them. As with building packages, there are many ways to use them, depending on the use case.

Find and Run

The simplest way to run a Spack binary is to find it and run it! In many cases, nothing more is needed because Spack builds binaries with RPATHs. Spack installation directories may be found with spack location --install-dir commands. For example:

$ spack location --install-dir cmake

This gives the root of the Spack package; relevant binaries may be found within it. For example:

$ CMAKE=`spack location --install-dir cmake`/bin/cmake

Standard UNIX tools can find binaries as well. For example:

$ find ~/spack/opt -name cmake | grep bin

These methods are suitable, for example, for setting up build processes or GUIs that need to know the location of particular tools. However, other more powerful methods are generally preferred for user environments.

Using spack load to Manage the User Environment

Suppose that Spack has been used to install a set of command-line programs, which users now wish to use. One can in principle put a number of spack load commands into .bashrc, for example, to load a set of Spack packages:

spack load modele-utils
spack load emacs
spack load ncview
spack load nco
spack load modele-control

Although simple load scripts like this are useful in many cases, they have some drawbacks:

  1. The set of packages loaded by them will in general not be consistent. They are a decent way to load commands to be called from command shells. See below for better ways to assemble a consistent set of packages for building application programs.

  2. The spack spec and spack install commands use a sophisticated concretization algorithm that chooses the “best” among several options, taking into account packages.yaml file. The spack load and spack module tcl loads commands, on the other hand, are not very smart: if the user-supplied spec matches more than one installed package, then spack module tcl loads will fail. This default behavior may change in the future. For now, the workaround is to either be more specific on any failing spack load commands or to use spack load --first to allow spack to load the first matching spec.

Generated Load Scripts

Another problem with using spack load is, it can be slow; a typical user environment could take several seconds to load, and would not be appropriate to put into .bashrc directly. This is because it requires the full start-up overhead of python/Spack for each command. In some circumstances it is preferable to use a series of spack module tcl loads (or spack module lmod loads) commands to pre-compute which modules to load. This will generate the modulenames to load the packages using environment modules, rather than Spack’s built-in support for environment modifications. These can be put in a script that is run whenever installed Spack packages change. For example:

# Generate module load commands in ~/env/spackenv

cat <<EOF | /bin/sh >$HOME/env/spackenv
FIND='spack module tcl loads --prefix linux-SuSE11-x86_64/'

\$FIND modele-utils
\$FIND emacs
\$FIND ncview
\$FIND nco
\$FIND modele-control

The output of this file is written in ~/env/spackenv:

# binutils@2.25%gcc@5.3.0+gold~krellpatch~libiberty arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/binutils-2.25-gcc-5.3.0-6w5d2t4
# python@2.7.12%gcc@5.3.0~tk~ucs4 arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/python-2.7.12-gcc-5.3.0-2azoju2
# ncview@2.1.7%gcc@5.3.0 arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/ncview-2.1.7-gcc-5.3.0-uw3knq2
# nco@4.5.5%gcc@5.3.0 arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/nco-4.5.5-gcc-5.3.0-7aqmimu
# modele-control@develop%gcc@5.3.0 arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/modele-control-develop-gcc-5.3.0-7rddsij
# zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/zlib-1.2.8-gcc-5.3.0-fe5onbi
# curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/curl-7.50.1-gcc-5.3.0-4vlev55
# hdf5@1.10.0-patch1%gcc@5.3.0+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/hdf5-1.10.0-patch1-gcc-5.3.0-pwnsr4w
# netcdf@4.4.1%gcc@5.3.0~hdf4+mpi arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/netcdf-4.4.1-gcc-5.3.0-rl5canv
# netcdf-fortran@4.4.4%gcc@5.3.0 arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/netcdf-fortran-4.4.4-gcc-5.3.0-stdk2xq
# modele-utils@cmake%gcc@5.3.0+aux+diags+ic arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/modele-utils-cmake-gcc-5.3.0-idyjul5
# everytrace@develop%gcc@5.3.0+fortran+mpi arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/everytrace-develop-gcc-5.3.0-p5wmb25

Users may now put source ~/env/spackenv into .bashrc.


Some module systems put a prefix on the names of modules created by Spack. For example, that prefix is linux-SuSE11-x86_64/ in the above case. If a prefix is not needed, you may omit the --prefix flag from spack module tcl loads.

Transitive Dependencies

In the script above, each spack module tcl loads command generates a single module load line. Transitive dependencies do not usually need to be loaded, only modules the user needs in $PATH. This is because Spack builds binaries with RPATH. Spack’s RPATH policy has some nice features:

  1. Modules for multiple inconsistent applications may be loaded simultaneously. In the above example (Multiple Applications), package A and package B can coexist together in the user’s $PATH, even though they use different MPIs.

  2. RPATH eliminates a whole class of strange errors that can happen in non-RPATH binaries when the wrong LD_LIBRARY_PATH is loaded.

  3. Recursive module systems such as LMod are not necessary.

  4. Modules are not needed at all to execute binaries. If a path to a binary is known, it may be executed. For example, the path for a Spack-built compiler can be given to an IDE without requiring the IDE to load that compiler’s module.

Unfortunately, Spack’s RPATH support does not work in every case. For example:

  1. Software comes in many forms — not just compiled ELF binaries, but also as interpreted code in Python, R, JVM bytecode, etc. Those systems almost universally use an environment variable analogous to LD_LIBRARY_PATH to dynamically load libraries.

  2. Although Spack generally builds binaries with RPATH, it does not currently do so for compiled Python extensions (for example, py-numpy). Any libraries that these extensions depend on (blas in this case, for example) must be specified in the LD_LIBRARY_PATH.`

  3. In some cases, Spack-generated binaries end up without a functional RPATH for no discernible reason.

In cases where RPATH support doesn’t make things “just work,” it can be necessary to load a module’s dependencies as well as the module itself. This is done by adding the --dependencies flag to the spack module tcl loads command. For example, the following line, added to the script above, would be used to load SciPy, along with Numpy, core Python, BLAS/LAPACK and anything else needed:

spack module tcl loads --dependencies py-scipy

Dummy Packages

As an alternative to a series of module load commands, one might consider dummy packages as a way to create a consistent set of packages that may be loaded as one unit. The idea here is pretty simple:

  1. Create a package (say, mydummy) with no URL and no install() method, just dependencies.

  2. Run spack install mydummy to install.

An advantage of this method is the set of packages produced will be consistent. This means that you can reliably build software against it. A disadvantage is the set of packages will be consistent; this means you cannot load up two applications this way if they are not consistent with each other.

Filesystem Views

Filesystem views offer an alternative to environment modules, another way to assemble packages in a useful way and load them into a user’s environment.

A single-prefix filesystem view is a single directory tree that is the union of the directory hierarchies of a number of installed packages; it is similar to the directory hierarchy that might exist under /usr/local. The files of the view’s installed packages are brought into the view by symbolic or hard links, referencing the original Spack installation.

A combinatorial filesystem view can contain more software than a single-prefix view. Combinatorial filesystem views are created by defining a projection for each spec or set of specs. The syntax for this will be discussed in the section for the spack view command under adding_projections_to_views.

The projection for a spec or set of specs specifies the naming scheme for the directory structure under the root of the view into which the package will be linked. For example, the spec zlib@1.2.8%gcc@4.4.7 could be projected to MYVIEW/zlib-1.2.8-gcc.

When software is built and installed, absolute paths are frequently “baked into” the software, making it non-relocatable. This happens not just in RPATHs, but also in shebangs, configuration files, and assorted other locations.

Therefore, programs run out of a Spack view will typically still look in the original Spack-installed location for shared libraries and other resources. This behavior is not easily changed; in general, there is no way to know where absolute paths might be written into an installed package, and how to relocate it. Therefore, the original Spack tree must be kept in place for a filesystem view to work, even if the view is built with hardlinks.

spack view

A filesystem view is created, and packages are linked in, by the spack view command’s symlink and hardlink sub-commands. The spack view remove command can be used to unlink some or all of the filesystem view.

The following example creates a filesystem view based on an installed cmake package and then removes from the view the files in the cmake package while retaining its dependencies.

$ spack view --verbose symlink myview cmake@3.5.2
==> Linking package: "ncurses"
==> Linking package: "zlib"
==> Linking package: "openssl"
==> Linking package: "cmake"

$ ls myview/
bin  doc  etc  include  lib  share

$ ls myview/bin/
captoinfo  clear  cpack     ctest    infotocap        openssl  tabs  toe   tset
ccmake     cmake  c_rehash  infocmp  ncurses6-config  reset    tic   tput

$ spack view --verbose --dependencies false rm myview cmake@3.5.2
==> Removing package: "cmake"

$ ls myview/bin/
captoinfo  c_rehash  infotocap        openssl  tabs  toe   tset
clear      infocmp   ncurses6-config  reset    tic   tput


If the set of packages being included in a view is inconsistent, then it is possible that two packages will provide the same file. Any conflicts of this type are handled on a first-come-first-served basis, and a warning is printed.


When packages are removed from a view, empty directories are purged.

Controlling View Projections

The default projection into a view is to link every package into the root of the view. This can be changed by adding a projections.yaml configuration file to the view. The projection configuration file for a view located at /my/view is stored in /my/view/.spack/projections.yaml.

When creating a view, the projection configuration file can also be specified from the command line using the --projection-file option to the spack view command.

The projections configuration file is a mapping of partial specs to spec format strings, defined by the format() function, as shown in the example below.

  zlib: {name}-{version}
  ^mpi: {name}-{version}/{^}-{^mpi.version}-{}-{compiler.version}
  all: {name}-{version}/{}-{compiler.version}

The entries in the projections configuration file must all be either specs or the keyword all. For each spec, the projection used will be the first non-all entry that the spec satisfies, or all if there is an entry for all and no other entry is satisfied by the spec. Where the keyword all appears in the file does not matter. Given the example above, any spec satisfying zlib@1.2.8 will be linked into /my/view/zlib-1.2.8/, any spec satisfying hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2 will be linked into /my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3, and any spec satisfying hdf5@1.8.10~mpi %gcc@4.9.3 will be linked into /my/view/hdf5-1.8.10/gcc-4.9.3.

If the keyword all does not appear in the projections configuration file, any spec that does not satisfy any entry in the file will be linked into the root of the view as in a single-prefix view. Any entries that appear below the keyword all in the projections configuration file will not be used, as all specs will use the projection under all before reaching those entries.

Fine-Grain Control

The --exclude and --dependencies option flags allow for fine-grained control over which packages and dependencies do or not get included in a view. For example, suppose you are developing the appsy package. You wish to build against a view of all appsy dependencies, but not appsy itself:

$ spack view --dependencies yes --exclude appsy symlink /path/to/MYVIEW/ appsy

Alternately, you wish to create a view whose purpose is to provide binary executables to end users. You only need to include applications they might want, and not those applications’ dependencies. In this case, you might use:

$ spack view --dependencies no symlink /path/to/MYVIEW/ cmake

Hybrid Filesystem Views

Although filesystem views are usually created by Spack, users are free to add to them by other means. For example, imagine a filesystem view, created by Spack, that looks something like:

/path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA
/path/to/MYVIEW/lib/ -> /path/to/spack/.../lib/

Now, the user may add to this view by non-Spack means; for example, by running a classic install script. For example:

$ tar -xf B.tar.gz
$ cd B/
$ ./configure --prefix=/path/to/MYVIEW \
$ make && make install

The result is a hybrid view:

/path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA
/path/to/MYVIEW/lib/ -> /path/to/spack/.../lib/

In this case, real files coexist, interleaved with the “view” symlinks. At any time one can delete /path/to/MYVIEW or use spack view to manage it surgically. None of this will affect the real Spack install area.

Global Activations

spack activate may be used as an alternative to loading Python (and similar systems) packages directly or creating a view. If extensions are globally activated, then spack load python will also load all the extensions activated for the given python. This reduces the need for users to load a large number of packages.

However, Spack global activations have two potential drawbacks:

  1. Activated packages that involve compiled C extensions may still need their dependencies to be loaded manually. For example, spack load openblas might be required to make py-numpy work.

  2. Global activations “break” a core feature of Spack, which is that multiple versions of a package can co-exist side-by-side. For example, suppose you wish to run a Python package in two different environments but the same basic Python — one with py-numpy@1.7 and one with py-numpy@1.8. Spack extensions will not support this potential debugging use case.

Discussion: Running Binaries

Modules, extension packages and filesystem views are all ways to assemble sets of Spack packages into a useful environment. They are all semantically similar, in that conflicting installed packages cannot simultaneously be loaded, activated or included in a view.

With all of these approaches, there is no guarantee that the environment created will be consistent. It is possible, for example, to simultaneously load application A that uses OpenMPI and application B that uses MPICH. Both applications will run just fine in this inconsistent environment because they rely on RPATHs, not the environment, to find their dependencies.

In general, environments set up using modules vs. views will work similarly. Both can be used to set up ephemeral or long-lived testing/development environments. Operational differences between the two approaches can make one or the other preferable in certain environments:

  • Filesystem views do not require environment module infrastructure. Although Spack can install environment-modules, users might be hostile to its use. Filesystem views offer a good solution for sysadmins serving users who just “want all the stuff I need in one place” and don’t want to hear about Spack.

  • Although modern build systems will find dependencies wherever they might be, some applications with hand-built make files expect their dependencies to be in one place. One common problem is makefiles that assume that netcdf and netcdf-fortran are installed in the same tree. Or, one might use an IDE that requires tedious configuration of dependency paths; and it’s easier to automate that administration in a view-building script than in the IDE itself. For all these cases, a view will be preferable to other ways to assemble an environment.

  • On systems with I-node quotas, modules might be preferable to views and extension packages.

  • Views and activated extensions maintain state that is semantically equivalent to the information in a spack module tcl loads script. Administrators might find things easier to maintain without the added “heavyweight” state of a view.

Using Spack to Replace Homebrew/Conda

Spack is an incredibly powerful package manager, designed for supercomputers where users have diverse installation needs. But Spack can also be used to handle simple single-user installations on your laptop. Most macOS users are already familiar with package managers like Homebrew and Conda, where all installed packages are symlinked to a single central location like /usr/local. In this section, we will show you how to emulate the behavior of Homebrew/Conda using Environments!


First, let’s create a new environment. We’ll assume that Spack is already set up correctly, and that you’ve already sourced the setup script for your shell. To create a new environment, simply run:

$ spack env create myenv
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
==> Created environment 'myenv' in /Users/me/spack/var/spack/environments/myenv
$ spack env activate myenv

Here, myenv can be anything you want to name your environment. Next, we can add a list of packages we would like to install into our environment. Let’s say we want a newer version of Bash than the one that comes with macOS, and we want a few Python libraries. We can run:

$ spack add bash
==> Adding bash to environment myenv
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
$ spack add python@3:
==> Adding python@3: to environment myenv
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
$ spack add py-numpy py-scipy py-matplotlib
==> Adding py-numpy to environment myenv
==> Adding py-scipy to environment myenv
==> Adding py-matplotlib to environment myenv
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view

Each package can be listed on a separate line, or combined into a single line. Notice that we’re explicitly asking for Python 3 here. You can use any spec you would normally use on the command line with other Spack commands.

Next, we want to manually configure a couple of things. In the myenv directory, we can find the spack.yaml that actually defines our environment.

$ vim ~/spack/var/spack/environments/myenv/spack.yaml
# This is a Spack Environment file.
# It describes a set of packages to be installed, along with
# configuration settings.
  # add package specs to the `specs` list
  specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
      root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
      projections: {}
  config: {}
  mirrors: {}
    enable: []
  packages: {}
  repos: []
  upstreams: {}
  definitions: []
  concretization: separately

You can see the packages we added earlier in the specs: section. If you ever want to add more packages, you can either use spack add or manually edit this file.

We also need to change the concretization: option. By default, Spack concretizes each spec separately, allowing multiple versions of the same package to coexist. Since we want a single consistent environment, we want to concretize all of the specs together.

Here is what your spack.yaml looks like with these new settings, and with some of the sections we don’t plan on using removed:

-  specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
+  specs:
+  - bash
+  - 'python@3:'
+  - py-numpy
+  - py-scipy
+  - py-matplotlib
-  view:
-    default:
-      root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
-      projections: {}
+  view: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
-  config: {}
-  mirrors: {}
-  modules:
-    enable: []
-  packages: {}
-  repos: []
-  upstreams: {}
-  definitions: []
+  concretization: together
-  concretization: separately


To actually concretize the environment, run:

$ spack concretize

This will tell you which if any packages are already installed, and alert you to any conflicting specs.

To actually install these packages and symlink them to your view: directory, simply run:

$ spack install

Now, when you type which python3, it should find the one you just installed.

In order to change the default shell to our newer Bash installation, we first need to add it to this list of acceptable shells. Run:

$ sudo vim /etc/shells

and add the absolute path to your bash executable. Then run:

$ chsh -s /path/to/bash

Now, when you log out and log back in, echo $SHELL should point to the newer version of Bash.

Updating Installed Packages

Let’s say you upgraded to a new version of macOS, or a new version of Python was released, and you want to rebuild your entire software stack. To do this, simply run the following commands:

$ spack env activate myenv
$ spack concretize --force
$ spack install

The --force flag tells Spack to overwrite its previous concretization decisions, allowing you to choose a new version of Python. If any of the new packages like Bash are already installed, spack install won’t re-install them, it will keep the symlinks in place.


If you decide that Spack isn’t right for you, uninstallation is simple. Just run:

$ spack env activate myenv
$ spack uninstall --all

This will uninstall all packages in your environment and remove the symlinks.

Using Spack on Travis-CI

Spack can be deployed as a provider for userland software in Travis-CI.

A starting-point for a .travis.yml file can look as follows. It uses caching for already built environments, so make sure to clean the Travis cache if you run into problems.

The main points that are implemented below:

  1. Travis is detected as having up to 34 cores available, but only 2 are actually allocated for the user. We limit the parallelism of the spack builds in the config. (The Travis yaml parser is a bit buggy on the echo command.)

  2. Without control for the user, Travis jobs will run on various x86_64 microarchitectures. If you plan to cache build results, e.g. to accelerate dependency builds, consider building for the generic x86_64 target only. Limiting the microarchitecture will also find more packages when working with the E4S Spack build cache.

  3. Builds over 10 minutes need to be prefixed with travis_wait. Alternatively, generate output once with spack install -v.

  4. Travis builds are non-interactive. This prevents using bash aliases and functions for modules. We fix that by sourcing /etc/profile first (or running everything in a subshell with bash -l -c '...').

language: cpp
sudo: false
dist: trusty

  apt: true
    - $HOME/.cache

      - ubuntu-toolchain-r-test
      - g++-4.9
      - environment-modules

    - SPACK_ROOT: $HOME/.cache/spack
    - PATH: $PATH:$HOME/.cache/spack/bin

  - export CXX=g++-4.9
  - export CC=gcc-4.9
  - export FC=gfortran-4.9
  - export CXXFLAGS="-std=c++11"

  - |
    if ! which spack >/dev/null; then
      mkdir -p $SPACK_ROOT &&
      git clone --depth 50 $SPACK_ROOT &&
      printf "config:\n  build_jobs: 2\n" > $SPACK_ROOT/etc/spack/config.yaml &&
      printf "packages:\n  all:\n    target: ['x86_64']\n" \
              > $SPACK_ROOT/etc/spack/packages.yaml;
  - travis_wait spack install cmake@3.7.2~openssl~ncurses
  - travis_wait spack install boost@1.62.0~graph~iostream~locale~log~wave
  - spack clean -a
  - source /etc/profile &&
    source $SPACK_ROOT/share/spack/
  - spack load cmake
  - spack load boost

  - mkdir -p $HOME/build
  - cd $HOME/build
  - make -j 2
  - make test

Upstream Bug Fixes

It is not uncommon to discover a bug in an upstream project while trying to build with Spack. Typically, the bug is in a package that serves a dependency to something else. This section describes procedure to work around and ultimately resolve these bugs, while not delaying the Spack user’s main goal.

Buggy New Version

Sometimes, the old version of a package works fine, but a new version is buggy. For example, it was once found that Adios did not build with hdf5@1.10. If the old version of hdf5 will work with adios, the suggested procedure is:

  1. Revert adios to the old version of hdf5. Put in its adios/

    # Adios does not build with HDF5 1.10
    # See:
  2. Determine whether the problem is with hdf5 or adios, and report the problem to the appropriate upstream project. In this case, the problem was with adios.

  3. Once a new version of adios comes out with the bugfix, modify adios/ to reflect it:

    # Adios up to v1.10.0 does not build with HDF5 1.10
    # See:
    depends_on('hdf5@:1.9', when='@:1.10.0')
    depends_on('hdf5', when='@1.10.1:')

No Version Works

Sometimes, no existing versions of a dependency work for a build. This typically happens when developing a new project: only then does the developer notice that existing versions of a dependency are all buggy, or the non-buggy versions are all missing a critical feature.

In the long run, the upstream project will hopefully fix the bug and release a new version. But that could take a while, even if a bugfix has already been pushed to the project’s repository. In the meantime, the Spack user needs things to work.

The solution is to create an unofficial Spack release of the project, as soon as the bug is fixed in some repository. A study of the Git history of py-proj/ is instructive here:

  1. On April 1, an initial bugfix was identified for the PyProj project and a pull request submitted to PyProj. Because the upstream authors had not yet fixed the bug, the py-proj Spack package downloads from a forked repository, set up by the package’s author. A non-numeric version number is used to make it easy to upgrade the package without recomputing checksums; however, this is an untrusted download method and should not be distributed. The package author has now become, temporarily, a maintainer of the upstream project:

    # We need the benefits of this PR
  2. By May 14, the upstream project had accepted a pull request with the required bugfix. At this point, the forked repository was deleted. However, the upstream project still had not released a new version with a bugfix. Therefore, a Spack-only release was created by specifying the desired hash in the main project repository. The version number @ was chosen for this “release” because it’s a descendent of the officially released version @ This is a trusted download method, and can be released to the Spack community:

    # This is not a tagged release of pyproj.
    # The changes in this "version" fix some bugs, especially with Python3 use.
    version('', 'd035e4bc704d136db79b43ab371b27d2',


    It would have been simpler to use Spack’s Git download method, which is also a trusted download in this case:

    # This is not a tagged release of pyproj.
    # The changes in this "version" fix some bugs, especially with Python3 use.


    In this case, the upstream project fixed the bug in its repository in a relatively timely manner. If that had not been the case, the numbered version in this step could have been released from the forked repository.

  3. The author of the Spack package has now become an unofficial release engineer for the upstream project. Depending on the situation, it may be advisable to put preferred=True on the latest officially released version.

  4. As of August 31, the upstream project still had not made a new release with the bugfix. In the meantime, Spack-built py-proj provides the bugfix needed by packages depending on it. As long as this works, there is no particular need for the upstream project to make a new official release.

  5. If the upstream project releases a new official version with the bugfix, then the unofficial version() line should be removed from the Spack package.


Spack’s source patching mechanism provides another way to fix bugs in upstream projects. This has advantages and disadvantages compared to the procedures above.


  1. It can fix bugs in existing released versions, and (probably) future releases as well.

  2. It is lightweight, does not require a new fork to be set up.


  1. It is harder to develop and debug a patch, vs. a branch in a repository. The user loses the automation provided by version control systems.

  2. Although patches of a few lines work OK, large patch files can be hard to create and maintain.