Spack for Homebrew/Conda Users

Spack is an incredibly powerful package manager, designed for supercomputers where users have diverse installation needs. But Spack can also be used to handle simple single-user installations on your laptop. Most macOS users are already familiar with package managers like Homebrew and Conda, where all installed packages are symlinked to a single central location like /usr/local. In this section, we will show you how to emulate the behavior of Homebrew/Conda using Environments (spack.yaml, spack.lock)!

Setup

First, let’s create a new environment. We’ll assume that Spack is already set up correctly, and that you’ve already sourced the setup script for your shell. To create a new environment, simply run:

$ spack env create myenv

Here, myenv can be anything you want to name your environment. Next, we can add a list of packages we would like to install into our environment. Let’s say we want a newer version of Bash than the one that comes with macOS, and we want a few Python libraries. We can run:

$ spack -e myenv add bash@5 python py-numpy py-scipy py-matplotlib

Each package can be listed on a separate line, or combined into a single line like we did above. Notice that we’re explicitly asking for Bash 5 here. You can use any spec you would normally use on the command line with other Spack commands.

Next, we want to manually configure a couple of things:

$ spack -e myenv config edit

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
  view: true

You can see the packages we added earlier in the specs: section. If you ever want to add more packages, you can either use spack add or manually edit this file.

We also need to change the concretizer:unify option. By default, Spack concretizes each spec separately, allowing multiple versions of the same package to coexist. Since we want a single consistent environment, we want to concretize all of the specs together.

Here is what your spack.yaml looks like with this new setting:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
  view: true
  concretizer:
    unify: true

Symlink location

Spack symlinks all installations to /Users/me/spack/var/spack/environments/myenv/.spack-env/view, which is the default when view: true. You can actually change this to any directory you want. For example, Homebrew uses /usr/local, while Conda uses /Users/me/anaconda. In order to access files in these locations, you need to update PATH and other environment variables to point to them. Activating the Spack environment does this automatically, but you can also manually set them in your .bashrc.

Warning

There are several reasons why you shouldn’t use /usr/local:

If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard for you. You may notice permissions issues on /usr/local due to their System Integrity Protection. By default, users don’t have permissions to install anything in /usr/local, and you can’t even change this using sudo chown or sudo chmod.
Other package managers like Homebrew will try to install things to the same directory. If you plan on using Homebrew in conjunction with Spack, don’t symlink things to /usr/local.
If you are on a shared workstation, or don’t have sudo privileges, you can’t do this.

If you still want to do this anyway, there are several ways around SIP. You could disable SIP by booting into recovery mode and running csrutil disable, but this is not recommended, as it can open up your OS to security vulnerabilities. Another technique is to run spack concretize and spack install using sudo. This is also not recommended.

The safest way I’ve found is to create your installation directories using sudo, then change ownership back to the user like so:

for directory in .spack bin contrib include lib man share
do
    sudo mkdir -p /usr/local/$directory
    sudo chown $(id -un):$(id -gn) /usr/local/$directory
done

Depending on the packages you install in your environment, the exact list of directories you need to create may vary. You may also find some packages like Java libraries that install a single file to the installation prefix instead of in a subdirectory. In this case, the action is the same, just replace mkdir -p with touch in the for-loop above.

But again, it’s safer just to use the default symlink location.

Installation

To actually concretize the environment, run:

$ spack -e myenv concretize

This will tell you which if any packages are already installed, and alert you to any conflicting specs.

To actually install these packages and symlink them to your view: directory, simply run:

$ spack -e myenv install
$ spack env activate myenv

Now, when you type which python3, it should find the one you just installed.

In order to change the default shell to our newer Bash installation, we first need to add it to this list of acceptable shells. Run:

$ sudo vim /etc/shells

and add the absolute path to your bash executable. Then run:

$ chsh -s /path/to/bash

Now, when you log out and log back in, echo $SHELL should point to the newer version of Bash.

Updating Installed Packages

Let’s say you upgraded to a new version of macOS, or a new version of Python was released, and you want to rebuild your entire software stack. To do this, simply run the following commands:

$ spack env activate myenv
$ spack concretize --fresh --force
$ spack install

The --fresh flag tells Spack to use the latest version of every package where possible instead of trying to optimize for reuse of existing installed packages.

The --force flag in addition tells Spack to overwrite its previous concretization decisions, allowing you to choose a new version of Python. If any of the new packages like Bash are already installed, spack install won’t re-install them, it will keep the symlinks in place.

Updating & Cleaning Up Old Packages

If you’re looking to mimic the behavior of Homebrew, you may also want to clean up out-of-date packages from your environment after an upgrade. To upgrade your entire software stack within an environment and clean up old package versions, simply run the following commands:

$ spack env activate myenv
$ spack mark -i --all
$ spack concretize --fresh --force
$ spack install
$ spack gc

Running spack mark -i --all tells Spack to mark all of the existing packages within an environment as “implicitly” installed. This tells spack’s garbage collection system that these packages should be cleaned up.

Don’t worry however, this will not remove your entire environment. Running spack install will reexamine your spack environment after a fresh concretization and will re-mark any packages that should remain installed as “explicitly” installed.

Note: if you use multiple spack environments you should re-run spack install in each of your environments prior to running spack gc to prevent spack from uninstalling any shared packages that are no longer required by the environment you just upgraded.

Uninstallation

If you decide that Spack isn’t right for you, uninstallation is simple. Just run:

$ spack env activate myenv
$ spack uninstall --all

This will uninstall all packages in your environment and remove the symlinks.