Container Images¶
Spack can be an ideal tool to setup images for containers since all the features discussed in Environments can greatly help to manage the installation of software during the image build process. Nonetheless, building a production image from scratch still requires a lot of boilerplate to:
- Get Spack working within the image, possibly running as root
- Minimize the physical size of the software installed
- Properly update the system software in the base image
To facilitate users with these tedious tasks, Spack provides a command to automatically generate recipes for container images based on Environments:
$ ls
spack.yaml
$ spack containerize
# Build stage with Spack pre-installed and ready to be used
FROM spack/centos7:latest as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs+mpi" \
&& echo " - mpich" \
&& echo " concretization: together" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack install && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM centos:7
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
RUN yum update -y && yum install -y epel-release && yum update -y \
&& yum install -y libgomp \
&& rm -rf /var/cache/yum && yum clean all
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
LABEL "app"="gromacs"
LABEL "mpi"="mpich"
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
The bits that make this automation possible are discussed in details below. All the images generated in this way will be based on multi-stage builds with:
- A fat
build
stage containing common build tools and Spack itself - A minimal
final
stage containing only the software requested by the user
Spack Base Images¶
Docker images with Spack preinstalled and ready to be used are
built on Docker Hub
at every push to develop
or to a release branch. The OS that
are currently supported are summarized in the table below:
Operating System | Base Image | Spack Image |
---|---|---|
Ubuntu 16.04 | ubuntu:16.04 |
spack/ubuntu-xenial |
Ubuntu 18.04 | ubuntu:18.04 |
spack/ubuntu-bionic |
CentOS 6 | centos:6 |
spack/centos6 |
CentOS 7 | centos:7 |
spack/centos7 |
All the images are tagged with the corresponding release of Spack:
with the exception of the latest
tag that points to the HEAD
of the develop
branch. These images are available for anyone
to use and take care of all the repetitive tasks that are necessary
to setup Spack within a container. All the container recipes generated
automatically by Spack use them as base images for their build
stage.
Environment Configuration¶
Any Spack Environment can be used for the automatic generation of container
recipes. Sensible defaults are provided for things like the base image or the
version of Spack used in the image. If a finer tuning is needed it can be
obtained by adding the relevant metadata under the container
attribute
of environments:
spack:
specs:
- gromacs+mpi
- mpich
container:
# Select the format of the recipe e.g. docker,
# singularity or anything else that is currently supported
format: docker
# Select from a valid list of images
base:
image: "centos:7"
spack: develop
# Whether or not to strip binaries
strip: true
# Additional system packages that are needed at runtime
os_packages:
- libgomp
# Extra instructions
extra_instructions:
final: |
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
# Labels for the image
labels:
app: "gromacs"
mpi: "mpich"
The tables below describe the configuration options that are currently supported:
Option Name | Description | Allowed Values | Required |
---|---|---|---|
format |
The format of the recipe | docker or singularity |
Yes |
base:image |
Base image for final stage |
See Supported operating systems | Yes |
base:spack |
Version of Spack | Valid tags for base:image |
Yes |
strip |
Whether to strip binaries | true (default) or false |
No |
os_packages |
System packages to be installed | Valid packages for the final OS |
No |
extra_instructions:build |
Extra instructions (e.g. RUN, COPY, etc.) at the end of the build stage |
Anything understood by the current format |
No |
extra_instructions:final |
Extra instructions (e.g. RUN, COPY, etc.) at the end of the final stage |
Anything understood by the current format |
No |
labels |
Labels to tag the image | Pairs of key-value strings | No |
Option Name | Description | Allowed Values | Required |
---|---|---|---|
singularity:runscript |
Content of %runscript |
Any valid script | No |
singularity:startscript |
Content of %startscript |
Any valid script | No |
singularity:test |
Content of %test |
Any valid script | No |
singularity:help |
Description of the image | Description string | No |
Once the Environment is properly configured a recipe for a container
image can be printed to standard output by issuing the following
command from the directory where the spack.yaml
resides:
$ spack containerize
The example spack.yaml
above would produce for instance the
following Dockerfile
:
# Build stage with Spack pre-installed and ready to be used
FROM spack/centos7:latest as builder
# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&& (echo "spack:" \
&& echo " specs:" \
&& echo " - gromacs+mpi" \
&& echo " - mpich" \
&& echo " concretization: together" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack install && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM centos:7
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
RUN yum update -y && yum install -y epel-release && yum update -y \
&& yum install -y libgomp \
&& rm -rf /var/cache/yum && yum clean all
RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
LABEL "app"="gromacs"
LABEL "mpi"="mpich"
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
Note
Spack can also produce Singularity definition files to build the image. The
minimum version of Singularity required to build a SIF (Singularity Image Format)
from them is 3.5.3
.