Multiple Build Systems¶
Quite frequently, a package will change build systems from one version to the next. For example, a small project that once used a single Makefile to build may now require Autotools to handle the increased number of files that need to be compiled. Or, a package that once used Autotools may switch to CMake for Windows support. In this case, it becomes a bit more challenging to write a single build recipe for this package in Spack.
There are several ways that this can be handled in Spack:
Subclass the new build system, and override phases as needed (preferred)
Subclass
Package
and implementinstall
as neededCreate separate
*-cmake
,*-autotools
, etc. packages for each build systemRename the old package to
*-legacy
and create a new packageMove the old package to a
legacy
repository and create a new packageDrop older versions that only support the older build system
Of these options, 1 is preferred, and will be demonstrated in this documentation. Options 3-5 have issues with concretization, so shouldn’t be used. Options 4-5 also don’t support more than two build systems. Option 6 only works if the old versions are no longer needed. Option 1 is preferred over 2 because it makes it easier to drop the old build system entirely.
The exact syntax of the package depends on which build systems you need to support. Below are a couple of common examples.
Makefile -> Autotools¶
Let’s say we have the following package:
class Foo(MakefilePackage):
version("1.2.0", sha256="...")
def edit(self, spec, prefix):
filter_file("CC=", "CC=" + spack_cc, "Makefile")
def install(self, spec, prefix):
install_tree(".", prefix)
The package subclasses from MakefilePackage, which has three phases:
edit
(does nothing by default)build
(runsmake
by default)install
(runsmake install
by default)
In this case, the install
phase needed to be overridden because the
Makefile did not have an install target. We also modify the Makefile to use
Spack’s compiler wrappers. The default build
phase is not changed.
Starting with version 1.3.0, we want to use Autotools to build instead. AutotoolsPackage has four phases:
autoreconf
(does not if a configure script already exists)configure
(runs./configure --prefix=...
by default)build
(runsmake
by default)install
(runsmake install
by default)
If the only version we need to support is 1.3.0, the package would look as simple as:
class Foo(AutotoolsPackage):
version("1.3.0", sha256="...")
def configure_args(self):
return ["--enable-shared"]
In this case, we use the default methods for each phase and only override
configure_args
to specify additional flags to pass to ./configure
.
If we wanted to write a single package that supports both versions 1.2.0 and 1.3.0, it would look something like:
class Foo(AutotoolsPackage):
version("1.3.0", sha256="...")
version("1.2.0", sha256="...", deprecated=True)
def configure_args(self):
return ["--enable-shared"]
# Remove the following once version 1.2.0 is dropped
@when("@:1.2")
def patch(self):
filter_file("CC=", "CC=" + spack_cc, "Makefile")
@when("@:1.2")
def autoreconf(self, spec, prefix):
pass
@when("@:1.2")
def configure(self, spec, prefix):
pass
@when("@:1.2")
def install(self, spec, prefix):
install_tree(".", prefix)
There are a few interesting things to note here:
We added
deprecated=True
to version 1.2.0. This signifies that version 1.2.0 is deprecated and shouldn’t be used. However, if a user still relies on version 1.2.0, it’s still there and builds just fine.We moved the contents of the
edit
phase to thepatch
function. SinceAutotoolsPackage
doesn’t have anedit
phase, the only way for this step to be executed is to move it to thepatch
function, which always gets run.The
autoreconf
andconfigure
phases become no-ops. Since the old Makefile-based build system doesn’t use these, we ignore these phases when buildingfoo@1.2.0
.The
@when
decorator is used to override these phases only for older versions. The default methods are used forfoo@1.3:
.
Once a new Spack release comes out, version 1.2.0 and everything below the comment can be safely deleted. The result is the same as if we had written a package for version 1.3.0 from scratch.
Autotools -> CMake¶
Let’s say we have the following package:
class Bar(AutotoolsPackage):
version("1.2.0", sha256="...")
def configure_args(self):
return ["--enable-shared"]
The package subclasses from AutotoolsPackage, which has four phases:
autoreconf
(does not if a configure script already exists)configure
(runs./configure --prefix=...
by default)build
(runsmake
by default)install
(runsmake install
by default)
In this case, we use the default methods for each phase and only override
configure_args
to specify additional flags to pass to ./configure
.
Starting with version 1.3.0, we want to use CMake to build instead. CMakePackage has three phases:
cmake
(runscmake ...
by default)build
(runsmake
by default)install
(runsmake install
by default)
If the only version we need to support is 1.3.0, the package would look as simple as:
class Bar(CMakePackage):
version("1.3.0", sha256="...")
def cmake_args(self):
return [self.define("BUILD_SHARED_LIBS", True)]
In this case, we use the default methods for each phase and only override
cmake_args
to specify additional flags to pass to cmake
.
If we wanted to write a single package that supports both versions 1.2.0 and 1.3.0, it would look something like:
class Bar(CMakePackage):
version("1.3.0", sha256="...")
version("1.2.0", sha256="...", deprecated=True)
def cmake_args(self):
return [self.define("BUILD_SHARED_LIBS", True)]
# Remove the following once version 1.2.0 is dropped
def configure_args(self):
return ["--enable-shared"]
@when("@:1.2")
def cmake(self, spec, prefix):
configure("--prefix=" + prefix, *self.configure_args())
There are a few interesting things to note here:
We added
deprecated=True
to version 1.2.0. This signifies that version 1.2.0 is deprecated and shouldn’t be used. However, if a user still relies on version 1.2.0, it’s still there and builds just fine.Since CMake and Autotools are so similar, we only need to override the
cmake
phase, we can use the defaultbuild
andinstall
phases.We override
cmake
to run./configure
for older versions.configure_args
remains the same.The
@when
decorator is used to override these phases only for older versions. The default methods are used forbar@1.3:
.
Once a new Spack release comes out, version 1.2.0 and everything below the comment can be safely deleted. The result is the same as if we had written a package for version 1.3.0 from scratch.
Multiple build systems for the same version¶
During the transition from one build system to another, developers often support multiple build systems at the same time. Spack can only use a single build system for a single version. To decide which build system to use for a particular version, take the following things into account:
If the developers explicitly state that one build system is preferred over another, use that one.
If one build system is considered “experimental” while another is considered “stable”, use the stable build system.
Otherwise, use the newer build system.
The developer preference for which build system to use can change over time as a newer build system becomes stable/recommended.
Dropping support for old build systems¶
When older versions of a package don’t support a newer build system, it can be tempting to simply delete them from a package. This significantly reduces package complexity and makes the build recipe much easier to maintain. However, other packages or Spack users may rely on these older versions. The recommended approach is to first support both build systems (as demonstrated above), deprecate versions that rely on the old build system, and remove those versions and any phases that needed to be overridden in the next Spack release.
Three or more build systems¶
In rare cases, a package may change build systems multiple times. For example, a package may start with Makefiles, then switch to Autotools, then switch to CMake. The same logic used above can be extended to any number of build systems. For example:
class Baz(CMakePackage):
version("1.4.0", sha256="...") # CMake
version("1.3.0", sha256="...") # Autotools
version("1.2.0", sha256="...") # Makefile
def cmake_args(self):
return [self.define("BUILD_SHARED_LIBS", True)]
# Remove the following once version 1.3.0 is dropped
def configure_args(self):
return ["--enable-shared"]
@when("@1.3")
def cmake(self, spec, prefix):
configure("--prefix=" + prefix, *self.configure_args())
# Remove the following once version 1.2.0 is dropped
@when("@:1.2")
def patch(self):
filter_file("CC=", "CC=" + spack_cc, "Makefile")
@when("@:1.2")
def cmake(self, spec, prefix):
pass
@when("@:1.2")
def install(self, spec, prefix):
install_tree(".", prefix)
Additional examples¶
When writing new packages, it often helps to see examples of existing packages. Here is an incomplete list of existing Spack packages that have changed build systems before:
Package |
Previous Build System |
New Build System |
---|---|---|
amber |
custom |
CMake |
arpack-ng |
Autotools |
CMake |
atk |
Autotools |
Meson |
blast |
None |
Autotools |
dyninst |
Autotools |
CMake |
evtgen |
Autotools |
CMake |
fish |
Autotools |
CMake |
gdk-pixbuf |
Autotools |
Meson |
glib |
Autotools |
Meson |
glog |
Autotools |
CMake |
gmt |
Autotools |
CMake |
gtkplus |
Autotools |
Meson |
hpl |
Makefile |
Autotools |
interproscan |
Perl |
Maven |
jasper |
Autotools |
CMake |
kahip |
SCons |
CMake |
kokkos |
Makefile |
CMake |
kokkos-kernels |
Makefile |
CMake |
leveldb |
Makefile |
CMake |
libdrm |
Autotools |
Meson |
libjpeg-turbo |
Autotools |
CMake |
mesa |
Autotools |
Meson |
metis |
None |
CMake |
mpifileutils |
Autotools |
CMake |
muparser |
Autotools |
CMake |
mxnet |
Makefile |
CMake |
nest |
Autotools |
CMake |
neuron |
Autotools |
CMake |
nsimd |
CMake |
nsconfig |
opennurbs |
Makefile |
CMake |
optional-lite |
None |
CMake |
plasma |
Makefile |
CMake |
preseq |
Makefile |
Autotools |
protobuf |
Autotools |
CMake |
py-pygobject |
Autotools |
Python |
singularity |
Autotools |
Makefile |
span-lite |
None |
CMake |
ssht |
Makefile |
CMake |
string-view-lite |
None |
CMake |
superlu |
Makefile |
CMake |
superlu-dist |
Makefile |
CMake |
uncrustify |
Autotools |
CMake |
Packages that support multiple build systems can be a bit confusing to write. Don’t hesitate to open an issue or draft pull request and ask for advice from other Spack developers!