spack.environment package

This package implements Spack environments.

spack.lock format

Spack environments have existed since Spack v0.12.0, and there have been 4 different spack.lock formats since then. The formats are documented here.

The high-level format of a Spack lockfile hasn’t changed much between versions, but the contents have. Lockfiles are JSON-formatted and their top-level sections are:

  1. _meta (object): this contains details about the file format, including:
    • file-type: always "spack-lockfile"

    • lockfile-version: an integer representing the lockfile format version

    • specfile-version: an integer representing the spec format version (since v0.17)

  2. spack (object): optional, this identifies information about Spack

    used to concretize the environment: * type: required, identifies form Spack version took (e.g., git, release) * commit: the commit if the version is from git * version: the Spack version

  3. roots (list): an ordered list of records representing the roots of the Spack

    environment. Each has two fields: * hash: a Spack spec hash uniquely identifying the concrete root spec * spec: a string representation of the abstract spec that was concretized

  4. concrete_specs: a dictionary containing the specs in the environment.

  5. include_concrete (dictionary): an optional dictionary that includes the roots and concrete specs from the included environments, keyed by the path to that environment

Compatibility

New versions of Spack can (so far) read all old lockfile formats – they are backward-compatible. Old versions cannot read new lockfile formats, and you’ll need to upgrade Spack to use them.

Lockfile version compatibility across Spack versions

Spack version

v1

v2

v3

v4

v5

v0.12:0.14

v0.15:0.16

v0.17

v0.18:

v0.22:

Version 1

When lockfiles were first created, there was only one hash in Spack: the DAG hash. This DAG hash (we’ll call it the old DAG hash) did not include build dependencies – it only included transitive link and run dependencies.

The spec format at this time was keyed by name. Each spec started with a key for its name, whose value was a dictionary of other spec attributes. The lockfile put these name-keyed specs into dictionaries keyed by their DAG hash, and the spec records did not actually have a “hash” field in the lockfile – you have to associate the hash from the key with the spec record after the fact.

Dependencies in original lockfiles were keyed by "hash", i.e. the old DAG hash.

{
    "_meta": {
        "file-type": "spack-lockfile",
        "lockfile-version": 1
    },
    "roots": [
        {
            "hash": "<old_dag_hash 1>",
            "spec": "<abstract spec 1>"
        },
        {
            "hash": "<old_dag_hash 2>",
            "spec": "<abstract spec 2>"
        }
    ],
    "concrete_specs": {
        "<old_dag_hash 1>": {
            "... <spec dict attributes> ...": { },
            "dependencies": {
                "depname_1": {
                    "hash": "<old_dag_hash for depname_1>",
                    "type": ["build", "link"]
                },
                "depname_2": {
                    "hash": "<old_dag_hash for depname_3>",
                    "type": ["build", "link"]
                }
            },
            "hash": "<old_dag_hash 1>"
        },
        "<old_dag_hash 2>": {
            "... <spec dict attributes> ...": { },
            "dependencies": {
                "depname_3": {
                    "hash": "<old_dag_hash for depname_3>",
                    "type": ["build", "link"]
                },
                "depname_4": {
                    "hash": "<old_dag_hash for depname_4>",
                    "type": ["build", "link"]
                },
            },
            "hash": "<old_dag_hash 2>"
        },
    }
}

Version 2

Version 2 changes one thing: specs in the lockfile are now keyed by build_hash instead of the old dag_hash. Specs have a hash attribute with their real DAG hash, so you can’t go by the dictionary key anymore to identify a spec – you have to read it in and look at "hash". Dependencies are still keyed by old DAG hash.

Even though we key lockfiles by build_hash, specs in Spack were still deployed with the old, coarser DAG hash. This means that in v2 and v3 lockfiles (which are keyed by build hash), there may be multiple versions of the same spec with different build dependencies, which means they will have different build hashes but the same DAG hash. Spack would only have been able to actually install one of these.

{
    "_meta": {
        "file-type": "spack-lockfile",
        "lockfile-version": 2
    },
    "roots": [
        {
            "hash": "<build_hash 1>",
            "spec": "<abstract spec 1>"
        },
        {
            "hash": "<build_hash 2>",
            "spec": "<abstract spec 2>"
        }
    ],
    "concrete_specs": {
        "<build_hash 1>": {
            "... <spec dict attributes> ...": { },
            "dependencies": {
                "depname_1": {
                    "hash": "<old_dag_hash for depname_1>",
                    "type": ["build", "link"]
                },
                "depname_2": {
                    "hash": "<old_dag_hash for depname_3>",
                    "type": ["build", "link"]
                }
            },
            "hash": "<old_dag_hash 1>",
        },
        "<build_hash 2>": {
            "... <spec dict attributes> ...": { },
            "dependencies": {
                "depname_3": {
                    "hash": "<old_dag_hash for depname_3>",
                    "type": ["build", "link"]
                },
                "depname_4": {
                    "hash": "<old_dag_hash for depname_4>",
                    "type": ["build", "link"]
                }
            },
            "hash": "<old_dag_hash 2>"
        }
    }
}

Version 3

Version 3 doesn’t change the top-level lockfile format, but this was when we changed the specfile format. Specs in concrete_specs are now keyed by the build hash, with no inner dictionary keyed by their package name. The package name is in a name field inside each spec dictionary. The dependencies field in the specs is a list instead of a dictionary, and each element of the list is a record with the name, dependency types, and hash of the dependency. Instead of a key called hash, dependencies are keyed by build_hash. Each spec still has a hash attribute.

Version 3 adds the specfile_version field to _meta and uses the new JSON spec format.

{
    "_meta": {
        "file-type": "spack-lockfile",
        "lockfile-version": 3,
        "specfile-version": 2
    },
    "roots": [
        {
            "hash": "<build_hash 1>",
            "spec": "<abstract spec 1>"
        },
        {
            "hash": "<build_hash 2>",
            "spec": "<abstract spec 2>"
        },
    ],
    "concrete_specs": {
        "<build_hash 1>": {
            "... <spec dict attributes> ...": { },
            "dependencies": [
                {
                    "name": "depname_1",
                    "build_hash": "<build_hash for depname_1>",
                    "type": ["build", "link"]
                },
                {
                    "name": "depname_2",
                    "build_hash": "<build_hash for depname_2>",
                    "type": ["build", "link"]
                },
            ],
            "hash": "<old_dag_hash 1>",
        },
        "<build_hash 2>": {
            "... <spec dict attributes> ...": { },
            "dependencies": [
                {
                    "name": "depname_3",
                    "build_hash": "<build_hash for depname_3>",
                    "type": ["build", "link"]
                },
                {
                    "name": "depname_4",
                    "build_hash": "<build_hash for depname_4>",
                    "type": ["build", "link"]
                },
            ],
            "hash": "<old_dag_hash 2>"
        }
    }
}

Version 4

Version 4 removes build hashes and is keyed by the new DAG hash (hash). The hash now includes build dependencies and a canonical hash of the package.py file. Dependencies are keyed by hash (DAG hash) as well. There are no more build_hash fields in the specs, and there are no more issues with lockfiles being able to store multiple specs with the same DAG hash (because the DAG hash is now finer-grained). An optional spack property may be included to track version information, such as the commit or version.

{
    "_meta": {
        "file-type": "spack-lockfile",
        "lockfile-version": 4,
        "specfile-version": 3
    },
    "roots": [
        {
            "hash": "<dag_hash 1>",
            "spec": "<abstract spec 1>"
        },
        {
            "hash": "<dag_hash 2>",
            "spec": "<abstract spec 2>"
        }
    ],
    "concrete_specs": {
        "<dag_hash 1>": {
            "... <spec dict attributes> ...": { },
            "dependencies": [
                {
                    "name": "depname_1",
                    "hash": "<dag_hash for depname_1>",
                    "type": ["build", "link"]
                },
                {
                    "name": "depname_2",
                    "hash": "<dag_hash for depname_2>",
                    "type": ["build", "link"]
                }
            ],
            "hash": "<dag_hash 1>",
        },
        "<daghash 2>": {
            "... <spec dict attributes> ...": { },
            "dependencies": [
                {
                    "name": "depname_3",
                    "hash": "<dag_hash for depname_3>",
                    "type": ["build", "link"]
                },
                {
                    "name": "depname_4",
                    "hash": "<dag_hash for depname_4>",
                    "type": ["build", "link"]
                }
            ],
            "hash": "<dag_hash 2>"
        }
    }
}

Version 5

Version 5 doesn’t change the top-level lockfile format, but an optional dictionary is added. The dictionary has the root and concrete_specs of the included environments, which are keyed by the path to that environment. Since this is optional if the environment does not have any included environments include_concrete will not be a part of the lockfile.

{
    "_meta": {
        "file-type": "spack-lockfile",
        "lockfile-version": 5,
        "specfile-version": 3
    },
    "roots": [
        {
            "hash": "<dag_hash 1>",
            "spec": "<abstract spec 1>"
        },
        {
            "hash": "<dag_hash 2>",
            "spec": "<abstract spec 2>"
        }
    ],
    "concrete_specs": {
        "<dag_hash 1>": {
            "... <spec dict attributes> ...": { },
            "dependencies": [
                {
                    "name": "depname_1",
                    "hash": "<dag_hash for depname_1>",
                    "type": ["build", "link"]
                },
                {
                    "name": "depname_2",
                    "hash": "<dag_hash for depname_2>",
                    "type": ["build", "link"]
                }
            ],
            "hash": "<dag_hash 1>",
        },
        "<daghash 2>": {
            "... <spec dict attributes> ...": { },
            "dependencies": [
                {
                    "name": "depname_3",
                    "hash": "<dag_hash for depname_3>",
                    "type": ["build", "link"]
                },
                {
                    "name": "depname_4",
                    "hash": "<dag_hash for depname_4>",
                    "type": ["build", "link"]
                }
            ],
            "hash": "<dag_hash 2>"
        }
    }
    "include_concrete": {
        "<path to environment>": {
            "roots": [
                {
                    "hash": "<dag_hash 1>",
                    "spec": "<abstract spec 1>"
                },
                {
                    "hash": "<dag_hash 2>",
                    "spec": "<abstract spec 2>"
                }
            ],
            "concrete_specs": {
                "<dag_hash 1>": {
                    "... <spec dict attributes> ...": { },
                    "dependencies": [
                        {
                            "name": "depname_1",
                            "hash": "<dag_hash for depname_1>",
                            "type": ["build", "link"]
                        },
                        {
                            "name": "depname_2",
                            "hash": "<dag_hash for depname_2>",
                            "type": ["build", "link"]
                        }
                    ],
                    "hash": "<dag_hash 1>",
                },
                "<daghash 2>": {
                    "... <spec dict attributes> ...": { },
                    "dependencies": [
                        {
                            "name": "depname_3",
                            "hash": "<dag_hash for depname_3>",
                            "type": ["build", "link"]
                        },
                        {
                            "name": "depname_4",
                            "hash": "<dag_hash for depname_4>",
                            "type": ["build", "link"]
                        }
                    ],
                    "hash": "<dag_hash 2>"
                }
            }
        }
    }
}
class spack.environment.Environment(manifest_dir: str | Path)[source]

Bases: object

A Spack environment, which bundles together configuration and a list of specs.

property active

True if this environment is currently active.

add(user_spec, list_name='specs')[source]

Add a single user_spec (non-concretized) to the Environment

Returns:

True if the spec was added, False if it was already

present and did not need to be added

Return type:

(bool)

add_view_to_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications[source]

Collect the environment modifications to activate an environment using the provided view. Removes duplicate paths.

Parameters:
  • env_mod – the environment modifications object that is modified.

  • view – the name of the view to activate.

added_specs()[source]

Specs that are not yet installed.

Yields the user spec for non-concretized specs, and the concrete spec for already concretized but not yet installed specs.

all_concretized_orders() List[str][source]

Returns all of the concretized order of the environment and its included environment(s).

all_concretized_user_specs() List[Spec][source]

Returns all of the concretized user specs of the environment and its included environment(s).

all_hashes()[source]

Return hashes of all specs.

all_matching_specs(*specs: Spec) List[Spec][source]

Returns all concretized specs in the environment satisfying any of the input specs

all_specs() List[Spec][source]

Returns a list of all concrete specs

all_specs_generator() Iterable[Spec][source]

Returns a generator for all concrete specs

change_existing_spec(change_spec: Spec, list_name: str = 'specs', match_spec: Spec | None = None, allow_changing_multiple_specs=False)[source]

Find the spec identified by match_spec and change it to change_spec.

Parameters:
  • change_spec – defines the spec properties that need to be changed. This will not change attributes of the matched spec unless they conflict with change_spec.

  • list_name – identifies the spec list in the environment that should be modified

  • match_spec – if set, this identifies the spec that should be changed. If not set, it is assumed we are looking for a spec with the same name as change_spec.

check_views()[source]

Checks if the environments default view can be activated.

clear(re_read=False)[source]

Clear the contents of the environment

Parameters:

re_read – If True, do not clear new_specs. This value cannot be read from yaml, and needs to be maintained when re-reading an existing environment.

concrete_roots()[source]

Same as concretized_specs, except it returns the list of concrete roots without associated user spec

concretize(force=False, tests=False)[source]

Concretize user_specs in this environment.

Only concretizes specs that haven’t been concretized yet unless force is True.

This only modifies the environment in memory. write() will write out a lockfile containing concretized specs.

Parameters:
  • force (bool) – re-concretize ALL specs, even those that were already concretized

  • tests (bool or list or set) – False to run no tests, True to test all packages, or a list of package names to run tests for some

Returns:

List of specs that have been concretized. Each entry is a tuple of the user spec and the corresponding concretized spec.

concretized_order: List[str]

Roots associated with the last concretization, in order

concretized_specs()[source]

Tuples of (user spec, concrete spec) for all concrete specs.

concretized_user_specs: List[Spec]

User specs from the last concretization

deconcretize(spec: Spec, concrete: bool = True)[source]

Remove specified spec from environment concretization

Parameters:
  • spec – Spec to deconcretize. This must be a root of the environment

  • concrete – If True, find all instances of spec as concrete in the environemnt. If False, find a single instance of the abstract spec as root of the environment.

property default_view
delete_default_view() None[source]

Deletes the default view associated with this environment.

destroy()[source]

Remove this environment from Spack entirely.

property dev_specs
ensure_env_directory_exists(dot_env: bool = False) None[source]

Ensure that the root directory of the environment exists

Parameters:

dot_env – if True also ensures that the <root>/.env directory exists

filter_specs(reader, json_specs_by_hash, order_concretized)[source]
get_by_hash(dag_hash: str) List[Spec][source]
get_one_by_hash(dag_hash)[source]

Returns the single spec from the environment which matches the provided hash. Raises an AssertionError if no specs match or if more than one spec matches.

has_view(view_name: str) bool[source]
include_concrete_envs()[source]

Copy and save the included envs’ specs internally

included_concrete_envs: List[str]

Environment paths for concrete (lockfile) included environments

included_concrete_spec_data: Dict[str, Dict[str, List[str]]]

First-level included concretized spec data from/to the lockfile.

included_concretized_order: Dict[str, List[str]]

Roots from included environments with the last concretization, in order

included_concretized_user_specs: Dict[str, List[Spec]]

User specs from included environments from the last concretization

included_specs_by_hash: Dict[str, Dict[str, Spec]]

Concretized specs by hash from the included environments

property included_user_specs: SpecList

Included concrete user (or root) specs from last concretization.

install_all(**install_args)[source]

Install all concretized specs in an environment.

Note: this does not regenerate the views for the environment; that needs to be done separately with a call to write().

Parameters:

install_args (dict) – keyword install arguments

install_specs(specs: List[Spec] | None = None, **install_args)[source]
invalidate_repository_cache()[source]
is_develop(spec)[source]

Returns true when the spec is built from local sources

property lock_path

Path to spack.lock file in this environment.

property manifest_path

Path to spack.yaml file in this environment.

manifest_uptodate_or_warn()[source]

Emits a warning if the manifest file is not up-to-date.

matching_spec(spec)[source]

Given a spec (likely not concretized), find a matching concretized spec in the environment.

The matching spec does not have to be installed in the environment, but must be concrete (specs added with spack add without an intervening spack concretize will not be matched).

If there is a single root spec that matches the provided spec or a single dependency spec that matches the provided spec, then the concretized instance of that spec will be returned.

If multiple root specs match the provided spec, or no root specs match and multiple dependency specs match, then this raises an error and reports all matching specs.

regenerate_views()[source]
remove(query_spec, list_name='specs', force=False)[source]

Remove specs from an environment that match a query_spec

removed_specs()[source]

Tuples of (user spec, concrete spec) for all specs that will be removed on next concretize.

property repo
property repos_path
rm_view_from_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications[source]

Collect the environment modifications to deactivate an environment using the provided view. Reverses the action of add_view_to_env.

Parameters:
  • env_mod – the environment modifications object that is modified.

  • view – the name of the view to deactivate.

roots()[source]

Specs explicitly requested by the user in this environment.

Yields both added and installed specs that have user specs in spack.yaml.

property scope_name

Name of the config scope of this environment’s manifest file.

set_included_concretized_user_specs(env_name: str, env_info: Dict[str, Dict[str, Any]], included_json_specs_by_hash: Dict[str, Dict[str, Any]]) Dict[str, Dict[str, Any]][source]

Sets all of the concretized user specs from included environments to include those from nested included environments.

Parameters:
  • env_name – the name (technically the path) of the included environment

  • env_info – included concrete environment data

  • included_json_specs_by_hash – concrete spec data keyed by hash

Returns: updated specs_by_hash

spec_lists: Dict[str, SpecList]

Specs from “spack.yaml”

specs_by_hash: Dict[str, Spec]

Concretized specs by hash

property unify
uninstalled_specs()[source]

Return root specs that are not installed, or are installed, but are development specs themselves or have those among their dependencies.

update_default_view(path_or_bool: str | bool) None[source]

Updates the path of the default view.

If the argument passed as input is False the default view is deleted, if present. The manifest will have an entry “view: false”.

If the argument passed as input is True a default view is created, if not already present. The manifest will have an entry “view: true”. If a default view is already declared, it will be left untouched.

If the argument passed as input is a path a default view pointing to that path is created, if not present already. If a default view is already declared, only its “root” will be changed.

Parameters:

path_or_bool – either True, or False or a path

update_environment_repository() None[source]

Updates the repository associated with the environment.

update_lockfile() None[source]
update_stale_references(from_list=None)[source]

Iterate over spec lists updating references.

property user_specs
property view_path_default
write(regenerate: bool = True) None[source]

Writes an in-memory environment to its location on disk.

Write out package files for each newly concretized spec. Also regenerate any views associated with the environment and run post-write hooks, if regenerate is True.

Parameters:

regenerate – regenerate views and run post-write hooks as well as writing if True.

write_transaction()[source]

Get a write lock context manager for use in a with block.

exception spack.environment.SpackEnvironmentConfigError(message, long_message=None)[source]

Bases: SpackEnvironmentError

Class for Spack environment-specific configuration errors.

exception spack.environment.SpackEnvironmentError(message, long_message=None)[source]

Bases: SpackError

Superclass for all errors to do with Spack environments.

exception spack.environment.SpackEnvironmentViewError(message, long_message=None)[source]

Bases: SpackEnvironmentError

Class for errors regarding view generation.

spack.environment.activate(env, use_env_repo=False)[source]

Activate an environment.

To activate an environment, we add its manifest’s configuration scope to the existing Spack configuration, and we set active to the current environment.

Parameters:
  • env (Environment) – the environment to activate

  • use_env_repo (bool) – use the packages exactly as they appear in the environment’s repository

spack.environment.active(name)[source]

True if the named environment is active.

spack.environment.active_environment() Environment | None[source]

Returns the active environment when there is any

spack.environment.all_environment_names()[source]

List the names of environments that currently exist.

spack.environment.all_environments()[source]

Generator for all managed Environments.

spack.environment.create(name: str, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment[source]

Create a managed environment in Spack and returns it.

A managed environment is created in a root directory managed by this Spack instance, so that Spack can keep track of them.

Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.

Parameters:
  • name – name of the managed environment

  • init_file – either a lockfile, a manifest file, or None

  • with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view

  • keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute

  • include_concrete – list of concrete environment names/paths to be included

spack.environment.create_in_dir(root: str | Path, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment[source]

Create an environment in the directory passed as input and returns it.

Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.

Parameters:
  • root – directory where to create the environment.

  • init_file – either a lockfile, a manifest file, or None

  • with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view

  • keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute

  • include_concrete – concrete environment names/paths to be included

spack.environment.deactivate()[source]

Undo any configuration or repo settings modified by activate().

spack.environment.default_manifest_yaml()[source]

default spack.yaml file to put in new environments

spack.environment.display_specs(specs)[source]

Displays a list of specs traversed breadth-first, covering nodes, with install status.

Parameters:

specs (list) – list of specs

spack.environment.environment_dir_from_name(name: str, exists_ok: bool = True) str[source]

Returns the directory associated with a named environment.

Parameters:
  • name – name of the environment

  • exists_ok – if False, raise an error if the environment exists already

Raises:

SpackEnvironmentError – if exists_ok is False and the environment exists already

spack.environment.exists(name)[source]

Whether an environment with this name exists or not.

spack.environment.initialize_environment_dir(environment_dir: str | Path, envfile: str | Path | None) None[source]

Initialize an environment directory starting from an envfile.

Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.

Parameters:
  • environment_dir – directory where the environment should be placed

  • envfile – manifest file or lockfile used to initialize the environment

Raises:

SpackEnvironmentError – if the directory can’t be initialized

spack.environment.installed_specs()[source]

Returns the specs of packages installed in the active environment or None if no packages are installed.

spack.environment.is_env_dir(path)[source]

Whether a directory contains a spack environment.

spack.environment.is_latest_format(manifest)[source]

Return False if the manifest file exists and is not in the latest schema format.

Parameters:

manifest (str) – manifest file to be analyzed

spack.environment.manifest_file(env_name_or_dir)[source]

Return the absolute path to a manifest file given the environment name or directory.

Parameters:

env_name_or_dir (str) – either the name of a valid environment or a directory where a manifest file resides

Raises:

AssertionError – if the environment is not found

spack.environment.no_active_environment()[source]

Deactivate the active environment for the duration of the context. Has no effect when there is no active environment.

spack.environment.read(name)[source]

Get an environment with the supplied name.

spack.environment.root(name)[source]

Get the root directory for an environment by name.

spack.environment.update_yaml(manifest, backup_file)[source]

Update a manifest file from an old format to the current one.

Parameters:
  • manifest (str) – path to a manifest file

  • backup_file (str) – file where to copy the original manifest

Returns:

True if the manifest was updated, False otherwise.

Raises:

AssertionError – in case anything goes wrong during the update

Submodules

spack.environment.depfile module

This module contains the traversal logic and models that can be used to generate depfiles from an environment.

class spack.environment.depfile.DepfileNode(target: Spec, prereqs: List[Spec], buildcache: UseBuildCache)[source]

Bases: object

Contains a spec, a subset of its dependencies, and a flag whether it should be buildcache only/never/auto.

class spack.environment.depfile.DepfileSpecVisitor(pkg_buildcache: UseBuildCache, deps_buildcache: UseBuildCache)[source]

Bases: object

This visitor produces an adjacency list of a (reduced) DAG, which is used to generate depfile targets with their prerequisites. Currently it only drops build deps when using buildcache only mode.

Note that the DAG could be reduced even more by dropping build edges of specs installed at the moment the depfile is generated, but that would produce stateful depfiles that would not fail when the database is wiped later.

accept(node)[source]
neighbors(node)[source]

Produce a list of spec to follow from node

class spack.environment.depfile.MakefileModel(env: Environment, roots: List[Spec], adjacency_list: List[DepfileNode], make_prefix: str | None, jobserver: bool)[source]

Bases: object

This class produces all data to render a makefile for specs of an environment.

property empty
static from_env(env: Environment, *, filter_specs: List[Spec] | None = None, pkg_buildcache: UseBuildCache = UseBuildCache.AUTO, dep_buildcache: UseBuildCache = UseBuildCache.AUTO, make_prefix: str | None = None, jobserver: bool = True) MakefileModel[source]

Produces a MakefileModel from an environment and a list of specs.

Parameters:
  • env – the environment to use

  • filter_specs – if provided, only these specs will be built from the environment, otherwise the environment roots are used.

  • pkg_buildcache – whether to only use the buildcache for top-level specs.

  • dep_buildcache – whether to only use the buildcache for non-top-level specs.

  • make_prefix – the prefix for the makefile targets

  • jobserver – when enabled, make will invoke Spack with jobserver support. For dry-run this should be disabled.

to_dict()[source]
class spack.environment.depfile.MakefileSpec(spec)[source]

Bases: object

Limited interface to spec to help generate targets etc. without introducing unwanted special characters.

safe_format(format_str)[source]
safe_name()[source]
spec_hash()[source]
unsafe_format(format_str)[source]
class spack.environment.depfile.UseBuildCache(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

AUTO = 3
NEVER = 2
ONLY = 1
static from_string(s: str) UseBuildCache[source]

spack.environment.environment module

class spack.environment.environment.Environment(manifest_dir: str | Path)[source]

Bases: object

A Spack environment, which bundles together configuration and a list of specs.

property active

True if this environment is currently active.

add(user_spec, list_name='specs')[source]

Add a single user_spec (non-concretized) to the Environment

Returns:

True if the spec was added, False if it was already

present and did not need to be added

Return type:

(bool)

add_view_to_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications[source]

Collect the environment modifications to activate an environment using the provided view. Removes duplicate paths.

Parameters:
  • env_mod – the environment modifications object that is modified.

  • view – the name of the view to activate.

added_specs()[source]

Specs that are not yet installed.

Yields the user spec for non-concretized specs, and the concrete spec for already concretized but not yet installed specs.

all_concretized_orders() List[str][source]

Returns all of the concretized order of the environment and its included environment(s).

all_concretized_user_specs() List[Spec][source]

Returns all of the concretized user specs of the environment and its included environment(s).

all_hashes()[source]

Return hashes of all specs.

all_matching_specs(*specs: Spec) List[Spec][source]

Returns all concretized specs in the environment satisfying any of the input specs

all_specs() List[Spec][source]

Returns a list of all concrete specs

all_specs_generator() Iterable[Spec][source]

Returns a generator for all concrete specs

change_existing_spec(change_spec: Spec, list_name: str = 'specs', match_spec: Spec | None = None, allow_changing_multiple_specs=False)[source]

Find the spec identified by match_spec and change it to change_spec.

Parameters:
  • change_spec – defines the spec properties that need to be changed. This will not change attributes of the matched spec unless they conflict with change_spec.

  • list_name – identifies the spec list in the environment that should be modified

  • match_spec – if set, this identifies the spec that should be changed. If not set, it is assumed we are looking for a spec with the same name as change_spec.

check_views()[source]

Checks if the environments default view can be activated.

clear(re_read=False)[source]

Clear the contents of the environment

Parameters:

re_read – If True, do not clear new_specs. This value cannot be read from yaml, and needs to be maintained when re-reading an existing environment.

concrete_roots()[source]

Same as concretized_specs, except it returns the list of concrete roots without associated user spec

concretize(force=False, tests=False)[source]

Concretize user_specs in this environment.

Only concretizes specs that haven’t been concretized yet unless force is True.

This only modifies the environment in memory. write() will write out a lockfile containing concretized specs.

Parameters:
  • force (bool) – re-concretize ALL specs, even those that were already concretized

  • tests (bool or list or set) – False to run no tests, True to test all packages, or a list of package names to run tests for some

Returns:

List of specs that have been concretized. Each entry is a tuple of the user spec and the corresponding concretized spec.

concretized_order: List[str]

Roots associated with the last concretization, in order

concretized_specs()[source]

Tuples of (user spec, concrete spec) for all concrete specs.

concretized_user_specs: List[Spec]

User specs from the last concretization

deconcretize(spec: Spec, concrete: bool = True)[source]

Remove specified spec from environment concretization

Parameters:
  • spec – Spec to deconcretize. This must be a root of the environment

  • concrete – If True, find all instances of spec as concrete in the environemnt. If False, find a single instance of the abstract spec as root of the environment.

property default_view
delete_default_view() None[source]

Deletes the default view associated with this environment.

destroy()[source]

Remove this environment from Spack entirely.

property dev_specs
ensure_env_directory_exists(dot_env: bool = False) None[source]

Ensure that the root directory of the environment exists

Parameters:

dot_env – if True also ensures that the <root>/.env directory exists

filter_specs(reader, json_specs_by_hash, order_concretized)[source]
get_by_hash(dag_hash: str) List[Spec][source]
get_one_by_hash(dag_hash)[source]

Returns the single spec from the environment which matches the provided hash. Raises an AssertionError if no specs match or if more than one spec matches.

has_view(view_name: str) bool[source]
include_concrete_envs()[source]

Copy and save the included envs’ specs internally

included_concrete_envs: List[str]

Environment paths for concrete (lockfile) included environments

included_concrete_spec_data: Dict[str, Dict[str, List[str]]]

First-level included concretized spec data from/to the lockfile.

included_concretized_order: Dict[str, List[str]]

Roots from included environments with the last concretization, in order

included_concretized_user_specs: Dict[str, List[Spec]]

User specs from included environments from the last concretization

included_specs_by_hash: Dict[str, Dict[str, Spec]]

Concretized specs by hash from the included environments

property included_user_specs: SpecList

Included concrete user (or root) specs from last concretization.

install_all(**install_args)[source]

Install all concretized specs in an environment.

Note: this does not regenerate the views for the environment; that needs to be done separately with a call to write().

Parameters:

install_args (dict) – keyword install arguments

install_specs(specs: List[Spec] | None = None, **install_args)[source]
invalidate_repository_cache()[source]
is_develop(spec)[source]

Returns true when the spec is built from local sources

property lock_path

Path to spack.lock file in this environment.

property manifest_path

Path to spack.yaml file in this environment.

manifest_uptodate_or_warn()[source]

Emits a warning if the manifest file is not up-to-date.

matching_spec(spec)[source]

Given a spec (likely not concretized), find a matching concretized spec in the environment.

The matching spec does not have to be installed in the environment, but must be concrete (specs added with spack add without an intervening spack concretize will not be matched).

If there is a single root spec that matches the provided spec or a single dependency spec that matches the provided spec, then the concretized instance of that spec will be returned.

If multiple root specs match the provided spec, or no root specs match and multiple dependency specs match, then this raises an error and reports all matching specs.

new_specs: List[Spec]
regenerate_views()[source]
remove(query_spec, list_name='specs', force=False)[source]

Remove specs from an environment that match a query_spec

removed_specs()[source]

Tuples of (user spec, concrete spec) for all specs that will be removed on next concretize.

property repo
property repos_path
rm_view_from_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications[source]

Collect the environment modifications to deactivate an environment using the provided view. Reverses the action of add_view_to_env.

Parameters:
  • env_mod – the environment modifications object that is modified.

  • view – the name of the view to deactivate.

roots()[source]

Specs explicitly requested by the user in this environment.

Yields both added and installed specs that have user specs in spack.yaml.

property scope_name

Name of the config scope of this environment’s manifest file.

set_included_concretized_user_specs(env_name: str, env_info: Dict[str, Dict[str, Any]], included_json_specs_by_hash: Dict[str, Dict[str, Any]]) Dict[str, Dict[str, Any]][source]

Sets all of the concretized user specs from included environments to include those from nested included environments.

Parameters:
  • env_name – the name (technically the path) of the included environment

  • env_info – included concrete environment data

  • included_json_specs_by_hash – concrete spec data keyed by hash

Returns: updated specs_by_hash

spec_lists: Dict[str, SpecList]

Specs from “spack.yaml”

specs_by_hash: Dict[str, Spec]

Concretized specs by hash

property unify
uninstalled_specs()[source]

Return root specs that are not installed, or are installed, but are development specs themselves or have those among their dependencies.

update_default_view(path_or_bool: str | bool) None[source]

Updates the path of the default view.

If the argument passed as input is False the default view is deleted, if present. The manifest will have an entry “view: false”.

If the argument passed as input is True a default view is created, if not already present. The manifest will have an entry “view: true”. If a default view is already declared, it will be left untouched.

If the argument passed as input is a path a default view pointing to that path is created, if not present already. If a default view is already declared, only its “root” will be changed.

Parameters:

path_or_bool – either True, or False or a path

update_environment_repository() None[source]

Updates the repository associated with the environment.

update_lockfile() None[source]
update_stale_references(from_list=None)[source]

Iterate over spec lists updating references.

property user_specs
property view_path_default
views: Dict[str, ViewDescriptor]
write(regenerate: bool = True) None[source]

Writes an in-memory environment to its location on disk.

Write out package files for each newly concretized spec. Also regenerate any views associated with the environment and run post-write hooks, if regenerate is True.

Parameters:

regenerate – regenerate views and run post-write hooks as well as writing if True.

write_transaction()[source]

Get a write lock context manager for use in a with block.

class spack.environment.environment.EnvironmentManifestFile(manifest_dir: Path | str)[source]

Bases: Mapping

Manages the in-memory representation of a manifest file, and its synchronization with the actual manifest on disk.

add_definition(user_spec: str, list_name: str) None[source]

Appends a user spec to the first active definition matching the name passed as argument.

Parameters:
  • user_spec – user spec to be appended

  • list_name – name of the definition where to append

Raises:

SpackEnvironmentError – is no valid definition exists already

add_user_spec(user_spec: str) None[source]

Appends the user spec passed as input to the list of root specs.

Parameters:

user_spec – user spec to be appended

property configuration

Return the dictionaries in the YAML, without the top level attribute

deactivate_config_scope() None[source]

Remove any of the manifest’s scopes from the global config path.

property env_config_scopes: List[ConfigScope]

A list of all configuration scopes for the environment manifest. On the first call this instantiates all the scopes, on subsequent calls it returns the cached list.

flush() None[source]

Synchronizes the object with the manifest file on disk.

static from_lockfile(manifest_dir: Path | str) EnvironmentManifestFile[source]

Returns an environment manifest file compatible with the lockfile already present in the environment directory.

This function also writes a spack.yaml file that is consistent with the spack.lock already existing in the directory.

Parameters:

manifest_dir – directory containing the manifest and lockfile

property included_config_scopes: List[ConfigScope]

List of included configuration scopes from the manifest.

Scopes are listed in the YAML file in order from highest to lowest precedence, so configuration from earlier scope will take precedence over later ones.

This routine returns them in the order they should be pushed onto the internal scope stack (so, in reverse, from lowest to highest).

Returns: Configuration scopes associated with the environment manifest

Raises:

SpackEnvironmentError – if the manifest includes a remote file but no configuration stage directory has been identified

override_definition(user_spec: str, *, override: str, list_name: str) None[source]

Overrides a user spec from an active definition that matches the name passed as argument.

Parameters:
  • user_spec – user spec to be overridden

  • override – new spec to be used

  • list_name – name of the definition where to override the spec

Raises:

SpackEnvironmentError – if the user spec cannot be overridden

override_user_spec(user_spec: str, idx: int) None[source]

Overrides the user spec at index idx with the one passed as input.

Parameters:
  • user_spec – new user spec

  • idx – index of the spec to be overridden

Raises:

SpackEnvironmentError – when the user spec cannot be overridden

prepare_config_scope() None[source]

Add the manifest’s scopes to the global configuration search path.

property pristine_configuration

Return the dictionaries in the pristine YAML, without the top level attribute

pristine_yaml_content

Pristine YAML content, without defaults being added

remove_default_view() None[source]

Removes the default view from the manifest file

remove_definition(user_spec: str, list_name: str) None[source]

Removes a user spec from an active definition that matches the name passed as argument.

Parameters:
  • user_spec – user spec to be removed

  • list_name – name of the definition where to remove the spec from

Raises:

SpackEnvironmentError – if the user spec cannot be removed from the list, or the list does not exist

remove_user_spec(user_spec: str) None[source]

Removes the user spec passed as input from the list of root specs

Parameters:

user_spec – user spec to be removed

Raises:

SpackEnvironmentError – when the user spec is not in the list

set_default_view(view: bool | str | Path | Dict[str, str]) None[source]

Sets the default view root in the manifest to the value passed as input.

Parameters:

view – If the value is a string or a path, it specifies the path to the view. If True the default view is used for the environment, if False there’s no view.

set_include_concrete(include_concrete: List[str]) None[source]

Sets the included concrete environments in the manifest to the value(s) passed as input.

Parameters:

include_concrete – list of already existing concrete environments to include

use_config()[source]

Ensure only the manifest’s configuration scopes are global.

yaml_content

YAML content with defaults added by Spack, if they’re missing

exception spack.environment.environment.SpackEnvironmentConfigError(message, long_message=None)[source]

Bases: SpackEnvironmentError

Class for Spack environment-specific configuration errors.

exception spack.environment.environment.SpackEnvironmentError(message, long_message=None)[source]

Bases: SpackError

Superclass for all errors to do with Spack environments.

exception spack.environment.environment.SpackEnvironmentViewError(message, long_message=None)[source]

Bases: SpackEnvironmentError

Class for errors regarding view generation.

class spack.environment.environment.ViewDescriptor(base_path, root, projections={}, select=[], exclude=[], link='all', link_type='symlink')[source]

Bases: object

content_hash(specs)[source]
exclude_fn(spec)[source]
static from_dict(base_path, d)[source]
get_projection_for_spec(spec)[source]

Get projection for spec. This function does not require the view to exist on the filesystem.

regenerate(concrete_roots: List[Spec]) None[source]
select_fn(spec)[source]
specs_for_view(concrete_roots: List[Spec]) List[Spec][source]

Flatten the DAGs of the concrete roots, keep only unique, selected, and installed specs in topological order from root to leaf.

to_dict()[source]
update_root(new_path)[source]
view(new: str | None = None) SimpleFilesystemView[source]

Returns a view object for the underlying view directory. This means that the self.root symlink is followed, and that the view has to exist on the filesystem (unless new). This function is useful when writing to the view.

Raise if new is None and there is no current view

Parameters:

new – If a string, create a FilesystemView rooted at that path. Default None. This should only be used to regenerate the view, and cannot be used to access specs.

spack.environment.environment.activate(env, use_env_repo=False)[source]

Activate an environment.

To activate an environment, we add its manifest’s configuration scope to the existing Spack configuration, and we set active to the current environment.

Parameters:
  • env (Environment) – the environment to activate

  • use_env_repo (bool) – use the packages exactly as they appear in the environment’s repository

spack.environment.environment.active(name)[source]

True if the named environment is active.

spack.environment.environment.active_environment() Environment | None[source]

Returns the active environment when there is any

spack.environment.environment.all_environment_names()[source]

List the names of environments that currently exist.

spack.environment.environment.all_environments()[source]

Generator for all managed Environments.

spack.environment.environment.create(name: str, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment[source]

Create a managed environment in Spack and returns it.

A managed environment is created in a root directory managed by this Spack instance, so that Spack can keep track of them.

Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.

Parameters:
  • name – name of the managed environment

  • init_file – either a lockfile, a manifest file, or None

  • with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view

  • keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute

  • include_concrete – list of concrete environment names/paths to be included

spack.environment.environment.create_in_dir(root: str | Path, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment[source]

Create an environment in the directory passed as input and returns it.

Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.

Parameters:
  • root – directory where to create the environment.

  • init_file – either a lockfile, a manifest file, or None

  • with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view

  • keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute

  • include_concrete – concrete environment names/paths to be included

spack.environment.environment.deactivate()[source]

Undo any configuration or repo settings modified by activate().

spack.environment.environment.default_env_path = '/home/docs/checkouts/readthedocs.org/user_builds/spack/checkouts/latest/lib/spack/docs/_spack_root/var/spack/environments'

default path where environments are stored in the spack tree

spack.environment.environment.default_manifest_yaml()[source]

default spack.yaml file to put in new environments

spack.environment.environment.display_specs(specs)[source]

Displays a list of specs traversed breadth-first, covering nodes, with install status.

Parameters:

specs (list) – list of specs

spack.environment.environment.ensure_env_root_path_exists()[source]
spack.environment.environment.ensure_no_disallowed_env_config_mods(scopes: List[ConfigScope]) None[source]
spack.environment.environment.env_root_path() str[source]

Override default root path if the user specified it

spack.environment.environment.env_subdir_name = '.spack-env'

Name of the directory where environments store repos, logs, views, configs

spack.environment.environment.env_subdir_path(manifest_dir: str | Path) str[source]

Path to where the environment stores repos, logs, views, configs.

Parameters:

manifest_dir – directory containing the environment manifest file

Returns: directory the environment uses to manage its files

spack.environment.environment.environment_dir_from_name(name: str, exists_ok: bool = True) str[source]

Returns the directory associated with a named environment.

Parameters:
  • name – name of the environment

  • exists_ok – if False, raise an error if the environment exists already

Raises:

SpackEnvironmentError – if exists_ok is False and the environment exists already

spack.environment.environment.environment_name(path: str | Path) str[source]

Human-readable representation of the environment.

This is the path for directory environments, and just the name for managed environments.

spack.environment.environment.exists(name)[source]

Whether an environment with this name exists or not.

spack.environment.environment.initialize_environment_dir(environment_dir: str | Path, envfile: str | Path | None) None[source]

Initialize an environment directory starting from an envfile.

Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.

Parameters:
  • environment_dir – directory where the environment should be placed

  • envfile – manifest file or lockfile used to initialize the environment

Raises:

SpackEnvironmentError – if the directory can’t be initialized

spack.environment.environment.installed_specs()[source]

Returns the specs of packages installed in the active environment or None if no packages are installed.

spack.environment.environment.is_env_dir(path)[source]

Whether a directory contains a spack environment.

spack.environment.environment.is_latest_format(manifest)[source]

Return False if the manifest file exists and is not in the latest schema format.

Parameters:

manifest (str) – manifest file to be analyzed

spack.environment.environment.lockfile_format_version = 5

version of the lockfile format. Must increase monotonically.

spack.environment.environment.lockfile_name = 'spack.lock'

Name of the input yaml file for an environment

spack.environment.environment.make_repo_path(root)[source]

Make a RepoPath from the repo subdirectories in an environment.

spack.environment.environment.manifest_file(env_name_or_dir)[source]

Return the absolute path to a manifest file given the environment name or directory.

Parameters:

env_name_or_dir (str) – either the name of a valid environment or a directory where a manifest file resides

Raises:

AssertionError – if the environment is not found

spack.environment.environment.manifest_name = 'spack.yaml'

Name of the input yaml file for an environment

spack.environment.environment.no_active_environment()[source]

Deactivate the active environment for the duration of the context. Has no effect when there is no active environment.

spack.environment.environment.read(name)[source]

Get an environment with the supplied name.

spack.environment.environment.root(name)[source]

Get the root directory for an environment by name.

spack.environment.environment.set_included_envs_to_env_paths(include_concrete: List[str]) None[source]

If the included environment(s) is the environment name it is replaced by the path to the environment

Parameters:

include_concrete – list of env name or path to env

spack.environment.environment.spack_env_var = 'SPACK_ENV'

environment variable used to indicate the active environment

spack.environment.environment.spack_env_view_var = 'SPACK_ENV_VIEW'

environment variable used to indicate the active environment view

spack.environment.environment.update_yaml(manifest, backup_file)[source]

Update a manifest file from an old format to the current one.

Parameters:
  • manifest (str) – path to a manifest file

  • backup_file (str) – file where to copy the original manifest

Returns:

True if the manifest was updated, False otherwise.

Raises:

AssertionError – in case anything goes wrong during the update

spack.environment.environment.valid_env_name(name)[source]
spack.environment.environment.valid_environment_name_re = '^\\w[\\w-]*$'

regex for validating enviroment names

spack.environment.environment.validate_env_name(name)[source]
spack.environment.environment.validate_included_envs_concrete(include_concrete: List[str]) None[source]

Checks that all of the included environments are concrete

Parameters:

include_concrete – list of already existing concrete environments to include

Raises:

SpackEnvironmentError – if any of the included environments are not concrete

spack.environment.environment.validate_included_envs_exists(include_concrete: List[str]) None[source]

Checks that all of the included environments exist

Parameters:

include_concrete – list of already existing concrete environments to include

Raises:

SpackEnvironmentError – if any of the included environments do not exist

spack.environment.environment.yaml_equivalent(first, second) bool[source]

Returns whether two spack yaml items are equivalent, including overrides

spack.environment.shell module

spack.environment.shell.activate(env: Environment, use_env_repo=False, view: str | None = 'default') EnvironmentModifications[source]

Activate an environment and append environment modifications

To activate an environment, we add its configuration scope to the existing Spack configuration, and we set active to the current environment.

Parameters:
  • env – the environment to activate

  • use_env_repo – use the packages exactly as they appear in the environment’s repository

  • view – generate commands to add runtime environment variables for named view

Returns:

Environment variables modifications to activate environment.

Return type:

spack.util.environment.EnvironmentModifications

spack.environment.shell.activate_header(env, shell, prompt=None, view: str | None = None)[source]
spack.environment.shell.deactivate() EnvironmentModifications[source]

Deactivate an environment and collect corresponding environment modifications.

Note: unloads the environment in its current state, not in the state it was

loaded in, meaning that specs that were removed from the spack environment after activation are not unloaded.

Returns:

Environment variables modifications to activate environment.

spack.environment.shell.deactivate_header(shell)[source]