spack.environment package
This package implements Spack environments.
spack.lock format
Spack environments have existed since Spack v0.12.0
, and there have been 4 different
spack.lock
formats since then. The formats are documented here.
The high-level format of a Spack lockfile hasn’t changed much between versions, but the contents have. Lockfiles are JSON-formatted and their top-level sections are:
_meta
(object): this contains details about the file format, including:
file-type
: always"spack-lockfile"
lockfile-version
: an integer representing the lockfile format version
specfile-version
: an integer representing the spec format version (sincev0.17
)
spack
(object): optional, this identifies information about Spackused to concretize the environment: *
type
: required, identifies form Spack version took (e.g.,git
,release
) *commit
: the commit if the version is from git *version
: the Spack version
roots
(list): an ordered list of records representing the roots of the Spackenvironment. Each has two fields: *
hash
: a Spack spec hash uniquely identifying the concrete root spec *spec
: a string representation of the abstract spec that was concretized
concrete_specs
: a dictionary containing the specs in the environment.
include_concrete
(dictionary): an optional dictionary that includes the roots and concrete specs from the included environments, keyed by the path to that environment
Compatibility
New versions of Spack can (so far) read all old lockfile formats – they are backward-compatible. Old versions cannot read new lockfile formats, and you’ll need to upgrade Spack to use them.
Spack version |
|
|
|
|
|
---|---|---|---|---|---|
|
✅ |
||||
|
✅ |
✅ |
|||
|
✅ |
✅ |
✅ |
||
|
✅ |
✅ |
✅ |
✅ |
|
|
✅ |
✅ |
✅ |
✅ |
✅ |
Version 1
When lockfiles were first created, there was only one hash in Spack: the DAG hash. This DAG hash (we’ll call it the old DAG hash) did not include build dependencies – it only included transitive link and run dependencies.
The spec format at this time was keyed by name. Each spec started with a key for its name, whose value was a dictionary of other spec attributes. The lockfile put these name-keyed specs into dictionaries keyed by their DAG hash, and the spec records did not actually have a “hash” field in the lockfile – you have to associate the hash from the key with the spec record after the fact.
Dependencies in original lockfiles were keyed by "hash"
, i.e. the old DAG hash.
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 1
},
"roots": [
{
"hash": "<old_dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<old_dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<old_dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_1": {
"hash": "<old_dag_hash for depname_1>",
"type": ["build", "link"]
},
"depname_2": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
}
},
"hash": "<old_dag_hash 1>"
},
"<old_dag_hash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_3": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
},
"depname_4": {
"hash": "<old_dag_hash for depname_4>",
"type": ["build", "link"]
},
},
"hash": "<old_dag_hash 2>"
},
}
}
Version 2
Version 2 changes one thing: specs in the lockfile are now keyed by build_hash
instead of the old dag_hash
. Specs have a hash
attribute with their real DAG
hash, so you can’t go by the dictionary key anymore to identify a spec – you have to
read it in and look at "hash"
. Dependencies are still keyed by old DAG hash.
Even though we key lockfiles by build_hash
, specs in Spack were still deployed with
the old, coarser DAG hash. This means that in v2 and v3 lockfiles (which are keyed by
build hash), there may be multiple versions of the same spec with different build
dependencies, which means they will have different build hashes but the same DAG hash.
Spack would only have been able to actually install one of these.
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 2
},
"roots": [
{
"hash": "<build_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<build_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<build_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_1": {
"hash": "<old_dag_hash for depname_1>",
"type": ["build", "link"]
},
"depname_2": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
}
},
"hash": "<old_dag_hash 1>",
},
"<build_hash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": {
"depname_3": {
"hash": "<old_dag_hash for depname_3>",
"type": ["build", "link"]
},
"depname_4": {
"hash": "<old_dag_hash for depname_4>",
"type": ["build", "link"]
}
},
"hash": "<old_dag_hash 2>"
}
}
}
Version 3
Version 3 doesn’t change the top-level lockfile format, but this was when we changed the
specfile format. Specs in concrete_specs
are now keyed by the build hash, with no
inner dictionary keyed by their package name. The package name is in a name
field
inside each spec dictionary. The dependencies
field in the specs is a list instead
of a dictionary, and each element of the list is a record with the name, dependency
types, and hash of the dependency. Instead of a key called hash
, dependencies are
keyed by build_hash
. Each spec still has a hash
attribute.
Version 3 adds the specfile_version
field to _meta
and uses the new JSON spec
format.
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 3,
"specfile-version": 2
},
"roots": [
{
"hash": "<build_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<build_hash 2>",
"spec": "<abstract spec 2>"
},
],
"concrete_specs": {
"<build_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"build_hash": "<build_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"build_hash": "<build_hash for depname_2>",
"type": ["build", "link"]
},
],
"hash": "<old_dag_hash 1>",
},
"<build_hash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"build_hash": "<build_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"build_hash": "<build_hash for depname_4>",
"type": ["build", "link"]
},
],
"hash": "<old_dag_hash 2>"
}
}
}
Version 4
Version 4 removes build hashes and is keyed by the new DAG hash (hash
). The hash
now includes build dependencies and a canonical hash of the package.py
file.
Dependencies are keyed by hash
(DAG hash) as well. There are no more build_hash
fields in the specs, and there are no more issues with lockfiles being able to store
multiple specs with the same DAG hash (because the DAG hash is now finer-grained).
An optional spack
property may be included to track version information, such as
the commit or version.
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 4,
"specfile-version": 3
},
"roots": [
{
"hash": "<dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"hash": "<dag_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"hash": "<dag_hash for depname_2>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 1>",
},
"<daghash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"hash": "<dag_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"hash": "<dag_hash for depname_4>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 2>"
}
}
}
Version 5
Version 5 doesn’t change the top-level lockfile format, but an optional dictionary is
added. The dictionary has the root
and concrete_specs
of the included
environments, which are keyed by the path to that environment. Since this is optional
if the environment does not have any included environments include_concrete
will
not be a part of the lockfile.
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 5,
"specfile-version": 3
},
"roots": [
{
"hash": "<dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"hash": "<dag_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"hash": "<dag_hash for depname_2>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 1>",
},
"<daghash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"hash": "<dag_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"hash": "<dag_hash for depname_4>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 2>"
}
}
"include_concrete": {
"<path to environment>": {
"roots": [
{
"hash": "<dag_hash 1>",
"spec": "<abstract spec 1>"
},
{
"hash": "<dag_hash 2>",
"spec": "<abstract spec 2>"
}
],
"concrete_specs": {
"<dag_hash 1>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_1",
"hash": "<dag_hash for depname_1>",
"type": ["build", "link"]
},
{
"name": "depname_2",
"hash": "<dag_hash for depname_2>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 1>",
},
"<daghash 2>": {
"... <spec dict attributes> ...": { },
"dependencies": [
{
"name": "depname_3",
"hash": "<dag_hash for depname_3>",
"type": ["build", "link"]
},
{
"name": "depname_4",
"hash": "<dag_hash for depname_4>",
"type": ["build", "link"]
}
],
"hash": "<dag_hash 2>"
}
}
}
}
}
- class spack.environment.Environment(manifest_dir: str | Path)[source]
Bases:
object
A Spack environment, which bundles together configuration and a list of specs.
- property active
True if this environment is currently active.
- add(user_spec, list_name='specs')[source]
Add a single user_spec (non-concretized) to the Environment
- Returns:
- True if the spec was added, False if it was already
present and did not need to be added
- Return type:
(bool)
- add_view_to_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications [source]
Collect the environment modifications to activate an environment using the provided view. Removes duplicate paths.
- Parameters:
env_mod – the environment modifications object that is modified.
view – the name of the view to activate.
- added_specs()[source]
Specs that are not yet installed.
Yields the user spec for non-concretized specs, and the concrete spec for already concretized but not yet installed specs.
- all_concretized_orders() List[str] [source]
Returns all of the concretized order of the environment and its included environment(s).
- all_concretized_user_specs() List[Spec] [source]
Returns all of the concretized user specs of the environment and its included environment(s).
- all_matching_specs(*specs: Spec) List[Spec] [source]
Returns all concretized specs in the environment satisfying any of the input specs
- change_existing_spec(change_spec: Spec, list_name: str = 'specs', match_spec: Spec | None = None, allow_changing_multiple_specs=False)[source]
Find the spec identified by match_spec and change it to change_spec.
- Parameters:
change_spec – defines the spec properties that need to be changed. This will not change attributes of the matched spec unless they conflict with change_spec.
list_name – identifies the spec list in the environment that should be modified
match_spec – if set, this identifies the spec that should be changed. If not set, it is assumed we are looking for a spec with the same name as change_spec.
- clear(re_read=False)[source]
Clear the contents of the environment
- Parameters:
re_read – If
True
, do not clearnew_specs
. This value cannot be read from yaml, and needs to be maintained when re-reading an existing environment.
- concrete_roots()[source]
Same as concretized_specs, except it returns the list of concrete roots without associated user spec
- concretize(force=False, tests=False)[source]
Concretize user_specs in this environment.
Only concretizes specs that haven’t been concretized yet unless force is
True
.This only modifies the environment in memory.
write()
will write out a lockfile containing concretized specs.- Parameters:
- Returns:
List of specs that have been concretized. Each entry is a tuple of the user spec and the corresponding concretized spec.
- deconcretize(spec: Spec, concrete: bool = True)[source]
Remove specified spec from environment concretization
- Parameters:
spec – Spec to deconcretize. This must be a root of the environment
concrete – If True, find all instances of spec as concrete in the environemnt. If False, find a single instance of the abstract spec as root of the environment.
- property default_view
- property dev_specs
- ensure_env_directory_exists(dot_env: bool = False) None [source]
Ensure that the root directory of the environment exists
- Parameters:
dot_env – if True also ensures that the <root>/.env directory exists
- get_one_by_hash(dag_hash)[source]
Returns the single spec from the environment which matches the provided hash. Raises an AssertionError if no specs match or if more than one spec matches.
- included_concrete_spec_data: Dict[str, Dict[str, List[str]]]
First-level included concretized spec data from/to the lockfile.
- included_concretized_order: Dict[str, List[str]]
Roots from included environments with the last concretization, in order
- included_concretized_user_specs: Dict[str, List[Spec]]
User specs from included environments from the last concretization
- included_specs_by_hash: Dict[str, Dict[str, Spec]]
Concretized specs by hash from the included environments
- property included_user_specs: SpecList
Included concrete user (or root) specs from last concretization.
- install_all(**install_args)[source]
Install all concretized specs in an environment.
Note: this does not regenerate the views for the environment; that needs to be done separately with a call to write().
- Parameters:
install_args (dict) – keyword install arguments
- property lock_path
Path to spack.lock file in this environment.
- property manifest_path
Path to spack.yaml file in this environment.
- matching_spec(spec)[source]
Given a spec (likely not concretized), find a matching concretized spec in the environment.
The matching spec does not have to be installed in the environment, but must be concrete (specs added with spack add without an intervening spack concretize will not be matched).
If there is a single root spec that matches the provided spec or a single dependency spec that matches the provided spec, then the concretized instance of that spec will be returned.
If multiple root specs match the provided spec, or no root specs match and multiple dependency specs match, then this raises an error and reports all matching specs.
- remove(query_spec, list_name='specs', force=False)[source]
Remove specs from an environment that match a query_spec
- removed_specs()[source]
Tuples of (user spec, concrete spec) for all specs that will be removed on next concretize.
- property repo
- property repos_path
- rm_view_from_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications [source]
Collect the environment modifications to deactivate an environment using the provided view. Reverses the action of
add_view_to_env
.- Parameters:
env_mod – the environment modifications object that is modified.
view – the name of the view to deactivate.
- roots()[source]
Specs explicitly requested by the user in this environment.
Yields both added and installed specs that have user specs in spack.yaml.
- property scope_name
Name of the config scope of this environment’s manifest file.
- set_included_concretized_user_specs(env_name: str, env_info: Dict[str, Dict[str, Any]], included_json_specs_by_hash: Dict[str, Dict[str, Any]]) Dict[str, Dict[str, Any]] [source]
Sets all of the concretized user specs from included environments to include those from nested included environments.
- Parameters:
env_name – the name (technically the path) of the included environment
env_info – included concrete environment data
included_json_specs_by_hash – concrete spec data keyed by hash
Returns: updated specs_by_hash
- property unify
- uninstalled_specs()[source]
Return root specs that are not installed, or are installed, but are development specs themselves or have those among their dependencies.
- update_default_view(path_or_bool: str | bool) None [source]
Updates the path of the default view.
If the argument passed as input is False the default view is deleted, if present. The manifest will have an entry “view: false”.
If the argument passed as input is True a default view is created, if not already present. The manifest will have an entry “view: true”. If a default view is already declared, it will be left untouched.
If the argument passed as input is a path a default view pointing to that path is created, if not present already. If a default view is already declared, only its “root” will be changed.
- Parameters:
path_or_bool – either True, or False or a path
- update_environment_repository() None [source]
Updates the repository associated with the environment.
- property user_specs
- property view_path_default
- write(regenerate: bool = True) None [source]
Writes an in-memory environment to its location on disk.
Write out package files for each newly concretized spec. Also regenerate any views associated with the environment and run post-write hooks, if regenerate is True.
- Parameters:
regenerate – regenerate views and run post-write hooks as well as writing if True.
- exception spack.environment.SpackEnvironmentConfigError(message, long_message=None)[source]
Bases:
SpackEnvironmentError
Class for Spack environment-specific configuration errors.
- exception spack.environment.SpackEnvironmentError(message, long_message=None)[source]
Bases:
SpackError
Superclass for all errors to do with Spack environments.
- exception spack.environment.SpackEnvironmentViewError(message, long_message=None)[source]
Bases:
SpackEnvironmentError
Class for errors regarding view generation.
- spack.environment.activate(env, use_env_repo=False)[source]
Activate an environment.
To activate an environment, we add its manifest’s configuration scope to the existing Spack configuration, and we set active to the current environment.
- Parameters:
env (Environment) – the environment to activate
use_env_repo (bool) – use the packages exactly as they appear in the environment’s repository
- spack.environment.active_environment() Environment | None [source]
Returns the active environment when there is any
- spack.environment.all_environment_names()[source]
List the names of environments that currently exist.
- spack.environment.as_env_dir(name_or_dir)[source]
Translate an environment name or directory to the environment directory
- spack.environment.create(name: str, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment [source]
Create a managed environment in Spack and returns it.
A managed environment is created in a root directory managed by this Spack instance, so that Spack can keep track of them.
Files with suffix
.json
or.lock
are considered lockfiles. Files with any other name are considered manifest files.- Parameters:
name – name of the managed environment
init_file – either a lockfile, a manifest file, or None
with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view
keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute
include_concrete – list of concrete environment names/paths to be included
- spack.environment.create_in_dir(root: str | Path, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment [source]
Create an environment in the directory passed as input and returns it.
Files with suffix
.json
or.lock
are considered lockfiles. Files with any other name are considered manifest files.- Parameters:
root – directory where to create the environment.
init_file – either a lockfile, a manifest file, or None
with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view
keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute
include_concrete – concrete environment names/paths to be included
- spack.environment.deactivate()[source]
Undo any configuration or repo settings modified by
activate()
.
- spack.environment.default_manifest_yaml()[source]
default spack.yaml file to put in new environments
- spack.environment.display_specs(specs)[source]
Displays a list of specs traversed breadth-first, covering nodes, with install status.
- Parameters:
specs (list) – list of specs
- spack.environment.environment_dir_from_name(name: str, exists_ok: bool = True) str [source]
Returns the directory associated with a named environment.
- Parameters:
name – name of the environment
exists_ok – if False, raise an error if the environment exists already
- Raises:
SpackEnvironmentError – if exists_ok is False and the environment exists already
- spack.environment.environment_from_name_or_dir(name_or_dir)[source]
Get an environment with the supplied name.
- spack.environment.environment_path_scopes(name: str, path: str) List[ConfigScope] | None [source]
Retrieve the suitably named environment path scopes
- Parameters:
name – configuration scope name
path – path to configuration file(s)
Returns: list of environment scopes, if any, or None
- spack.environment.initialize_environment_dir(environment_dir: str | Path, envfile: str | Path | None) None [source]
Initialize an environment directory starting from an envfile.
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.
- Parameters:
environment_dir – directory where the environment should be placed
envfile – manifest file or lockfile used to initialize the environment
- Raises:
SpackEnvironmentError – if the directory can’t be initialized
- spack.environment.installed_specs()[source]
Returns the specs of packages installed in the active environment or None if no packages are installed.
- spack.environment.is_latest_format(manifest)[source]
Return False if the manifest file exists and is not in the latest schema format.
- Parameters:
manifest (str) – manifest file to be analyzed
- spack.environment.manifest_file(env_name_or_dir)[source]
Return the absolute path to a manifest file given the environment name or directory.
- Parameters:
env_name_or_dir (str) – either the name of a valid environment or a directory where a manifest file resides
- Raises:
AssertionError – if the environment is not found
- spack.environment.no_active_environment()[source]
Deactivate the active environment for the duration of the context. Has no effect when there is no active environment.
- spack.environment.update_yaml(manifest, backup_file)[source]
Update a manifest file from an old format to the current one.
- Parameters:
- Returns:
True if the manifest was updated, False otherwise.
- Raises:
AssertionError – in case anything goes wrong during the update
Submodules
spack.environment.depfile module
This module contains the traversal logic and models that can be used to generate depfiles from an environment.
- class spack.environment.depfile.DepfileNode(target: Spec, prereqs: List[Spec], buildcache: UseBuildCache)[source]
Bases:
object
Contains a spec, a subset of its dependencies, and a flag whether it should be buildcache only/never/auto.
- class spack.environment.depfile.DepfileSpecVisitor(pkg_buildcache: UseBuildCache, deps_buildcache: UseBuildCache)[source]
Bases:
object
This visitor produces an adjacency list of a (reduced) DAG, which is used to generate depfile targets with their prerequisites. Currently it only drops build deps when using buildcache only mode.
Note that the DAG could be reduced even more by dropping build edges of specs installed at the moment the depfile is generated, but that would produce stateful depfiles that would not fail when the database is wiped later.
- class spack.environment.depfile.MakefileModel(env: Environment, roots: List[Spec], adjacency_list: List[DepfileNode], make_prefix: str | None, jobserver: bool)[source]
Bases:
object
This class produces all data to render a makefile for specs of an environment.
- property empty
- static from_env(env: Environment, *, filter_specs: List[Spec] | None = None, pkg_buildcache: UseBuildCache = UseBuildCache.AUTO, dep_buildcache: UseBuildCache = UseBuildCache.AUTO, make_prefix: str | None = None, jobserver: bool = True) MakefileModel [source]
Produces a MakefileModel from an environment and a list of specs.
- Parameters:
env – the environment to use
filter_specs – if provided, only these specs will be built from the environment, otherwise the environment roots are used.
pkg_buildcache – whether to only use the buildcache for top-level specs.
dep_buildcache – whether to only use the buildcache for non-top-level specs.
make_prefix – the prefix for the makefile targets
jobserver – when enabled, make will invoke Spack with jobserver support. For dry-run this should be disabled.
spack.environment.environment module
- class spack.environment.environment.Environment(manifest_dir: str | Path)[source]
Bases:
object
A Spack environment, which bundles together configuration and a list of specs.
- property active
True if this environment is currently active.
- add(user_spec, list_name='specs')[source]
Add a single user_spec (non-concretized) to the Environment
- Returns:
- True if the spec was added, False if it was already
present and did not need to be added
- Return type:
(bool)
- add_view_to_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications [source]
Collect the environment modifications to activate an environment using the provided view. Removes duplicate paths.
- Parameters:
env_mod – the environment modifications object that is modified.
view – the name of the view to activate.
- added_specs()[source]
Specs that are not yet installed.
Yields the user spec for non-concretized specs, and the concrete spec for already concretized but not yet installed specs.
- all_concretized_orders() List[str] [source]
Returns all of the concretized order of the environment and its included environment(s).
- all_concretized_user_specs() List[Spec] [source]
Returns all of the concretized user specs of the environment and its included environment(s).
- all_matching_specs(*specs: Spec) List[Spec] [source]
Returns all concretized specs in the environment satisfying any of the input specs
- change_existing_spec(change_spec: Spec, list_name: str = 'specs', match_spec: Spec | None = None, allow_changing_multiple_specs=False)[source]
Find the spec identified by match_spec and change it to change_spec.
- Parameters:
change_spec – defines the spec properties that need to be changed. This will not change attributes of the matched spec unless they conflict with change_spec.
list_name – identifies the spec list in the environment that should be modified
match_spec – if set, this identifies the spec that should be changed. If not set, it is assumed we are looking for a spec with the same name as change_spec.
- clear(re_read=False)[source]
Clear the contents of the environment
- Parameters:
re_read – If
True
, do not clearnew_specs
. This value cannot be read from yaml, and needs to be maintained when re-reading an existing environment.
- concrete_roots()[source]
Same as concretized_specs, except it returns the list of concrete roots without associated user spec
- concretize(force=False, tests=False)[source]
Concretize user_specs in this environment.
Only concretizes specs that haven’t been concretized yet unless force is
True
.This only modifies the environment in memory.
write()
will write out a lockfile containing concretized specs.- Parameters:
- Returns:
List of specs that have been concretized. Each entry is a tuple of the user spec and the corresponding concretized spec.
- deconcretize(spec: Spec, concrete: bool = True)[source]
Remove specified spec from environment concretization
- Parameters:
spec – Spec to deconcretize. This must be a root of the environment
concrete – If True, find all instances of spec as concrete in the environemnt. If False, find a single instance of the abstract spec as root of the environment.
- property default_view
- property dev_specs
- ensure_env_directory_exists(dot_env: bool = False) None [source]
Ensure that the root directory of the environment exists
- Parameters:
dot_env – if True also ensures that the <root>/.env directory exists
- get_one_by_hash(dag_hash)[source]
Returns the single spec from the environment which matches the provided hash. Raises an AssertionError if no specs match or if more than one spec matches.
- included_concrete_spec_data: Dict[str, Dict[str, List[str]]]
First-level included concretized spec data from/to the lockfile.
- included_concretized_order: Dict[str, List[str]]
Roots from included environments with the last concretization, in order
- included_concretized_user_specs: Dict[str, List[Spec]]
User specs from included environments from the last concretization
- included_specs_by_hash: Dict[str, Dict[str, Spec]]
Concretized specs by hash from the included environments
- property included_user_specs: SpecList
Included concrete user (or root) specs from last concretization.
- install_all(**install_args)[source]
Install all concretized specs in an environment.
Note: this does not regenerate the views for the environment; that needs to be done separately with a call to write().
- Parameters:
install_args (dict) – keyword install arguments
- property lock_path
Path to spack.lock file in this environment.
- property manifest_path
Path to spack.yaml file in this environment.
- matching_spec(spec)[source]
Given a spec (likely not concretized), find a matching concretized spec in the environment.
The matching spec does not have to be installed in the environment, but must be concrete (specs added with spack add without an intervening spack concretize will not be matched).
If there is a single root spec that matches the provided spec or a single dependency spec that matches the provided spec, then the concretized instance of that spec will be returned.
If multiple root specs match the provided spec, or no root specs match and multiple dependency specs match, then this raises an error and reports all matching specs.
- remove(query_spec, list_name='specs', force=False)[source]
Remove specs from an environment that match a query_spec
- removed_specs()[source]
Tuples of (user spec, concrete spec) for all specs that will be removed on next concretize.
- property repo
- property repos_path
- rm_view_from_env(env_mod: EnvironmentModifications, view: str) EnvironmentModifications [source]
Collect the environment modifications to deactivate an environment using the provided view. Reverses the action of
add_view_to_env
.- Parameters:
env_mod – the environment modifications object that is modified.
view – the name of the view to deactivate.
- roots()[source]
Specs explicitly requested by the user in this environment.
Yields both added and installed specs that have user specs in spack.yaml.
- property scope_name
Name of the config scope of this environment’s manifest file.
- set_included_concretized_user_specs(env_name: str, env_info: Dict[str, Dict[str, Any]], included_json_specs_by_hash: Dict[str, Dict[str, Any]]) Dict[str, Dict[str, Any]] [source]
Sets all of the concretized user specs from included environments to include those from nested included environments.
- Parameters:
env_name – the name (technically the path) of the included environment
env_info – included concrete environment data
included_json_specs_by_hash – concrete spec data keyed by hash
Returns: updated specs_by_hash
- property unify
- uninstalled_specs()[source]
Return root specs that are not installed, or are installed, but are development specs themselves or have those among their dependencies.
- update_default_view(path_or_bool: str | bool) None [source]
Updates the path of the default view.
If the argument passed as input is False the default view is deleted, if present. The manifest will have an entry “view: false”.
If the argument passed as input is True a default view is created, if not already present. The manifest will have an entry “view: true”. If a default view is already declared, it will be left untouched.
If the argument passed as input is a path a default view pointing to that path is created, if not present already. If a default view is already declared, only its “root” will be changed.
- Parameters:
path_or_bool – either True, or False or a path
- update_environment_repository() None [source]
Updates the repository associated with the environment.
- property user_specs
- property view_path_default
- views: Dict[str, ViewDescriptor]
- write(regenerate: bool = True) None [source]
Writes an in-memory environment to its location on disk.
Write out package files for each newly concretized spec. Also regenerate any views associated with the environment and run post-write hooks, if regenerate is True.
- Parameters:
regenerate – regenerate views and run post-write hooks as well as writing if True.
- class spack.environment.environment.EnvironmentManifestFile(manifest_dir: Path | str, name: str | None = None)[source]
Bases:
Mapping
Manages the in-memory representation of a manifest file, and its synchronization with the actual manifest on disk.
- add_definition(user_spec: str, list_name: str) None [source]
Appends a user spec to the first active definition matching the name passed as argument.
- Parameters:
user_spec – user spec to be appended
list_name – name of the definition where to append
- Raises:
SpackEnvironmentError – is no valid definition exists already
- add_user_spec(user_spec: str) None [source]
Appends the user spec passed as input to the list of root specs.
- Parameters:
user_spec – user spec to be appended
- property configuration
Return the dictionaries in the pristine YAML, without the top level attribute
- deactivate_config_scope() None [source]
Remove any of the manifest’s scopes from the global config path.
- property env_config_scopes: List[ConfigScope]
A list of all configuration scopes for the environment manifest. On the first call this instantiates all the scopes, on subsequent calls it returns the cached list.
- static from_lockfile(manifest_dir: Path | str) EnvironmentManifestFile [source]
Returns an environment manifest file compatible with the lockfile already present in the environment directory.
This function also writes a spack.yaml file that is consistent with the spack.lock already existing in the directory.
- Parameters:
manifest_dir – directory containing the manifest and lockfile
- property included_config_scopes: List[ConfigScope]
List of included configuration scopes from the manifest.
Scopes are listed in the YAML file in order from highest to lowest precedence, so configuration from earlier scope will take precedence over later ones.
This routine returns them in the order they should be pushed onto the internal scope stack (so, in reverse, from lowest to highest).
Returns: Configuration scopes associated with the environment manifest
- Raises:
SpackEnvironmentError – if the manifest includes a remote file but no configuration stage directory has been identified
- override_definition(user_spec: str, *, override: str, list_name: str) None [source]
Overrides a user spec from an active definition that matches the name passed as argument.
- Parameters:
user_spec – user spec to be overridden
override – new spec to be used
list_name – name of the definition where to override the spec
- Raises:
SpackEnvironmentError – if the user spec cannot be overridden
- override_user_spec(user_spec: str, idx: int) None [source]
Overrides the user spec at index idx with the one passed as input.
- Parameters:
user_spec – new user spec
idx – index of the spec to be overridden
- Raises:
SpackEnvironmentError – when the user spec cannot be overridden
- prepare_config_scope() None [source]
Add the manifest’s scopes to the global configuration search path.
- remove_definition(user_spec: str, list_name: str) None [source]
Removes a user spec from an active definition that matches the name passed as argument.
- Parameters:
user_spec – user spec to be removed
list_name – name of the definition where to remove the spec from
- Raises:
SpackEnvironmentError – if the user spec cannot be removed from the list, or the list does not exist
- remove_user_spec(user_spec: str) None [source]
Removes the user spec passed as input from the list of root specs
- Parameters:
user_spec – user spec to be removed
- Raises:
SpackEnvironmentError – when the user spec is not in the list
- set_default_view(view: bool | str | Path | Dict[str, str]) None [source]
Sets the default view root in the manifest to the value passed as input.
- Parameters:
view – If the value is a string or a path, it specifies the path to the view. If True the default view is used for the environment, if False there’s no view.
- exception spack.environment.environment.SpackEnvironmentConfigError(message, long_message=None)[source]
Bases:
SpackEnvironmentError
Class for Spack environment-specific configuration errors.
- exception spack.environment.environment.SpackEnvironmentError(message, long_message=None)[source]
Bases:
SpackError
Superclass for all errors to do with Spack environments.
- exception spack.environment.environment.SpackEnvironmentViewError(message, long_message=None)[source]
Bases:
SpackEnvironmentError
Class for errors regarding view generation.
- class spack.environment.environment.ViewDescriptor(base_path, root, projections={}, select=[], exclude=[], link='all', link_type='symlink')[source]
Bases:
object
- get_projection_for_spec(spec)[source]
Get projection for spec. This function does not require the view to exist on the filesystem.
- specs_for_view(concrete_roots: List[Spec]) List[Spec] [source]
Flatten the DAGs of the concrete roots, keep only unique, selected, and installed specs in topological order from root to leaf.
- view(new: str | None = None) SimpleFilesystemView [source]
Returns a view object for the underlying view directory. This means that the self.root symlink is followed, and that the view has to exist on the filesystem (unless
new
). This function is useful when writing to the view.Raise if new is None and there is no current view
- Parameters:
new – If a string, create a FilesystemView rooted at that path. Default None. This should only be used to regenerate the view, and cannot be used to access specs.
- spack.environment.environment.activate(env, use_env_repo=False)[source]
Activate an environment.
To activate an environment, we add its manifest’s configuration scope to the existing Spack configuration, and we set active to the current environment.
- Parameters:
env (Environment) – the environment to activate
use_env_repo (bool) – use the packages exactly as they appear in the environment’s repository
- spack.environment.environment.active_environment() Environment | None [source]
Returns the active environment when there is any
- spack.environment.environment.all_environment_names()[source]
List the names of environments that currently exist.
- spack.environment.environment.as_env_dir(name_or_dir)[source]
Translate an environment name or directory to the environment directory
- spack.environment.environment.create(name: str, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment [source]
Create a managed environment in Spack and returns it.
A managed environment is created in a root directory managed by this Spack instance, so that Spack can keep track of them.
Files with suffix
.json
or.lock
are considered lockfiles. Files with any other name are considered manifest files.- Parameters:
name – name of the managed environment
init_file – either a lockfile, a manifest file, or None
with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view
keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute
include_concrete – list of concrete environment names/paths to be included
- spack.environment.environment.create_in_dir(root: str | Path, init_file: str | Path | None = None, with_view: str | Path | bool | None = None, keep_relative: bool = False, include_concrete: List[str] | None = None) Environment [source]
Create an environment in the directory passed as input and returns it.
Files with suffix
.json
or.lock
are considered lockfiles. Files with any other name are considered manifest files.- Parameters:
root – directory where to create the environment.
init_file – either a lockfile, a manifest file, or None
with_view – whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view
keep_relative – if True, develop paths are copied verbatim into the new environment file, otherwise they are made absolute
include_concrete – concrete environment names/paths to be included
- spack.environment.environment.deactivate()[source]
Undo any configuration or repo settings modified by
activate()
.
- spack.environment.environment.default_env_path = '/home/docs/checkouts/readthedocs.org/user_builds/spack/checkouts/latest/lib/spack/docs/_spack_root/var/spack/environments'
default path where environments are stored in the spack tree
- spack.environment.environment.default_manifest_yaml()[source]
default spack.yaml file to put in new environments
- spack.environment.environment.display_specs(specs)[source]
Displays a list of specs traversed breadth-first, covering nodes, with install status.
- Parameters:
specs (list) – list of specs
- spack.environment.environment.ensure_no_disallowed_env_config_mods(scopes: List[ConfigScope]) None [source]
- spack.environment.environment.env_root_path() str [source]
Override default root path if the user specified it
- spack.environment.environment.env_subdir_name = '.spack-env'
Name of the directory where environments store repos, logs, views, configs
- spack.environment.environment.env_subdir_path(manifest_dir: str | Path) str [source]
Path to where the environment stores repos, logs, views, configs.
- Parameters:
manifest_dir – directory containing the environment manifest file
Returns: directory the environment uses to manage its files
- spack.environment.environment.environment_dir_from_name(name: str, exists_ok: bool = True) str [source]
Returns the directory associated with a named environment.
- Parameters:
name – name of the environment
exists_ok – if False, raise an error if the environment exists already
- Raises:
SpackEnvironmentError – if exists_ok is False and the environment exists already
- spack.environment.environment.environment_from_name_or_dir(name_or_dir)[source]
Get an environment with the supplied name.
- spack.environment.environment.environment_name(path: str | Path) str [source]
Human-readable representation of the environment.
This is the path for directory environments, and just the name for managed environments.
- spack.environment.environment.environment_path_scopes(name: str, path: str) List[ConfigScope] | None [source]
Retrieve the suitably named environment path scopes
- Parameters:
name – configuration scope name
path – path to configuration file(s)
Returns: list of environment scopes, if any, or None
- spack.environment.environment.exists(name)[source]
Whether an environment with this name exists or not.
- spack.environment.environment.initialize_environment_dir(environment_dir: str | Path, envfile: str | Path | None) None [source]
Initialize an environment directory starting from an envfile.
Files with suffix .json or .lock are considered lockfiles. Files with any other name are considered manifest files.
- Parameters:
environment_dir – directory where the environment should be placed
envfile – manifest file or lockfile used to initialize the environment
- Raises:
SpackEnvironmentError – if the directory can’t be initialized
- spack.environment.environment.installed_specs()[source]
Returns the specs of packages installed in the active environment or None if no packages are installed.
- spack.environment.environment.is_env_dir(path)[source]
Whether a directory contains a spack environment.
- spack.environment.environment.is_latest_format(manifest)[source]
Return False if the manifest file exists and is not in the latest schema format.
- Parameters:
manifest (str) – manifest file to be analyzed
- spack.environment.environment.lockfile_format_version = 5
version of the lockfile format. Must increase monotonically.
- spack.environment.environment.lockfile_name = 'spack.lock'
Name of the input yaml file for an environment
- spack.environment.environment.make_repo_path(root)[source]
Make a RepoPath from the repo subdirectories in an environment.
- spack.environment.environment.manifest_file(env_name_or_dir)[source]
Return the absolute path to a manifest file given the environment name or directory.
- Parameters:
env_name_or_dir (str) – either the name of a valid environment or a directory where a manifest file resides
- Raises:
AssertionError – if the environment is not found
- spack.environment.environment.manifest_name = 'spack.yaml'
Name of the input yaml file for an environment
- spack.environment.environment.no_active_environment()[source]
Deactivate the active environment for the duration of the context. Has no effect when there is no active environment.
- spack.environment.environment.root(name)[source]
Get the root directory for an environment by name.
- spack.environment.environment.set_included_envs_to_env_paths(include_concrete: List[str]) None [source]
If the included environment(s) is the environment name it is replaced by the path to the environment
- Parameters:
include_concrete – list of env name or path to env
- spack.environment.environment.spack_env_var = 'SPACK_ENV'
environment variable used to indicate the active environment
- spack.environment.environment.spack_env_view_var = 'SPACK_ENV_VIEW'
environment variable used to indicate the active environment view
- spack.environment.environment.update_yaml(manifest, backup_file)[source]
Update a manifest file from an old format to the current one.
- Parameters:
- Returns:
True if the manifest was updated, False otherwise.
- Raises:
AssertionError – in case anything goes wrong during the update
- spack.environment.environment.valid_environment_name_re = '^\\w[\\w-]*$'
regex for validating enviroment names
- spack.environment.environment.validate_included_envs_concrete(include_concrete: List[str]) None [source]
Checks that all of the included environments are concrete
- Parameters:
include_concrete – list of already existing concrete environments to include
- Raises:
SpackEnvironmentError – if any of the included environments are not concrete
- spack.environment.environment.validate_included_envs_exists(include_concrete: List[str]) None [source]
Checks that all of the included environments exist
- Parameters:
include_concrete – list of already existing concrete environments to include
- Raises:
SpackEnvironmentError – if any of the included environments do not exist
spack.environment.shell module
- spack.environment.shell.activate(env: Environment, use_env_repo=False, view: str | None = 'default') EnvironmentModifications [source]
Activate an environment and append environment modifications
To activate an environment, we add its configuration scope to the existing Spack configuration, and we set active to the current environment.
- Parameters:
env – the environment to activate
use_env_repo – use the packages exactly as they appear in the environment’s repository
view – generate commands to add runtime environment variables for named view
- Returns:
Environment variables modifications to activate environment.
- Return type:
- spack.environment.shell.deactivate() EnvironmentModifications [source]
Deactivate an environment and collect corresponding environment modifications.
- Note: unloads the environment in its current state, not in the state it was
loaded in, meaning that specs that were removed from the spack environment after activation are not unloaded.
- Returns:
Environment variables modifications to activate environment.