spack.util package

Submodules

spack.util.classes module

spack.util.classes.list_classes(parent_module, mod_path)[source]

Given a parent path (e.g., spack.platforms or spack.analyzers), use list_modules to derive the module names, and then mod_to_class to derive class names. Import the classes and return them in a list

spack.util.compression module

spack.util.compression.allowed_archive(path)[source]
spack.util.compression.bz2_support()[source]
spack.util.compression.decompressor_for(path, ext)[source]

Returns a function pointer to appropriate decompression algorithm based on extension type.

Parameters
  • path (str) – path of the archive file requiring decompression

  • ext (str) – Extension of archive file

spack.util.compression.extension(path)[source]

Get the archive extension for a path.

spack.util.compression.gzip_support()[source]
spack.util.compression.lzma_support()[source]
spack.util.compression.strip_extension(path)[source]

Get the part of a path that does not include its compressed type extension.

spack.util.compression.tar_support()[source]

spack.util.cpus module

spack.util.cpus.cpus_available()[source]

Returns the number of CPUs available for the current process, or the number of phyiscal CPUs when that information cannot be retrieved. The number of available CPUs might differ from the number of physical CPUs when using spack through Slurm or container runtimes.

spack.util.crypto module

class spack.util.crypto.Checker(hexdigest, **kwargs)[source]

Bases: object

A checker checks files against one particular hex digest. It will automatically determine what hashing algorithm to used based on the length of the digest it’s initialized with. e.g., if the digest is 32 hex characters long this will use md5.

Example: know your tarball should hash to ‘abc123’. You want to check files against this. You would use this class like so:

hexdigest = 'abc123'
checker = Checker(hexdigest)
success = checker.check('downloaded.tar.gz')

After the call to check, the actual checksum is available in checker.sum, in case it’s needed for error output.

You can trade read performance and memory usage by adjusting the block_size optional arg. By default it’s a 1MB (2**20 bytes) buffer.

check(filename)[source]

Read the file with the specified name and check its checksum against self.hexdigest. Return True if they match, False otherwise. Actual checksum is stored in self.sum.

property hash_name

Get the name of the hash function this Checker is using.

class spack.util.crypto.DeprecatedHash(hash_alg, alert_fn, disable_security_check)[source]

Bases: object

spack.util.crypto.bit_length(num)[source]

Number of bits required to represent an integer in binary.

spack.util.crypto.checksum(hashlib_algo, filename, **kwargs)[source]

Returns a hex digest of the filename generated using an algorithm from hashlib.

spack.util.crypto.hash_algo_for_digest(hexdigest)[source]

Gets name of the hash algorithm for a hex digest.

spack.util.crypto.hash_fun_for_algo(algo)[source]

Get a function that can perform the specified hash algorithm.

spack.util.crypto.hash_fun_for_digest(hexdigest)[source]

Gets a hash function corresponding to a hex digest.

spack.util.crypto.hashes = {'md5': 16, 'sha1': 20, 'sha224': 28, 'sha256': 32, 'sha384': 48, 'sha512': 64}

Set of hash algorithms that Spack can use, mapped to digest size in bytes

spack.util.crypto.prefix_bits(byte_array, bits)[source]

Return the first <bits> bits of a byte array as an integer.

spack.util.debug module

Debug signal handler: prints a stack trace and enters interpreter.

register_interrupt_handler() enables a ctrl-C handler that prints a stack trace and drops the user into an interpreter.

class spack.util.debug.ForkablePdb(stdout_fd=None, stderr_fd=None)[source]

Bases: Pdb

This class allows the python debugger to follow forked processes and can set tracepoints allowing the Python Debugger Pdb to be used from a python multiprocessing child process.

This is used the same way one would normally use Pdb, simply import this class and use as a drop in for Pdb, although the syntax here is slightly different, requiring the instantiton of this class, i.e. ForkablePdb().set_trace().

This should be used when attempting to call a debugger from a child process spawned by the python multiprocessing such as during the run of Spack.install, or any where else Spack spawns a child process.

spack.util.debug.debug_handler(sig, frame)[source]

Interrupt running process, and provide a python prompt for interactive debugging.

spack.util.debug.register_interrupt_handler()[source]

Print traceback and enter an interpreter on Ctrl-C

spack.util.editor module

Module for finding the user’s preferred text editor.

Defines one function, editor(), which invokes the editor defined by the user’s VISUAL environment variable if set. We fall back to the editor defined by the EDITOR environment variable if VISUAL is not set or the specified editor fails (e.g. no DISPLAY for a graphical editor). If neither variable is set, we fall back to one of several common editors, raising an EnvironmentError if we are unable to find one.

spack.util.editor.editor(*args, **kwargs)[source]

Invoke the user’s editor.

This will try to execute the following, in order:

  1. $VISUAL <args> # the “visual” editor (per POSIX)

  2. $EDITOR <args> # the regular editor (per POSIX)

  3. some default editor (see _default_editors) with <args>

If an environment variable isn’t defined, it is skipped. If it points to something that can’t be executed, we’ll print a warning. And if we can’t find anything that can be executed after searching the full list above, we’ll raise an error.

Parameters

args (list) – args to pass to editor

Optional Arguments:

_exec_func (function): invoke this function instead of os.execv()

spack.util.environment module

Utilities for setting and modifying environment variables.

class spack.util.environment.AppendFlagsEnv(name, value, **kwargs)[source]

Bases: NameValueModifier

execute(env)[source]
class spack.util.environment.AppendPath(name, value, **kwargs)[source]

Bases: NameValueModifier

execute(env)[source]
class spack.util.environment.DeprioritizeSystemPaths(name, **kwargs)[source]

Bases: NameModifier

execute(env)[source]
class spack.util.environment.EnvironmentModifications(other=None, traced=None)[source]

Bases: object

Keeps track of requests to modify the current environment.

Each call to a method to modify the environment stores the extra information on the caller in the request:

  • ‘filename’ : filename of the module where the caller is defined

  • ‘lineno’: line number where the request occurred

  • ‘context’ : line of code that issued the request that failed

append_flags(name, value, sep=' ', **kwargs)[source]

Stores in the current object a request to append to an env variable

Parameters
  • name – name of the environment variable to be appended to

  • value – value to append to the environment variable

Appends with spaces separating different additions to the variable

append_path(name, path, **kwargs)[source]

Stores a request to append a path to a path list.

Parameters
  • name – name of the path list in the environment

  • path – path to be appended

apply_modifications(env=None)[source]

Applies the modifications and clears the list.

clear()[source]

Clears the current list of modifications

deprioritize_system_paths(name, **kwargs)[source]

Stores a request to deprioritize system paths in a path list, otherwise preserving the order.

Parameters

name – name of the path list in the environment.

extend(other)[source]
static from_environment_diff(before, after, clean=False)[source]

Constructs an instance of a spack.util.environment.EnvironmentModifications object from the diff of two dictionaries.

Parameters
  • before (dict) – environment before the modifications are applied

  • after (dict) – environment after the modifications are applied

  • clean (bool) – in addition to removing empty entries, also remove duplicate entries

static from_sourcing_file(filename, *arguments, **kwargs)[source]

Constructs an instance of a spack.util.environment.EnvironmentModifications object that has the same effect as sourcing a file.

Parameters
  • filename (str) – the file to be sourced

  • *arguments (list) – arguments to pass on the command line

Keyword Arguments
  • shell (str) – the shell to use (default: bash)

  • shell_options (str) – options passed to the shell (default: -c)

  • source_command (str) – the command to run (default: source)

  • suppress_output (str) – redirect used to suppress output of command (default: &> /dev/null)

  • concatenate_on_success (str) – operator used to execute a command only when the previous command succeeds (default: &&)

  • exclude ([str or re]) – ignore any modifications of these variables (default: [])

  • include ([str or re]) – always respect modifications of these variables (default: []). Supersedes any excluded variables.

  • clean (bool) – in addition to removing empty entries, also remove duplicate entries (default: False).

group_by_name()[source]

Returns a dict of the modifications grouped by variable name.

Returns

dict mapping the environment variable name to the modifications to be done on it

is_unset(var_name)[source]
prepend_path(name, path, **kwargs)[source]

Same as append_path, but the path is pre-pended.

Parameters
  • name – name of the path list in the environment

  • path – path to be pre-pended

prune_duplicate_paths(name, **kwargs)[source]

Stores a request to remove duplicates from a path list, otherwise preserving the order.

Parameters

name – name of the path list in the environment.

remove_flags(name, value, sep=' ', **kwargs)[source]

Stores in the current object a request to remove flags from an env variable

Parameters
  • name – name of the environment variable to be removed from

  • value – value to remove to the environment variable

  • sep – separator to assume for environment variable

remove_path(name, path, **kwargs)[source]

Stores a request to remove a path from a path list.

Parameters
  • name – name of the path list in the environment

  • path – path to be removed

reversed()[source]

Returns the EnvironmentModifications object that will reverse self

Only creates reversals for additions to the environment, as reversing unset and remove_path modifications is impossible.

Reversable operations are set(), prepend_path(), append_path(), set_path(), and append_flags().

set(name, value, **kwargs)[source]

Stores a request to set an environment variable.

Parameters
  • name – name of the environment variable to be set

  • value – value of the environment variable

set_path(name, elements, **kwargs)[source]

Stores a request to set a path generated from a list.

Parameters
  • name – name o the environment variable to be set.

  • elements – elements of the path to set.

shell_modifications(shell='sh', explicit=False, env=None)[source]

Return shell code to apply the modifications and clears the list.

unset(name, **kwargs)[source]

Stores a request to unset an environment variable.

Parameters

name – name of the environment variable to be unset

class spack.util.environment.NameModifier(name, **kwargs)[source]

Bases: object

update_args(**kwargs)[source]
class spack.util.environment.NameValueModifier(name, value, **kwargs)[source]

Bases: object

update_args(**kwargs)[source]
class spack.util.environment.PrependPath(name, value, **kwargs)[source]

Bases: NameValueModifier

execute(env)[source]
class spack.util.environment.PruneDuplicatePaths(name, **kwargs)[source]

Bases: NameModifier

execute(env)[source]
class spack.util.environment.RemoveFlagsEnv(name, value, **kwargs)[source]

Bases: NameValueModifier

execute(env)[source]
class spack.util.environment.RemovePath(name, value, **kwargs)[source]

Bases: NameValueModifier

execute(env)[source]
class spack.util.environment.SetEnv(name, value, **kwargs)[source]

Bases: NameValueModifier

execute(env)[source]
class spack.util.environment.SetPath(name, value, **kwargs)[source]

Bases: NameValueModifier

execute(env)[source]
class spack.util.environment.UnsetEnv(name, **kwargs)[source]

Bases: NameModifier

execute(env)[source]
spack.util.environment.concatenate_paths(paths, separator=':')[source]

Concatenates an iterable of paths into a string of paths separated by separator, defaulting to colon.

Parameters
  • paths – iterable of paths

  • separator – the separator to use, default ‘;’ windows, ‘:’ otherwise

Returns

string

spack.util.environment.deprioritize_system_paths(paths)[source]

Put system paths at the end of paths, otherwise preserving order.

spack.util.environment.dump_environment(path, environment=None)[source]

Dump an environment dictionary to a source-able file.

spack.util.environment.env_flag(name)[source]
spack.util.environment.env_var_to_source_line(var, val)[source]
spack.util.environment.environment_after_sourcing_files(*files, **kwargs)[source]

Returns a dictionary with the environment that one would have after sourcing the files passed as argument.

Parameters

*files – each item can either be a string containing the path of the file to be sourced or a sequence, where the first element is the file to be sourced and the remaining are arguments to be passed to the command line

Keyword Arguments
  • env (dict) – the initial environment (default: current environment)

  • shell (str) – the shell to use (default: /bin/bash)

  • shell_options (str) – options passed to the shell (default: -c)

  • source_command (str) – the command to run (default: source)

  • suppress_output (str) – redirect used to suppress output of command (default: &> /dev/null)

  • concatenate_on_success (str) – operator used to execute a command only when the previous command succeeds (default: &&)

spack.util.environment.filter_system_paths(paths)[source]

Return only paths that are not system paths.

spack.util.environment.get_host_environment()[source]

Return a dictionary (lookup) with host information (not including the os.environ).

spack.util.environment.get_host_environment_metadata()[source]

Get the host environment, reduce to a subset that we can store in the install directory, and add the spack version.

spack.util.environment.get_path(name)[source]
spack.util.environment.inspect_path(root, inspections, exclude=None)[source]

Inspects root to search for the subdirectories in inspections. Adds every path found to a list of prepend-path commands and returns it.

Parameters
  • root (str) – absolute path where to search for subdirectories

  • inspections (dict) – maps relative paths to a list of environment variables that will be modified if the path exists. The modifications are not performed immediately, but stored in a command object that is returned to client

  • exclude (Callable) – optional callable. If present it must accept an absolute path and return True if it should be excluded from the inspection

Examples:

The following lines execute an inspection in /usr to search for /usr/include and /usr/lib64. If found we want to prepend /usr/include to CPATH and /usr/lib64 to MY_LIB64_PATH.

# Set up the dictionary containing the inspection
inspections = {
    'include': ['CPATH'],
    'lib64': ['MY_LIB64_PATH']
}

# Get back the list of command needed to modify the environment
env = inspect_path('/usr', inspections)

# Eventually execute the commands
env.apply_modifications()
Returns

instance of EnvironmentModifications containing the requested modifications

spack.util.environment.is_system_path(path)[source]

Predicate that given a path returns True if it is a system path, False otherwise.

Parameters

path (str) – path to a directory

Returns

True or False

spack.util.environment.path_put_first(var_name, directories)[source]

Puts the provided directories first in the path, adding them if they’re not already there.

spack.util.environment.path_set(var_name, directories)[source]
spack.util.environment.pickle_environment(path, environment=None)[source]

Pickle an environment dictionary to a file.

spack.util.environment.preserve_environment(*variables)[source]

Ensures that the value of the environment variables passed as arguments is the same before entering to the context manager and after exiting it.

Variables that are unset before entering the context manager will be explicitly unset on exit.

Parameters

variables (list) – list of environment variables to be preserved

spack.util.environment.prune_duplicate_paths(paths)[source]

Returns the paths with duplicates removed, order preserved.

spack.util.environment.sanitize(environment, exclude, include)[source]

Returns a copy of the input dictionary where all the keys that match an excluded pattern and don’t match an included pattern are removed.

Parameters
  • environment (dict) – input dictionary

  • exclude (list) – literals or regex patterns to be excluded

  • include (list) – literals or regex patterns to be included

spack.util.environment.set_env(**kwargs)[source]

Temporarily sets and restores environment variables.

Variables can be set as keyword arguments to this function.

spack.util.environment.set_or_unset_not_first(variable, changes, errstream)[source]

Check if we are going to set or unset something after other modifications have already been requested.

spack.util.environment.validate(env, errstream)[source]

Validates the environment modifications to check for the presence of suspicious patterns. Prompts a warning for everything that was found.

Current checks: - set or unset variables after other changes on the same variable

Parameters

env – list of environment modifications

spack.util.executable module

class spack.util.executable.Executable(name)[source]

Bases: object

Class representing a program that can be run on the command line.

add_default_arg(arg)[source]

Add a default argument to the command.

add_default_env(key, value)[source]

Set an environment variable when the command is run.

Parameters
  • key – The environment variable to set

  • value – The value to set it to

add_default_envmod(envmod)[source]

Set an EnvironmentModifications to use when the command is run.

property command

The command-line string.

Returns

The executable and default arguments

Return type

str

property name

The executable name.

Returns

The basename of the executable

Return type

str

property path

The path to the executable.

Returns

The path to the executable

Return type

str

exception spack.util.executable.ProcessError(message, long_message=None)[source]

Bases: SpackError

ProcessErrors are raised when Executables exit with an error code.

spack.util.executable.which(*args, **kwargs)[source]

Finds an executable in the path like command-line which.

If given multiple executables, returns the first one that is found. If no executables are found, returns None.

Parameters

*args (str) – One or more executables to search for

Keyword Arguments
  • path (list or str) – The path to search. Defaults to PATH

  • required (bool) – If set to True, raise an error if executable not found

Returns

The first executable that is found in the path

Return type

Executable

spack.util.file_cache module

exception spack.util.file_cache.CacheError(message, long_message=None)[source]

Bases: SpackError

class spack.util.file_cache.FileCache(root, timeout=120)[source]

Bases: object

This class manages cached data in the filesystem.

  • Cache files are fetched and stored by unique keys. Keys can be relative paths, so that there can be some hierarchy in the cache.

  • The FileCache handles locking cache files for reading and writing, so client code need not manage locks for cache entries.

cache_path(key)[source]

Path to the file in the cache for a particular key.

destroy()[source]

Remove all files under the cache root.

init_entry(key)[source]

Ensure we can access a cache file. Create a lock for it if needed.

Return whether the cache file exists yet or not.

mtime(key)[source]

Return modification time of cache file, or 0 if it does not exist.

Time is in units returned by os.stat in the mtime field, which is platform-dependent.

read_transaction(key)[source]

Get a read transaction on a file cache item.

Returns a ReadTransaction context manager and opens the cache file for reading. You can use it like this:

with file_cache_object.read_transaction(key) as cache_file:

cache_file.read()

remove(key)[source]
write_transaction(key)[source]

Get a write transaction on a file cache item.

Returns a WriteTransaction context manager that opens a temporary file for writing. Once the context manager finishes, if nothing went wrong, moves the file into place on top of the old file atomically.

spack.util.file_permissions module

exception spack.util.file_permissions.InvalidPermissionsError(message, long_message=None)[source]

Bases: SpackError

Error class for invalid permission setters

spack.util.file_permissions.set_permissions(path, perms, group=None)[source]
spack.util.file_permissions.set_permissions_by_spec(path, spec)[source]

spack.util.gcs module

This file contains the definition of the GCS Blob storage Class used to integrate GCS Blob storage with spack buildcache.

class spack.util.gcs.GCSBlob(url, client=None)[source]

Bases: object

GCS Blob object

Wraps some blob methods for spack functionality

delete_blob()[source]
exists()[source]
get()[source]
get_blob_byte_stream()[source]
get_blob_headers()[source]
upload_to_blob(local_file_path)[source]
class spack.util.gcs.GCSBucket(url, client=None)[source]

Bases: object

GCS Bucket Object Create a wrapper object for a GCS Bucket. Provides methods to wrap spack related tasks, such as destroy.

blob(blob_path)[source]
create()[source]
destroy(recursive=False, **kwargs)[source]

Bucket destruction method

Deletes all blobs within the bucket, and then deletes the bucket itself.

Uses GCS Batch operations to bundle several delete operations together.

exists()[source]
get_all_blobs(recursive=True, relative=True)[source]

Get a list of all blobs Returns a list of all blobs within this bucket.

Parameters

relative

If true (default), print blob paths

relative to ‘build_cache’ directory.

If false, print absolute blob paths (useful for

destruction of bucket)

get_blob(blob_path)[source]
spack.util.gcs.gcs_client()[source]

Create a GCS client Creates an authenticated GCS client to access GCS buckets and blobs

spack.util.gpg module

spack.util.gpg.GNUPGHOME = None

GNUPGHOME environment variable in the context of this Python module

spack.util.gpg.GPG = None

Executable instance for “gpg”, initialized lazily

spack.util.gpg.GPGCONF = None

Executable instance for “gpgconf”, initialized lazily

spack.util.gpg.SOCKET_DIR = None

Socket directory required if a non default home directory is used

exception spack.util.gpg.SpackGPGError(message, long_message=None)[source]

Bases: SpackError

Class raised when GPG errors are detected.

spack.util.gpg.clear()[source]

Reset the global state to uninitialized.

spack.util.gpg.create(**kwargs)[source]

Create a new key pair.

spack.util.gpg.export_keys(location, keys, secret=False)[source]

Export public keys to a location passed as argument.

Parameters
  • location (str) – where to export the keys

  • keys (list) – keys to be exported

  • secret (bool) – whether to export secret keys or not

spack.util.gpg.gnupghome_override(dir)[source]

Set the GNUPGHOME to a new location for this context.

Parameters

dir (str) – new value for GNUPGHOME

spack.util.gpg.init(gnupghome=None, force=False)[source]

Initialize the global objects in the module, if not set.

When calling any gpg executable, the GNUPGHOME environment variable is set to:

  1. The value of the gnupghome argument, if not None

  2. The value of the “SPACK_GNUPGHOME” environment variable, if set

  3. The default gpg path for Spack otherwise

Parameters
  • gnupghome (str) – value to be used for GNUPGHOME when calling GnuPG executables

  • force (bool) – if True forces the re-initialization even if the global objects are set already

spack.util.gpg.list(trusted, signing)[source]

List known keys.

Parameters
  • trusted (bool) – if True list public keys

  • signing (bool) – if True list private keys

spack.util.gpg.public_keys(*args)[source]

Return a list of fingerprints

spack.util.gpg.public_keys_to_fingerprint(*args)[source]

Return the keys that can be used to verify binaries.

spack.util.gpg.sign(key, file, output, clearsign=False)[source]

Sign a file with a key.

Parameters
  • key – key to be used to sign

  • file (str) – file to be signed

  • output (str) – output file (either the clearsigned file or the detached signature)

  • clearsign (bool) – if True wraps the document in an ASCII-armored signature, if False creates a detached signature

spack.util.gpg.signing_keys(*args)[source]

Return the keys that can be used to sign binaries.

spack.util.gpg.trust(keyfile)[source]

Import a public key from a file and trust it.

Parameters

keyfile (str) – file with the public key

spack.util.gpg.untrust(signing, *keys)[source]

Delete known keys.

Parameters
  • signing (bool) – if True deletes the secret keys

  • *keys – keys to be deleted

spack.util.gpg.verify(signature, file=None, suppress_warnings=False)[source]

Verify the signature on a file.

Parameters
  • signature (str) – signature of the file (or clearsigned file)

  • file (str) – file to be verified. If None, then signature is assumed to be a clearsigned file.

  • suppress_warnings (bool) – whether or not to suppress warnings from GnuPG

spack.util.hash module

spack.util.hash.b32_hash(content)[source]

Return the b32 encoded sha1 hash of the input string as a string.

spack.util.hash.base32_prefix_bits(hash_string, bits)[source]

Return the first <bits> bits of a base32 string as an integer.

spack.util.lock module

Wrapper for llnl.util.lock allows locking to be enabled/disabled.

class spack.util.lock.Lock(*args, **kwargs)[source]

Bases: Lock

Lock that can be disabled.

This overrides the _lock() and _unlock() methods from llnl.util.lock so that all the lock API calls will succeed, but the actual locking mechanism can be disabled via _enable_locks.

cleanup(*args)[source]
spack.util.lock.check_lock_safety(path)[source]

Do some extra checks to ensure disabling locks is safe.

This will raise an error if path can is group- or world-writable AND the current user can write to the directory (i.e., if this user AND others could write to the path).

This is intended to run on the Spack prefix, but can be run on any path for testing.

spack.util.log_parse module

spack.util.log_parse.make_log_context(log_events, width=None)[source]

Get error context from a log file.

Parameters
  • log_events (list) – list of events created by ctest_log_parser.parse()

  • width (int or None) – wrap width; 0 for no limit; None to auto-size for terminal

Returns

context from the build log with errors highlighted

Return type

str

Parses the log file for lines containing errors, and prints them out with line numbers and context. Errors are highlighted with ‘>>’ and with red highlighting (if color is enabled).

Events are sorted by line number before they are displayed.

spack.util.log_parse.parse_log_events(stream, context=6, jobs=None, profile=False)[source]

Extract interesting events from a log file as a list of LogEvent.

Parameters
  • stream (str or IO) – build log name or file object

  • context (int) – lines of context to extract around each log event

  • jobs (int) – number of jobs to parse with; default ncpus

  • profile (bool) – print out profile information for parsing

Returns

two lists containig BuildError and

BuildWarning objects.

Return type

(tuple)

This is a wrapper around ctest_log_parser.CTestLogParser that lazily constructs a single CTestLogParser object. This ensures that all the regex compilation is only done once.

spack.util.mock_package module

Infrastructure used by tests for mocking packages and repos.

class spack.util.mock_package.MockPackageMultiRepo[source]

Bases: object

Mock package repository, mimicking spack.repo.Repo.

add_package(name, dependencies=None, dependency_types=None, conditions=None)[source]

Factory method for creating mock packages.

This creates a new subclass of MockPackageBase, ensures that its name and __name__ properties are set up correctly, and returns a new instance.

We use a factory function here because many functions and properties of packages need to be class functions.

Parameters
  • name (str) – name of the new package

  • dependencies (list) – list of mock packages to be dependencies for this new package (optional; no deps if not provided)

  • dependency_type (list) – list of deptypes for each dependency (optional; will be default_deptype if not provided)

  • conditions (list) – condition specs for each dependency (optional)

exists(name)[source]
get(spec)[source]
get_pkg_class(name)[source]
is_virtual(name, use_index=True)[source]
property provider_index
repo_for_pkg(name)[source]

spack.util.module_cmd module

This module contains routines related to the module command for accessing and parsing environment modules.

spack.util.module_cmd.get_path_args_from_module_line(line)[source]
spack.util.module_cmd.get_path_from_module_contents(text, module_name)[source]
spack.util.module_cmd.load_module(mod)[source]

Takes a module name and removes modules until it is possible to load that module. It then loads the provided module. Depends on the modulecmd implementation of modules used in cray and lmod.

spack.util.module_cmd.module(*args, **kwargs)[source]
spack.util.module_cmd.path_from_modules(modules)[source]

Inspect a list of TCL modules for entries that indicate the absolute path at which the library supported by said module can be found.

Parameters

modules (list) – module files to be loaded to get an external package

Returns

Guess of the prefix path where the package

spack.util.naming module

class spack.util.naming.NamespaceTrie(separator='.')[source]

Bases: object

class Element(value)[source]

Bases: object

has_value(namespace)[source]

True if there is a value set for the given namespace.

is_leaf(namespace)[source]

True if this namespace has no children in the trie.

is_prefix(namespace)[source]

True if the namespace has a value, or if it’s the prefix of one that does.

spack.util.naming.mod_to_class(mod_name)[source]

Convert a name from module style to class name style. Spack mostly follows PEP-8:

  • Module and package names use lowercase_with_underscores.

  • Class names use the CapWords convention.

Regular source code follows these convetions. Spack is a bit more liberal with its Package names and Compiler names:

  • They can contain ‘-‘ as well as ‘_’, but cannot start with ‘-‘.

  • They can start with numbers, e.g. “3proxy”.

This function converts from the module convention to the class convention by removing _ and - and converting surrounding lowercase text to CapWords. If mod_name starts with a number, the class name returned will be prepended with ‘_’ to make a valid Python identifier.

spack.util.naming.possible_spack_module_names(python_mod_name)[source]

Given a Python module name, return a list of all possible spack module names that could correspond to it.

spack.util.naming.simplify_name(name)[source]

Simplify package name to only lowercase, digits, and dashes.

Simplifies a name which may include uppercase letters, periods, underscores, and pluses. In general, we want our package names to only contain lowercase letters, digits, and dashes.

Parameters

name (str) – The original name of the package

Returns

The new name of the package

Return type

str

spack.util.naming.spack_module_to_python_module(mod_name)[source]

Given a Spack module name, returns the name by which it can be imported in Python.

spack.util.naming.valid_fully_qualified_module_name(mod_name)[source]

Return whether mod_name is a valid namespaced module name.

spack.util.naming.valid_module_name(mod_name)[source]

Return whether mod_name is valid for use in Spack.

spack.util.naming.validate_fully_qualified_module_name(mod_name)[source]

Raise an exception if mod_name is not a valid namespaced module name.

spack.util.naming.validate_module_name(mod_name)[source]

Raise an exception if mod_name is not valid.

spack.util.package_hash module

exception spack.util.package_hash.PackageHashError(message, long_message=None)[source]

Bases: SpackError

Raised for all errors encountered during package hashing.

class spack.util.package_hash.RemoveDirectives(spec)[source]

Bases: NodeTransformer

Remove Spack directives from a package AST.

This removes Spack directives (e.g., depends_on, conflicts, etc.) and metadata attributes (e.g., tags, homepage, url) in a top-level class definition within a package.py, but it does not modify nested classes or functions.

If removing directives causes a for, with, or while statement to have an empty body, we remove the entire statement. Similarly, If removing directives causes an if statement to have an empty body or else block, we’ll remove the block (or replace the body with pass if there is an else block but no body).

visit_Assign(node)[source]
visit_ClassDef(node)[source]
visit_Expr(node)[source]
visit_For(node)[source]
visit_FunctionDef(node)[source]
visit_If(node)[source]
visit_While(node)[source]
visit_With(node)[source]
class spack.util.package_hash.RemoveDocstrings[source]

Bases: NodeTransformer

Transformer that removes docstrings from a Python AST.

This removes all strings that aren’t on the RHS of an assignment statement from the body of functions, classes, and modules – even if they’re not directly after the declaration.

remove_docstring(node)[source]
visit_ClassDef(node)[source]
visit_FunctionDef(node)[source]
visit_Module(node)[source]
class spack.util.package_hash.ResolveMultiMethods(methods)[source]

Bases: NodeTransformer

Remove multi-methods when we know statically that they won’t be used.

Say we have multi-methods like this:

class SomePackage:
    def foo(self): print("implementation 1")

    @when("@1.0")
    def foo(self): print("implementation 2")

    @when("@2.0")
    @when(sys.platform == "darwin")
    def foo(self): print("implementation 3")

    @when("@3.0")
    def foo(self): print("implementation 4")

The multimethod that will be chosen at runtime depends on the package spec and on whether we’re on the darwin platform at build time (the darwin condition for implementation 3 is dynamic). We know the package spec statically; we don’t know statically what the runtime environment will be. We need to include things that can possibly affect package behavior in the package hash, and we want to exclude things when we know that they will not affect package behavior.

If we’re at version 4.0, we know that implementation 1 will win, because some @when for 2, 3, and 4 will be False. We should only include implementation 1.

If we’re at version 1.0, we know that implementation 2 will win, because it overrides implementation 1. We should only include implementation 2.

If we’re at version 3.0, we know that implementation 4 will win, because it overrides implementation 1 (the default), and some @when on all others will be False.

If we’re at version 2.0, it’s a bit more complicated. We know we can remove implementations 2 and 4, because their @when’s will never be satisfied. But, the choice between implementations 1 and 3 will happen at runtime (this is a bad example because the spec itself has platform information, and we should prefer to use that, but we allow arbitrary boolean expressions in @when’s, so this example suffices). For this case, we end up needing to include both implementation 1 and 3 in the package hash, because either could be chosen.

resolve(impl_conditions)[source]

Given list of nodes and conditions, figure out which node will be chosen.

visit_FunctionDef(func)[source]
class spack.util.package_hash.TagMultiMethods(spec)[source]

Bases: NodeVisitor

Tag @when-decorated methods in a package AST.

visit_FunctionDef(func)[source]
spack.util.package_hash.canonical_source(spec, filter_multimethods=True, source=None)[source]

Get canonical source for a spec’s package.py by unparsing its AST.

Parameters
  • filter_multimethods (bool) – By default, filter multimethods out of the AST if they are known statically to be unused. Supply False to disable.

  • source (str) – Optionally provide a string to read python code from.

spack.util.package_hash.package_ast(spec, filter_multimethods=True, source=None)[source]

Get the AST for the package.py file corresponding to spec.

Parameters
  • filter_multimethods (bool) – By default, filter multimethods out of the AST if they are known statically to be unused. Supply False to disable.

  • source (str) – Optionally provide a string to read python code from.

spack.util.package_hash.package_hash(spec, source=None)[source]

Get a hash of a package’s canonical source code.

This function is used to determine whether a spec needs a rebuild when a package’s source code changes.

Parameters

source (str) – Optionally provide a string to read python code from.

spack.util.parallel module

class spack.util.parallel.ErrorFromWorker(exc_cls, exc, tb)[source]

Bases: object

Wrapper class to report an error from a worker process

property stacktrace
class spack.util.parallel.Task(func)[source]

Bases: object

Wrapped task that trap every Exception and return it as an ErrorFromWorker object.

We are using a wrapper class instead of a decorator since the class is pickleable, while a decorator with an inner closure is not.

spack.util.parallel.num_processes(max_processes=None)[source]

Return the number of processes in a pool.

Currently the function return the minimum between the maximum number of processes and the cpus available.

When a maximum number of processes is not specified return the cpus available.

Parameters

max_processes (int or None) – maximum number of processes allowed

spack.util.parallel.parallel_map(func, arguments, max_processes=None, debug=False)[source]

Map a task object to the list of arguments, return the list of results.

Parameters
  • func (Task) – user defined task object

  • arguments (list) – list of arguments for the task

  • max_processes (int or None) – maximum number of processes allowed

  • debug (bool) – if False, raise an exception containing just the error messages from workers, if True an exception with complete stacktraces

Raises

RuntimeError – if any error occurred in the worker processes

spack.util.parallel.pool(*args, **kwargs)[source]

Context manager to start and terminate a pool of processes, similar to the default one provided in Python 3.X

Arguments are forwarded to the multiprocessing.Pool.__init__ method.

spack.util.parallel.raise_if_errors(*results, **kwargs)[source]

Analyze results from worker Processes to search for ErrorFromWorker objects. If found print all of them and raise an exception.

Parameters
  • *results – results from worker processes

  • debug – if True show complete stacktraces

Raises

RuntimeError – if ErrorFromWorker objects are in the results

spack.util.path module

Utilities for managing paths in Spack.

TODO: this is really part of spack.config. Consolidate it.

spack.util.path.canonicalize_path(path)[source]

Same as substitute_path_variables, but also take absolute path.

spack.util.path.substitute_config_variables(path)[source]

Substitute placeholders into paths.

Spack allows paths in configs to have some placeholders, as follows:

  • $env The active Spack environment.

  • $spack The Spack instance’s prefix

  • $tempdir Default temporary directory returned by tempfile.gettempdir()

  • $user The current user’s username

  • $user_cache_path The user cache directory (~/.spack, unless overridden)

These are substituted case-insensitively into the path, and users can use either $var or ${var} syntax for the variables. $env is only replaced if there is an active environment, and should only be used in environment yaml files.

spack.util.path.substitute_path_variables(path)[source]

Substitute config vars, expand environment vars, expand user home.

spack.util.pattern module

class spack.util.pattern.Args(*flags, **kwargs)[source]

Bases: Bunch

Subclass of Bunch to write argparse args more naturally.

class spack.util.pattern.Bunch(**kwargs)[source]

Bases: object

Carries a bunch of named attributes (from Alex Martelli bunch)

class spack.util.pattern.Composite(fns_to_delegate)[source]

Bases: list

class spack.util.pattern.Delegate(name, container)[source]

Bases: object

spack.util.pattern.composite(interface=None, method_list=None, container=<class 'list'>)[source]

Decorator implementing the GoF composite pattern.

Parameters
  • interface (type) – class exposing the interface to which the composite object must conform. Only non-private and non-special methods will be taken into account

  • method_list (list) – names of methods that should be part of the composite

  • container (MutableSequence) – container for the composite object (default = list). Must fulfill the MutableSequence contract. The composite class will expose the container API to manage object composition

Returns

a class decorator that patches a class adding all the methods it needs to be a composite for a given interface.

spack.util.prefix module

This file contains utilities for managing the installation prefix of a package.

class spack.util.prefix.Prefix[source]

Bases: str

This class represents an installation prefix, but provides useful attributes for referring to directories inside the prefix.

Attributes of this object are created on the fly when you request them, so any of the following is valid:

>>> prefix = Prefix('/usr')
>>> prefix.bin
/usr/bin
>>> prefix.lib64
/usr/lib64
>>> prefix.share.man
/usr/share/man
>>> prefix.foo.bar.baz
/usr/foo/bar/baz
>>> prefix.join('dashed-directory').bin64
/usr/dashed-directory/bin64

Prefix objects behave identically to strings. In fact, they subclass str. So operators like + are legal:

print('foobar ' + prefix)

This prints foobar /usr. All of this is meant to make custom installs easy.

join(string)[source]

Concatenates a string to a prefix.

Parameters

string (str) – the string to append to the prefix

Returns

the newly created installation prefix

Return type

Prefix

spack.util.py2 module

spack.util.py2.b32encode(digest)[source]

spack.util.s3 module

spack.util.s3.create_s3_session(url, connection={})[source]
spack.util.s3.get_mirror_connection(url, url_type='push')[source]
spack.util.s3.get_mirror_s3_connection_info(connection)[source]

spack.util.spack_json module

Simple wrapper around JSON to guarantee consistent use of load/dump.

exception spack.util.spack_json.SpackJSONError(msg, json_error)[source]

Bases: SpackError

Raised when there are issues with JSON parsing.

spack.util.spack_json.decode_json_dict(data)[source]

Converts str to python 2 unicodes in JSON data.

spack.util.spack_json.dump(data, stream=None)[source]

Dump JSON with a reasonable amount of indentation and separation.

spack.util.spack_json.encode_json_dict(data)[source]

Converts python 2 unicodes to str in JSON data.

spack.util.spack_json.load(stream)[source]

Spack JSON needs to be ordered to support specs.

spack.util.spack_yaml module

Enhanced YAML parsing for Spack.

  • load() preserves YAML Marks on returned objects – this allows us to access file and line information later.

  • Our load methods use ``OrderedDict class instead of YAML’s default unorderd dict.

exception spack.util.spack_yaml.SpackYAMLError(msg, yaml_error)[source]

Bases: SpackError

Raised when there are issues with YAML parsing.

spack.util.spack_yaml.dump(obj, default_flow_style=False, stream=None)[source]
spack.util.spack_yaml.load(*args, **kwargs)[source]

spack.util.string module

spack.util.string.comma_and(sequence)[source]
spack.util.string.comma_list(sequence, article='')[source]
spack.util.string.comma_or(sequence)[source]
spack.util.string.plural(n, singular, plural=None, show_n=True)[source]

Pluralize <singular> word by adding an s if n != 1.

Parameters
  • n (int) – number of things there are

  • singular (str) – singular form of word

  • plural (str or None) – optional plural form, for when it’s not just singular + ‘s’

  • show_n (bool) – whether to include n in the result string (default True)

Returns

“1 thing” if n == 1 or “n things” if n != 1

Return type

(str)

spack.util.string.quote(sequence, q="'")[source]

spack.util.timer module

Debug signal handler: prints a stack trace and enters interpreter.

register_interrupt_handler() enables a ctrl-C handler that prints a stack trace and drops the user into an interpreter.

class spack.util.timer.Timer[source]

Bases: object

Simple timer for timing phases of a solve or install

phase(name)[source]
stop()[source]

Stop the timer to record a total time, if desired.

property total

Return the total time

write_json(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)[source]

Write a json object with times to file

write_tty(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)[source]

spack.util.url module

Utility functions for parsing, formatting, and manipulating URLs.

spack.util.url.escape_file_url(url)[source]
spack.util.url.format(parsed_url)[source]

Format a URL string

Returns a canonicalized format of the given URL as a string.

spack.util.url.join(base_url, path, *extra, **kwargs)[source]

Joins a base URL with one or more local URL path components

If resolve_href is True, treat the base URL as though it where the locator of a web page, and the remaining URL path components as though they formed a relative URL to be resolved against it (i.e.: as in posixpath.join(…)). The result is an absolute URL to the resource to which a user’s browser would navigate if they clicked on a link with an “href” attribute equal to the relative URL.

If resolve_href is False (default), then the URL path components are joined as in posixpath.join().

Note: file:// URL path components are not canonicalized as part of this operation. To canonicalize, pass the joined url to format().

Examples

base_url = ‘s3://bucket/index.html’ body = fetch_body(prefix) link = get_href(body) # link == ‘../other-bucket/document.txt’

# wrong - link is a local URL that needs to be resolved against base_url spack.util.url.join(base_url, link) ‘s3://bucket/other_bucket/document.txt’

# correct - resolve local URL against base_url spack.util.url.join(base_url, link, resolve_href=True) ‘s3://other_bucket/document.txt’

prefix = ‘https://mirror.spack.io/build_cache

# wrong - prefix is just a URL prefix spack.util.url.join(prefix, ‘my-package’, resolve_href=True) ‘https://mirror.spack.io/my-package

# correct - simply append additional URL path components spack.util.url.join(prefix, ‘my-package’, resolve_href=False) # default ‘https://mirror.spack.io/build_cache/my-package

# For canonicalizing file:// URLs, take care to explicitly differentiate # between absolute and relative join components.

# ‘$spack’ is not an absolute path component join_result = spack.util.url.join(‘/a/b/c’, ‘$spack’) ; join_result ‘file:///a/b/c/$spack’ spack.util.url.format(join_result) ‘file:///a/b/c/opt/spack

# ‘/$spack’ is an absolute path component join_result = spack.util.url.join(‘/a/b/c’, ‘/$spack’) ; join_result ‘file:///$spack’ spack.util.url.format(join_result) ‘file:///opt/spack

spack.util.url.local_file_path(url)[source]

Get a local file path from a url.

If url is a file:// URL, return the absolute path to the local file or directory referenced by it. Otherwise, return None.

spack.util.url.parse(url, scheme='file')[source]

Parse a url.

For file:// URLs, the netloc and path components are concatenated and passed through spack.util.path.canoncalize_path().

Otherwise, the returned value is the same as urllib’s urlparse() with allow_fragments=False.

spack.util.url.parse_git_url(url)[source]

Parse git URL into components.

This parses URLs that look like:

  • https://host.com:443/path/to/repo.git, or

  • git@host.com:path/to/repo.git

Anything not matching those patterns is likely a local file or invalid.

Returned components are as follows (optional values can be None):

  1. scheme (optional): git, ssh, http, https

  2. user (optional): git@ for github, username for http or ssh

  3. hostname: domain of server

  4. port (optional): port on server

  5. path: path on the server, e.g. spack/spack

Returns

tuple containing URL components as above

Return type

(tuple)

Raises ValueError for invalid URLs.

spack.util.url.require_url_format(url)[source]

spack.util.web module

exception spack.util.web.HTMLParseError[source]

Bases: Exception

class spack.util.web.LinkParser[source]

Bases: HTMLParser

This parser just takes an HTML page and strips out the hrefs on the links. Good enough for a really simple spider.

handle_starttag(tag, attrs)[source]
exception spack.util.web.NoNetworkConnectionError(message, url)[source]

Bases: SpackWebError

Raised when an operation can’t get an internet connection.

spack.util.web.SPACK_USER_AGENT = 'Spackbot/0.19.0.dev0'

User-Agent used in Request objects

exception spack.util.web.SpackWebError(message, long_message=None)[source]

Bases: SpackError

Superclass for Spack web spidering errors.

spack.util.web.find_versions_of_archive(archive_urls, list_url=None, list_depth=0, concurrency=32, reference_package=None)[source]

Scrape web pages for new versions of a tarball. This function prefers URLs in the following order: links found on the scraped page that match a url generated by the reference package, found and in the archive_urls list, found and derived from those in the archive_urls list, and if none are found for a version then the item in the archive_urls list is included for the version.

Parameters
  • archive_urls (str or list or tuple) – URL or sequence of URLs for different versions of a package. Typically these are just the tarballs from the package file itself. By default, this searches the parent directories of archives.

  • list_url (str or None) – URL for a listing of archives. Spack will scrape these pages for download links that look like the archive URL.

  • list_depth (int) – max depth to follow links on list_url pages. Defaults to 0.

  • concurrency (int) – maximum number of concurrent requests

  • reference_package (spack.package_base.Package or None) – a spack package used as a reference for url detection. Uses the url_for_version method on the package to produce reference urls which, if found, are preferred.

spack.util.web.get_header(headers, header_name)[source]

Looks up a dict of headers for the given header value.

Looks up a dict of headers, [headers], for a header value given by [header_name]. Returns headers[header_name] if header_name is in headers. Otherwise, the first fuzzy match is returned, if any.

This fuzzy matching is performed by discarding word separators and capitalization, so that for example, “Content-length”, “content_length”, “conTENtLength”, etc., all match. In the case of multiple fuzzy-matches, the returned value is the “first” such match given the underlying mapping’s ordering, or unspecified if no such ordering is defined.

If header_name is not in headers, and no such fuzzy match exists, then a KeyError is raised.

spack.util.web.list_url(url, recursive=False)[source]
spack.util.web.push_to_url(local_file_path, remote_path, keep_original=True, extra_args=None)[source]
spack.util.web.read_from_url(url, accept_content_type=None)[source]
spack.util.web.remove_url(url, recursive=False)[source]
spack.util.web.spider(root_urls, depth=0, concurrency=32)[source]

Get web pages from root URLs.

If depth is specified (e.g., depth=2), then this will also follow up to <depth> levels of links from each root.

Parameters
  • root_urls (str or list) – root urls used as a starting point for spidering

  • depth (int) – level of recursion into links

  • concurrency (int) – number of simultaneous requests that can be sent

Returns

A dict of pages visited (URL) mapped to their full text and the set of visited links.

spack.util.web.url_exists(url)[source]
spack.util.web.uses_ssl(parsed_url)[source]
spack.util.web.warn_no_ssl_cert_checking()[source]

Module contents