spack.util package

Submodules

spack.util.compression module

spack.util.compression.allowed_archive(path)
spack.util.compression.decompressor_for(path, extension=None)

Get the appropriate decompressor for a path.

spack.util.compression.extension(path)

Get the archive extension for a path.

spack.util.compression.strip_extension(path)

Get the part of a path that does not include its compressed type extension.

spack.util.crypto module

class spack.util.crypto.Checker(hexdigest, **kwargs)

Bases: object

A checker checks files against one particular hex digest. It will automatically determine what hashing algorithm to used based on the length of the digest it’s initialized with. e.g., if the digest is 32 hex characters long this will use md5.

Example: know your tarball should hash to ‘abc123’. You want to check files against this. You would use this class like so:

hexdigest = 'abc123'
checker = Checker(hexdigest)
success = checker.check('downloaded.tar.gz')

After the call to check, the actual checksum is available in checker.sum, in case it’s needed for error output.

You can trade read performance and memory usage by adjusting the block_size optional arg. By default it’s a 1MB (2**20 bytes) buffer.

check(filename)

Read the file with the specified name and check its checksum against self.hexdigest. Return True if they match, False otherwise. Actual checksum is stored in self.sum.

hash_name

Get the name of the hash function this Checker is using.

class spack.util.crypto.DeprecatedHash(hash_alg, alert_fn, disable_security_check)

Bases: object

spack.util.crypto.bit_length(num)

Number of bits required to represent an integer in binary.

spack.util.crypto.checksum(hashlib_algo, filename, **kwargs)

Returns a hex digest of the filename generated using an algorithm from hashlib.

spack.util.crypto.hash_algo_for_digest(hexdigest)

Gets name of the hash algorithm for a hex digest.

spack.util.crypto.hash_fun_for_algo(algo)

Get a function that can perform the specified hash algorithm.

spack.util.crypto.hash_fun_for_digest(hexdigest)

Gets a hash function corresponding to a hex digest.

spack.util.crypto.hashes = {'md5': 16, 'sha1': 20, 'sha224': 28, 'sha256': 32, 'sha384': 48, 'sha512': 64}

Set of hash algorithms that Spack can use, mapped to digest size in bytes

spack.util.crypto.prefix_bits(byte_array, bits)

Return the first <bits> bits of a byte array as an integer.

spack.util.debug module

Debug signal handler: prints a stack trace and enters interpreter.

register_interrupt_handler() enables a ctrl-C handler that prints a stack trace and drops the user into an interpreter.

spack.util.debug.debug_handler(sig, frame)

Interrupt running process, and provide a python prompt for interactive debugging.

spack.util.debug.register_interrupt_handler()

Print traceback and enter an interpreter on Ctrl-C

spack.util.editor module

Module for finding the user’s preferred text editor.

Defines one variable: editor, which is a spack.util.executable.Executable object that can be called to invoke the editor.

If no editor is found, an EnvironmentError is raised when editor is invoked.

spack.util.environment module

spack.util.environment.dump_environment(path)

Dump the current environment out to a file.

spack.util.environment.env_flag(name)
spack.util.environment.filter_system_paths(paths)
spack.util.environment.get_path(name)
spack.util.environment.is_system_path(path)

Predicate that given a path returns True if it is a system path, False otherwise.

Parameters:path (str) – path to a directory
Returns:True or False
spack.util.environment.path_put_first(var_name, directories)

Puts the provided directories first in the path, adding them if they’re not already there.

spack.util.environment.path_set(var_name, directories)
spack.util.environment.set_env(**kwargs)

Temporarily sets and restores environment variables.

Variables can be set as keyword arguments to this function.

spack.util.executable module

class spack.util.executable.Executable(name)

Bases: object

Class representing a program that can be run on the command line.

add_default_arg(arg)

Add a default argument to the command.

add_default_env(key, value)

Set an environment variable when the command is run.

Parameters:
  • key – The environment variable to set
  • value – The value to set it to
command

The command-line string.

Returns:The executable and default arguments
Return type:str
name

The executable name.

Returns:The basename of the executable
Return type:str
path

The path to the executable.

Returns:The path to the executable
Return type:str
spack.util.executable.which(*args, **kwargs)

Finds an executable in the path like command-line which.

If given multiple executables, returns the first one that is found. If no executables are found, returns None.

Parameters:

*args (str) – One or more executables to search for

Keyword Arguments:
 
  • path (list() or str) – The path to search. Defaults to PATH
  • required (bool) – If set to True, raise an error if executable not found
Returns:

The first executable that is found in the path

Return type:

Executable

exception spack.util.executable.ProcessError(message, long_message=None)

Bases: spack.error.SpackError

ProcessErrors are raised when Executables exit with an error code.

spack.util.file_cache module

exception spack.util.file_cache.CacheError(message, long_message=None)

Bases: spack.error.SpackError

class spack.util.file_cache.FileCache(root)

Bases: object

This class manages cached data in the filesystem.

  • Cache files are fetched and stored by unique keys. Keys can be relative paths, so that there can be some hierarchy in the cache.
  • The FileCache handles locking cache files for reading and writing, so client code need not manage locks for cache entries.
cache_path(key)

Path to the file in the cache for a particular key.

destroy()

Remove all files under the cache root.

init_entry(key)

Ensure we can access a cache file. Create a lock for it if needed.

Return whether the cache file exists yet or not.

mtime(key)

Return modification time of cache file, or 0 if it does not exist.

Time is in units returned by os.stat in the mtime field, which is platform-dependent.

read_transaction(key)

Get a read transaction on a file cache item.

Returns a ReadTransaction context manager and opens the cache file for reading. You can use it like this:

with file_cache_object.read_transaction(key) as cache_file:
cache_file.read()
remove(key)
write_transaction(key)

Get a write transaction on a file cache item.

Returns a WriteTransaction context manager that opens a temporary file for writing. Once the context manager finishes, if nothing went wrong, moves the file into place on top of the old file atomically.

spack.util.gpg module

class spack.util.gpg.Gpg

Bases: object

classmethod create(**kwargs)
classmethod export_keys(location, *keys)
static gpg()
classmethod list(trusted, signing)
classmethod sign(key, file, output, clearsign=False)
classmethod signing_keys()
classmethod trust(keyfile)
classmethod untrust(signing, *keys)
classmethod verify(signature, file)

spack.util.lock module

Wrapper for llnl.util.lock allows locking to be enabled/disabled.

class spack.util.lock.Lock(*args, **kwargs)

Bases: llnl.util.lock.Lock

Lock that can be disabled.

This overrides the _lock() and _unlock() methods from llnl.util.lock so that all the lock API calls will succeed, but the actual locking mechanism can be disabled via _enable_locks.

spack.util.lock.check_lock_safety(path)

Do some extra checks to ensure disabling locks is safe.

This will raise an error if path can is group- or world-writable AND the current user can write to the directory (i.e., if this user AND others could write to the path).

This is intended to run on the Spack prefix, but can be run on any path for testing.

spack.util.log_parse module

spack.util.log_parse.parse_log_events(stream, context=6, jobs=None, profile=False)

Extract interesting events from a log file as a list of LogEvent.

Parameters:
  • stream (str or fileobject) – build log name or file object
  • context (int) – lines of context to extract around each log event
  • jobs (int) – number of jobs to parse with; default ncpus
  • profile (bool) – print out profile information for parsing
Returns:

two lists containig BuildError and

BuildWarning objects.

Return type:

(tuple)

This is a wrapper around ctest_log_parser.CTestLogParser that lazily constructs a single CTestLogParser object. This ensures that all the regex compilation is only done once.

spack.util.log_parse.make_log_context(log_events, width=None)

Get error context from a log file.

Parameters:
  • log_events (list of LogEvent) – list of events created by ctest_log_parser.parse()
  • width (int or None) – wrap width; 0 for no limit; None to auto-size for terminal
Returns:

context from the build log with errors highlighted

Return type:

str

Parses the log file for lines containing errors, and prints them out with line numbers and context. Errors are highlighted with ‘>>’ and with red highlighting (if color is enabled).

Events are sorted by line number before they are displayed.

spack.util.module_cmd module

This module contains routines related to the module command for accessing and parsing environment modules.

exception spack.util.module_cmd.ModuleError

Bases: Exception

Raised the the module_cmd utility to indicate errors.

spack.util.module_cmd.get_argument_from_module_line(line)
spack.util.module_cmd.get_module_cmd(bashopts='')
spack.util.module_cmd.get_module_cmd_from_bash(bashopts='')
spack.util.module_cmd.get_module_cmd_from_which()
spack.util.module_cmd.get_path_from_module(mod)

Inspects a TCL module for entries that indicate the absolute path at which the library supported by said module can be found.

spack.util.module_cmd.load_module(mod)

Takes a module name and removes modules until it is possible to load that module. It then loads the provided module. Depends on the modulecmd implementation of modules used in cray and lmod.

spack.util.module_cmd.unload_module(mod)

Takes a module name and unloads the module from the environment. It does not check whether conflicts arise from the unloaded module

spack.util.naming module

spack.util.naming.mod_to_class(mod_name)

Convert a name from module style to class name style. Spack mostly follows PEP-8:

  • Module and package names use lowercase_with_underscores.
  • Class names use the CapWords convention.

Regular source code follows these convetions. Spack is a bit more liberal with its Package names and Compiler names:

  • They can contain ‘-‘ as well as ‘_’, but cannot start with ‘-‘.
  • They can start with numbers, e.g. “3proxy”.

This function converts from the module convention to the class convention by removing _ and - and converting surrounding lowercase text to CapWords. If mod_name starts with a number, the class name returned will be prepended with ‘_’ to make a valid Python identifier.

spack.util.naming.spack_module_to_python_module(mod_name)

Given a Spack module name, returns the name by which it can be imported in Python.

spack.util.naming.valid_module_name(mod_name)

Return whether mod_name is valid for use in Spack.

spack.util.naming.valid_fully_qualified_module_name(mod_name)

Return whether mod_name is a valid namespaced module name.

spack.util.naming.validate_fully_qualified_module_name(mod_name)

Raise an exception if mod_name is not a valid namespaced module name.

spack.util.naming.validate_module_name(mod_name)

Raise an exception if mod_name is not valid.

spack.util.naming.possible_spack_module_names(python_mod_name)

Given a Python module name, return a list of all possible spack module names that could correspond to it.

spack.util.naming.simplify_name(name)

Simplify package name to only lowercase, digits, and dashes.

Simplifies a name which may include uppercase letters, periods, underscores, and pluses. In general, we want our package names to only contain lowercase letters, digits, and dashes.

Parameters:name (str) – The original name of the package
Returns:The new name of the package
Return type:str
class spack.util.naming.NamespaceTrie(separator='.')

Bases: object

class Element(value)

Bases: object

has_value(namespace)

True if there is a value set for the given namespace.

is_leaf(namespace)

True if this namespace has no children in the trie.

is_prefix(namespace)

True if the namespace has a value, or if it’s the prefix of one that does.

spack.util.package_hash module

exception spack.util.package_hash.PackageHashError(message, long_message=None)

Bases: spack.error.SpackError

Raised for all errors encountered during package hashing.

class spack.util.package_hash.RemoveDirectives(spec)

Bases: ast.NodeTransformer

Remove Spack directives from a package AST.

is_directive(node)
is_spack_attr(node)
visit_ClassDef(node)
class spack.util.package_hash.RemoveDocstrings

Bases: ast.NodeTransformer

Transformer that removes docstrings from a Python AST.

remove_docstring(node)
visit_ClassDef(node)
visit_FunctionDef(node)
visit_Module(node)
class spack.util.package_hash.ResolveMultiMethods(methods)

Bases: ast.NodeTransformer

Remove methods which do not exist if their @when is not satisfied.

resolve(node)
visit_FunctionDef(node)
class spack.util.package_hash.TagMultiMethods(spec)

Bases: ast.NodeVisitor

Tag @when-decorated methods in a spec.

visit_FunctionDef(node)
spack.util.package_hash.package_ast(spec)
spack.util.package_hash.package_content(spec)
spack.util.package_hash.package_hash(spec, content=None)

spack.util.path module

Utilities for managing paths in Spack.

TODO: this is really part of spack.config. Consolidate it.

spack.util.path.substitute_config_variables(path)

Substitute placeholders into paths.

Spack allows paths in configs to have some placeholders, as follows:

  • $spack The Spack instance’s prefix
  • $user The current user’s username
  • $tempdir Default temporary directory returned by tempfile.gettempdir()

These are substituted case-insensitively into the path, and users can use either $var or ${var} syntax for the variables.

spack.util.path.canonicalize_path(path)

Substitute config vars, expand environment vars, expand user home, take abspath.

spack.util.pattern module

class spack.util.pattern.Args(*flags, **kwargs)

Bases: spack.util.pattern.Bunch

Subclass of Bunch to write argparse args more naturally.

class spack.util.pattern.Bunch(**kwargs)

Bases: object

Carries a bunch of named attributes (from Alex Martelli bunch)

spack.util.pattern.composite(interface=None, method_list=None, container=<class 'list'>)

Decorator implementing the GoF composite pattern.

Parameters:
  • interface (type) – class exposing the interface to which the composite object must conform. Only non-private and non-special methods will be taken into account
  • method_list (list of str) – names of methods that should be part of the composite
  • container (MutableSequence) – container for the composite object (default = list). Must fulfill the MutableSequence contract. The composite class will expose the container API to manage object composition
Returns:

a class decorator that patches a class adding all the methods it needs to be a composite for a given interface.

spack.util.prefix module

This file contains utilities for managing the installation prefix of a package.

class spack.util.prefix.Prefix

Bases: str

This class represents an installation prefix, but provides useful attributes for referring to directories inside the prefix.

Attributes of this object are created on the fly when you request them, so any of the following is valid:

>>> prefix = Prefix('/usr')
>>> prefix.bin
/usr/bin
>>> prefix.lib64
/usr/lib64
>>> prefix.share.man
/usr/share/man
>>> prefix.foo.bar.baz
/usr/foo/bar/baz
>>> prefix.join('dashed-directory').bin64
/usr/dashed-directory/bin64

Prefix objects behave identically to strings. In fact, they subclass str. So operators like + are legal:

print('foobar ' + prefix)

This prints foobar /usr. All of this is meant to make custom installs easy.

join(string)

Concatenates a string to a prefix.

Parameters:string (str) – the string to append to the prefix
Returns:the newly created installation prefix
Return type:Prefix

spack.util.spack_json module

Simple wrapper around JSON to guarantee consistent use of load/dump.

spack.util.spack_json.load(stream)

Spack JSON needs to be ordered to support specs.

spack.util.spack_json.dump(data, stream=None)

Dump JSON with a reasonable amount of indentation and separation.

exception spack.util.spack_json.SpackJSONError(msg, json_error)

Bases: spack.error.SpackError

Raised when there are issues with JSON parsing.

spack.util.spack_yaml module

Enhanced YAML parsing for Spack.

  • load() preserves YAML Marks on returned objects – this allows us to access file and line information later.
  • Our load methods use ``OrderedDict class instead of YAML’s default unorderd dict.
spack.util.spack_yaml.load(*args, **kwargs)

Load but modify the loader instance so that it will add __line__ atrributes to the returned object.

spack.util.spack_yaml.dump(*args, **kwargs)
exception spack.util.spack_yaml.SpackYAMLError(msg, yaml_error)

Bases: spack.error.SpackError

Raised when there are issues with YAML parsing.

spack.util.string module

spack.util.string.comma_and(sequence)
spack.util.string.comma_list(sequence, article='')
spack.util.string.comma_or(sequence)
spack.util.string.quote(sequence, q="'")

spack.util.web module

exception spack.util.web.HTMLParseError

Bases: Exception

class spack.util.web.LinkParser

Bases: html.parser.HTMLParser

This parser just takes an HTML page and strips out the hrefs on the links. Good enough for a really simple spider.

handle_starttag(tag, attrs)
exception spack.util.web.NoNetworkConnectionError(message, url)

Bases: spack.util.web.SpackWebError

Raised when an operation can’t get an internet connection.

class spack.util.web.NonDaemonPool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, context=None)

Bases: multiprocessing.pool.Pool

Pool that uses non-daemon processes

Process

alias of NonDaemonProcess

class spack.util.web.NonDaemonProcess(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)

Bases: multiprocessing.context.Process

Process tha allows sub-processes, so pools can have sub-pools.

daemon
exception spack.util.web.SpackWebError(message, long_message=None)

Bases: spack.error.SpackError

Superclass for Spack web spidering errors.

exception spack.util.web.VersionFetchError(message, long_message=None)

Bases: spack.util.web.SpackWebError

Raised when we can’t determine a URL to fetch a package.

spack.util.web.find_versions_of_archive(archive_urls, list_url=None, list_depth=0)

Scrape web pages for new versions of a tarball.

Parameters:

archive_urls – URL or sequence of URLs for different versions of a package. Typically these are just the tarballs from the package file itself. By default, this searches the parent directories of archives.

Keyword Arguments:
 
  • list_url – URL for a listing of archives. Spack wills scrape these pages for download links that look like the archive URL.
  • list_depth – Max depth to follow links on list_url pages. Default 0.
spack.util.web.get_checksums_for_versions(url_dict, name, first_stage_function=None, keep_stage=False)

Fetches and checksums archives from URLs.

This function is called by both spack checksum and spack create. The first_stage_function argument allows the caller to inspect the first downloaded archive, e.g., to determine the build system.

Parameters:
  • url_dict (dict) – A dictionary of the form: version -> URL
  • name (str) – The name of the package
  • first_stage_function (callable) – function that takes a Stage and a URL; this is run on the stage of the first URL downloaded
  • keep_stage (bool) – whether to keep staging area when command completes
Returns:

A multi-line string containing versions and corresponding hashes

Return type:

(str)

spack.util.web.spider(root_url, depth=0)

Gets web pages from a root URL.

If depth is specified (e.g., depth=2), then this will also follow up to <depth> levels of links from the root.

This will spawn processes to fetch the children, for much improved performance over a sequential fetch.

Module contents