spack.util package

Submodules

spack.util.compression module

spack.util.compression.allowed_archive(path)
spack.util.compression.decompressor_for(path, extension=None)

Get the appropriate decompressor for a path.

spack.util.compression.extension(path)

Get the archive extension for a path.

spack.util.compression.strip_extension(path)

Get the part of a path that does not include its compressed type extension.

spack.util.crypto module

class spack.util.crypto.Checker(hexdigest, **kwargs)

Bases: object

A checker checks files against one particular hex digest. It will automatically determine what hashing algorithm to used based on the length of the digest it’s initialized with. e.g., if the digest is 32 hex characters long this will use md5.

Example: know your tarball should hash to ‘abc123’. You want to check files against this. You would use this class like so:

hexdigest = 'abc123'
checker = Checker(hexdigest)
success = checker.check('downloaded.tar.gz')

After the call to check, the actual checksum is available in checker.sum, in case it’s needed for error output.

You can trade read performance and memory usage by adjusting the block_size optional arg. By default it’s a 1MB (2**20 bytes) buffer.

check(filename)

Read the file with the specified name and check its checksum against self.hexdigest. Return True if they match, False otherwise. Actual checksum is stored in self.sum.

hash_name

Get the name of the hash function this Checker is using.

spack.util.crypto.bit_length(num)

Number of bits required to represent an integer in binary.

spack.util.crypto.checksum(hashlib_algo, filename, **kwargs)

Returns a hex digest of the filename generated using an algorithm from hashlib.

spack.util.crypto.prefix_bits(byte_array, bits)

Return the first <bits> bits of a byte array as an integer.

spack.util.debug module

Debug signal handler: prints a stack trace and enters interpreter.

register_interrupt_handler() enables a ctrl-C handler that prints a stack trace and drops the user into an interpreter.

spack.util.debug.debug_handler(sig, frame)

Interrupt running process, and provide a python prompt for interactive debugging.

spack.util.debug.register_interrupt_handler()

Print traceback and enter an interpreter on Ctrl-C

spack.util.environment module

spack.util.environment.dump_environment(path)

Dump the current environment out to a file.

spack.util.environment.env_flag(name)
spack.util.environment.filter_system_bin_paths(paths)
spack.util.environment.filter_system_paths(paths)
spack.util.environment.get_path(name)
spack.util.environment.path_put_first(var_name, directories)

Puts the provided directories first in the path, adding them if they’re not already there.

spack.util.environment.path_set(var_name, directories)

spack.util.executable module

class spack.util.executable.Executable(name)

Bases: object

Class representing a program that can be run on the command line.

add_default_arg(arg)
add_default_env(key, value)
command
spack.util.executable.which(name, **kwargs)

Finds an executable in the path like command-line which.

exception spack.util.executable.ProcessError(message, long_message=None)

Bases: spack.error.SpackError

ProcessErrors are raised when Executables exit with an error code.

spack.util.multiproc module

This implements a parallel map operation but it can accept more values than multiprocessing.Pool.apply() can. For example, apply() will fail to pickle functions if they’re passed indirectly as parameters.

spack.util.multiproc.spawn(f)
spack.util.multiproc.parmap(f, X)
class spack.util.multiproc.Barrier(n, timeout=None)

Simple reusable semaphore barrier.

Python 2.6 doesn’t have multiprocessing barriers so we implement this.

See http://greenteapress.com/semaphores/downey08semaphores.pdf, p. 41.

wait()

spack.util.naming module

spack.util.naming.mod_to_class(mod_name)

Convert a name from module style to class name style. Spack mostly follows PEP-8:

  • Module and package names use lowercase_with_underscores.
  • Class names use the CapWords convention.

Regular source code follows these convetions. Spack is a bit more liberal with its Package names nad Compiler names:

  • They can contain ‘-‘ as well as ‘_’, but cannot start with ‘-‘.
  • They can start with numbers, e.g. “3proxy”.

This function converts from the module convention to the class convention by removing _ and - and converting surrounding lowercase text to CapWords. If mod_name starts with a number, the class name returned will be prepended with ‘_’ to make a valid Python identifier.

spack.util.naming.spack_module_to_python_module(mod_name)

Given a Spack module name, returns the name by which it can be imported in Python.

spack.util.naming.valid_module_name(mod_name)

Return whether mod_name is valid for use in Spack.

spack.util.naming.valid_fully_qualified_module_name(mod_name)

Return whether mod_name is a valid namespaced module name.

spack.util.naming.validate_fully_qualified_module_name(mod_name)

Raise an exception if mod_name is not a valid namespaced module name.

spack.util.naming.validate_module_name(mod_name)

Raise an exception if mod_name is not valid.

spack.util.naming.possible_spack_module_names(python_mod_name)

Given a Python module name, return a list of all possible spack module names that could correspond to it.

class spack.util.naming.NamespaceTrie(separator='.')

Bases: object

class Element(value)

Bases: object

NamespaceTrie.has_value(namespace)

True if there is a value set for the given namespace.

NamespaceTrie.is_leaf(namespace)

True if this namespace has no children in the trie.

NamespaceTrie.is_prefix(namespace)

True if the namespace has a value, or if it’s the prefix of one that does.

spack.util.path module

Utilities for managing paths in Spack.

spack.util.path.substitute_config_variables(path)

Substitute placeholders into paths.

Spack allows paths in configs to have some placeholders, as follows:

  • $spack The Spack instance’s prefix
  • $user The current user’s username
  • $tempdir Default temporary directory returned by tempfile.gettempdir()

These are substituted case-insensitively into the path, and users can use either $var or ${var} syntax for the variables.

spack.util.path.canonicalize_path(path)

Substitute config vars, expand user home, take abspath.

spack.util.pattern module

class spack.util.pattern.Args(*flags, **kwargs)

Bases: spack.util.pattern.Bunch

Subclass of Bunch to write argparse args more naturally.

class spack.util.pattern.Bunch(**kwargs)

Bases: object

Carries a bunch of named attributes (from Alex Martelli bunch)

spack.util.pattern.composite(interface=None, method_list=None, container=<type 'list'>)

Returns a class decorator that patches a class adding all the methods it needs to be a composite for a given interface.

Parameters:
  • interface – class exposing the interface to which the composite object must conform. Only non-private and non-special methods will be taken into account
  • method_list – names of methods that should be part of the composite
  • container – container for the composite object (default = list). Must fulfill the MutableSequence contract. The composite class will expose the container API to manage object composition
Returns:

class decorator

spack.util.prefix module

This file contains utilities to help with installing packages.

class spack.util.prefix.Prefix

Bases: str

This class represents an installation prefix, but provides useful attributes for referring to directories inside the prefix.

For example, you can do something like this:

prefix = Prefix('/usr')
print prefix.lib
print prefix.lib64
print prefix.bin
print prefix.share
print prefix.man4

This program would print:

/usr/lib /usr/lib64 /usr/bin /usr/share /usr/share/man/man4

Prefix objects behave identically to strings. In fact, they subclass str. So operators like + are legal:

print “foobar ” + prefix

This prints ‘foobar /usr”. All of this is meant to make custom installs easy.

spack.util.spack_json module

Simple wrapper around JSON to guarantee consistent use of load/dump.

spack.util.spack_json.load(stream)

Spack JSON needs to be ordered to support specs.

spack.util.spack_json.dump(data, stream=None)

Dump JSON with a reasonable amount of indentation and separation.

exception spack.util.spack_json.SpackJSONError(msg, yaml_error)

Bases: spack.error.SpackError

Raised when there are issues with JSON parsing.

spack.util.spack_yaml module

Enhanced YAML parsing for Spack.

  • load() preserves YAML Marks on returned objects – this allows us to access file and line information later.
  • Our load methods use ``OrderedDict class instead of YAML’s default unorderd dict.
spack.util.spack_yaml.load(*args, **kwargs)

Load but modify the loader instance so that it will add __line__ atrributes to the returned object.

spack.util.spack_yaml.dump(*args, **kwargs)
exception spack.util.spack_yaml.SpackYAMLError(msg, yaml_error)

Bases: spack.error.SpackError

Raised when there are issues with YAML parsing.

spack.util.string module

spack.util.string.comma_and(sequence)
spack.util.string.comma_list(sequence, article='')
spack.util.string.comma_or(sequence)

spack.util.web module

class spack.util.web.LinkParser

Bases: HTMLParser.HTMLParser

This parser just takes an HTML page and strips out the hrefs on the links. Good enough for a really simple spider.

handle_starttag(tag, attrs)
spack.util.web.find_versions_of_archive(*archive_urls, **kwargs)

Scrape web pages for new versions of a tarball.

Arguments:
archive_urls:
URLs for different versions of a package. Typically these are just the tarballs from the package file itself. By default, this searches the parent directories of archives.
Keyword Arguments:

list_url:

URL for a listing of archives. Spack wills scrape these pages for download links that look like the archive URL.
list_depth:
Max depth to follow links on list_url pages.
spack.util.web.spider(root_url, **kwargs)

Gets web pages from a root URL. If depth is specified (e.g., depth=2), then this will also fetches pages linked from the root and its children up to depth.

This will spawn processes to fetch the children, for much improved performance over a sequential fetch.

Module contents