llnl.util package

Submodules

llnl.util.argparsewriter module

class llnl.util.argparsewriter.ArgparseRstWriter(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>, rst_levels=['=', '-', '^', '~', ':', '`'], strip_root_prog=True)

Bases: llnl.util.argparsewriter.ArgparseWriter

Write argparse output as rst sections.

begin_command(prog)
begin_optionals()
begin_positionals()
begin_subcommands(subcommands)
description(description)
line(string='')
optional(opts, help)
positional(name, help)
usage(usage)
class llnl.util.argparsewriter.ArgparseWriter(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

Bases: object

Analyzes an argparse ArgumentParser for easy generation of help.

begin_command(prog)
begin_optionals()
begin_positionals()
begin_subcommands(subcommands)
description(description)
end_command(prog)
end_optionals()
end_positionals()
end_subcommands(subcommands)
optional(option, help)
positional(name, help)
usage(usage)
write(parser, root=True)

Write out details about an ArgumentParser.

Parameters
  • parser (ArgumentParser) – an argparse parser

  • root (bool or int) – if bool, whether to include the root parser; or 1 to flatten the root parser with first-level subcommands

llnl.util.filesystem module

class llnl.util.filesystem.FileFilter(*filenames)

Bases: object

Convenience class for calling filter_file a lot.

filter(regex, repl, **kwargs)
class llnl.util.filesystem.FileList(files)

Bases: collections.abc.Sequence

Sequence of absolute paths to files.

Provides a few convenience methods to manipulate file paths.

basenames

Stable de-duplication of the base-names in the list

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir3/liba.a'])
>>> l.basenames
['liba.a', 'libb.a']
>>> h = HeaderList(['/dir1/a.h', '/dir2/b.h', '/dir3/a.h'])
>>> h.basenames
['a.h', 'b.h']
Returns

A list of base-names

Return type

list of strings

directories

Stable de-duplication of the directories where the files reside.

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/libc.a'])
>>> l.directories
['/dir1', '/dir2']
>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.directories
['/dir1', '/dir2']
Returns

A list of directories

Return type

list of strings

joined(separator=' ')
class llnl.util.filesystem.HeaderList(files)

Bases: llnl.util.filesystem.FileList

Sequence of absolute paths to headers.

Provides a few convenience methods to manipulate header paths and get commonly used compiler flags or names.

add_macro(macro)

Add a macro definition

Parameters

macro (str) – The macro to add

cpp_flags

Include flags + macro definitions

>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.cpp_flags
'-I/dir1 -I/dir2'
>>> h.add_macro('-DBOOST_DYN_LINK')
>>> h.cpp_flags
'-I/dir1 -I/dir2 -DBOOST_DYN_LINK'
Returns

A joined list of include flags and macro definitions

Return type

str

directories

Directories to be searched for header files.

headers

Stable de-duplication of the headers.

Returns

A list of header files

Return type

list of strings

include_flags

Include flags

>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.include_flags
'-I/dir1 -I/dir2'
Returns

A joined list of include flags

Return type

str

include_regex = re.compile('(.*)(\\binclude\\b)(.*)')
macro_definitions

Macro definitions

>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.add_macro('-DBOOST_LIB_NAME=boost_regex')
>>> h.add_macro('-DBOOST_DYN_LINK')
>>> h.macro_definitions
'-DBOOST_LIB_NAME=boost_regex -DBOOST_DYN_LINK'
Returns

A joined list of macro definitions

Return type

str

names

Stable de-duplication of header names in the list without extensions

>>> h = HeaderList(['/dir1/a.h', '/dir2/b.h', '/dir3/a.h'])
>>> h.names
['a', 'b']
Returns

A list of files without extensions

Return type

list of strings

class llnl.util.filesystem.LibraryList(files)

Bases: llnl.util.filesystem.FileList

Sequence of absolute paths to libraries

Provides a few convenience methods to manipulate library paths and get commonly used compiler flags or names

ld_flags

Search flags + link flags

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.ld_flags
'-L/dir1 -L/dir2 -la -lb'
Returns

A joined list of search flags and link flags

Return type

str

libraries

Stable de-duplication of library files.

Returns

A list of library files

Return type

list of strings

Link flags for the libraries

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.link_flags
'-la -lb'
Returns

A joined list of link flags

Return type

str

names

Stable de-duplication of library names in the list

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir3/liba.so'])
>>> l.names
['a', 'b']
Returns

A list of library names

Return type

list of strings

search_flags

Search flags for the libraries

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.search_flags
'-L/dir1 -L/dir2'
Returns

A joined list of search flags

Return type

str

llnl.util.filesystem.ancestor(dir, n=1)

Get the nth ancestor of a directory.

llnl.util.filesystem.can_access(file_name)

True if we have read/write access to the file.

llnl.util.filesystem.change_sed_delimiter(old_delim, new_delim, *filenames)

Find all sed search/replace commands and change the delimiter.

e.g., if the file contains seds that look like 's///', you can call change_sed_delimiter('/', '@', file) to change the delimiter to '@'.

Note that this routine will fail if the delimiter is ' or ". Handling those is left for future work.

Parameters
  • old_delim (str) – The delimiter to search for

  • new_delim (str) – The delimiter to replace with

  • *filenames – One or more files to search and replace

llnl.util.filesystem.copy_mode(src, dest)

Set the mode of dest to that of src unless it is a link.

llnl.util.filesystem.filter_file(regex, repl, *filenames, **kwargs)

Like sed, but uses python regular expressions.

Filters every line of each file through regex and replaces the file with a filtered version. Preserves mode of filtered files.

As with re.sub, repl can be either a string or a callable. If it is a callable, it is passed the match object and should return a suitable replacement string. If it is a string, it can contain \1, \2, etc. to represent back-substitution as sed would allow.

Parameters
  • regex (str) – The regular expression to search for

  • repl (str) – The string to replace matches with

  • *filenames – One or more files to search and replace

Keyword Arguments
  • string (bool) – Treat regex as a plain string. Default it False

  • backup (bool) – Make backup file(s) suffixed with ~. Default is True

  • ignore_absent (bool) – Ignore any files that don’t exist. Default is False

llnl.util.filesystem.find(root, files, recursive=True)

Search for files starting from the root directory.

Like GNU/BSD find but written entirely in Python.

Examples:

$ find /usr -name python

is equivalent to:

>>> find('/usr', 'python')
$ find /usr/local/bin -maxdepth 1 -name python

is equivalent to:

>>> find('/usr/local/bin', 'python', recursive=False)

Accepts any glob characters accepted by fnmatch:

Pattern

Meaning

matches everything

?

matches any single character

[seq]

matches any character in seq

[!seq]

matches any character not in seq

Parameters
  • root (str) – The root directory to start searching from

  • files (str or collections.Sequence) – Library name(s) to search for

  • recurse (bool, optional) – if False search only root folder, if True descends top-down from the root. Defaults to True.

Returns

The files that have been found

Return type

list of strings

llnl.util.filesystem.find_headers(headers, root, recursive=False)

Returns an iterable object containing a list of full paths to headers if found.

Accepts any glob characters accepted by fnmatch:

Pattern

Meaning

matches everything

?

matches any single character

[seq]

matches any character in seq

[!seq]

matches any character not in seq

Parameters
  • headers (str or list of str) – Header name(s) to search for

  • root (str) – The root directory to start searching from

  • recursive (bool, optional) – if False search only root folder, if True descends top-down from the root. Defaults to False.

Returns

The headers that have been found

Return type

HeaderList

llnl.util.filesystem.find_all_headers(root)

Convenience function that returns the list of all headers found in the directory passed as argument.

Parameters

root (path) – directory where to look recursively for header files

Returns

List of all headers found in root and subdirectories.

llnl.util.filesystem.find_libraries(libraries, root, shared=True, recursive=False)

Returns an iterable of full paths to libraries found in a root dir.

Accepts any glob characters accepted by fnmatch:

Pattern

Meaning

matches everything

?

matches any single character

[seq]

matches any character in seq

[!seq]

matches any character not in seq

Parameters
  • libraries (str or list of str) – Library name(s) to search for

  • root (str) – The root directory to start searching from

  • shared (bool, optional) – if True searches for shared libraries, otherwise for static. Defaults to True.

  • recursive (bool, optional) – if False search only root folder, if True descends top-down from the root. Defaults to False.

Returns

The libraries that have been found

Return type

LibraryList

llnl.util.filesystem.find_system_libraries(libraries, shared=True)

Searches the usual system library locations for libraries.

Search order is as follows:

  1. /lib64

  2. /lib

  3. /usr/lib64

  4. /usr/lib

  5. /usr/local/lib64

  6. /usr/local/lib

Accepts any glob characters accepted by fnmatch:

Pattern

Meaning

matches everything

?

matches any single character

[seq]

matches any character in seq

[!seq]

matches any character not in seq

Parameters
  • libraries (str or list of str) – Library name(s) to search for

  • shared (bool, optional) – if True searches for shared libraries, otherwise for static. Defaults to True.

Returns

The libraries that have been found

Return type

LibraryList

llnl.util.filesystem.fix_darwin_install_name(path)

Fix install name of dynamic libraries on Darwin to have full path.

There are two parts of this task:

  1. Use install_name('-id', ...) to change install name of a single lib

  2. Use install_name('-change', ...) to change the cross linking between libs. The function assumes that all libraries are in one folder and currently won’t follow subfolders.

Parameters

path (str) – directory in which .dylib files are located

llnl.util.filesystem.force_remove(*paths)

Remove files without printing errors. Like rm -f, does NOT remove directories.

llnl.util.filesystem.copy(src, dest, _permissions=False)

Copies the file src to the file or directory dest.

If dest specifies a directory, the file will be copied into dest using the base filename from src.

Parameters
  • src (str) – the file to copy

  • dest (str) – the destination file or directory

  • _permissions (bool) – for internal use only

llnl.util.filesystem.install(src, dest)

Installs the file src to the file or directory dest.

Same as copy() with the addition of setting proper permissions on the installed file.

Parameters
  • src (str) – the file to install

  • dest (str) – the destination file or directory

llnl.util.filesystem.copy_tree(src, dest, symlinks=True, ignore=None, _permissions=False)

Recursively copy an entire directory tree rooted at src.

If the destination directory dest does not already exist, it will be created as well as missing parent directories.

If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree and the metadata of the original links will be copied as far as the platform allows; if false, the contents and metadata of the linked files are copied to the new tree.

If ignore is set, then each path relative to src will be passed to this function; the function returns whether that path should be skipped.

Parameters
  • src (str) – the directory to copy

  • dest (str) – the destination directory

  • symlinks (bool) – whether or not to preserve symlinks

  • ignore (function) – function indicating which files to ignore

  • _permissions (bool) – for internal use only

llnl.util.filesystem.install_tree(src, dest, symlinks=True, ignore=None)

Recursively install an entire directory tree rooted at src.

Same as copy_tree() with the addition of setting proper permissions on the installed files and directories.

Parameters
  • src (str) – the directory to install

  • dest (str) – the destination directory

  • symlinks (bool) – whether or not to preserve symlinks

  • ignore (function) – function indicating which files to ignore

llnl.util.filesystem.is_exe(path)

True if path is an executable file.

llnl.util.filesystem.join_path(prefix, *args)
llnl.util.filesystem.mkdirp(*paths, **kwargs)

Creates a directory, as well as parent directories if needed.

Parameters

paths (str) – paths to create with mkdirp

Keyword Aguments:
mode (permission bits or None, optional): optional permissions to set

on the created directory – use OS default if not provided

group (group name or None, optional): optional group for permissions of

final created directory – use OS default if not provided. Only used if world write permissions are not set

default_perms (‘parents’ or ‘args’, optional): The default permissions

that are set for directories that are not themselves an argument for mkdirp. ‘parents’ means intermediate directories get the permissions of their direct parent directory, ‘args’ means intermediate get the same permissions specified in the arguments to mkdirp – default value is ‘args’

Recursively removes any dead link that is present in root.

Parameters

root (str) – path where to search for dead links

Removes the argument if it is a dead link.

Parameters

path (str) – The potential dead link

llnl.util.filesystem.remove_linked_tree(path)

Removes a directory and its contents.

If the directory is a symlink, follows the link and removes the real directory before removing the link.

Parameters

path (str) – Directory to be removed

llnl.util.filesystem.set_executable(path)
llnl.util.filesystem.set_install_permissions(path)

Set appropriate permissions on the installed file.

llnl.util.filesystem.touch(path)

Creates an empty file at the specified path.

llnl.util.filesystem.touchp(path)

Like touch, but creates any parent directories needed for the file.

llnl.util.filesystem.traverse_tree(source_root, dest_root, rel_path='', **kwargs)

Traverse two filesystem trees simultaneously.

Walks the LinkTree directory in pre or post order. Yields each file in the source directory with a matching path from the dest directory, along with whether the file is a directory. e.g., for this tree:

root/
  a/
    file1
    file2
  b/
    file3

When called on dest, this yields:

('root',         'dest')
('root/a',       'dest/a')
('root/a/file1', 'dest/a/file1')
('root/a/file2', 'dest/a/file2')
('root/b',       'dest/b')
('root/b/file3', 'dest/b/file3')
Keyword Arguments
  • order (str) – Whether to do pre- or post-order traversal. Accepted values are ‘pre’ and ‘post’

  • ignore (function) – function indicating which files to ignore

  • follow_nonexisting (bool) – Whether to descend into directories in src that do not exit in dest. Default is True

  • follow_links (bool) – Whether to descend into symlinks in src

llnl.util.filesystem.unset_executable_mode(path)
llnl.util.filesystem.working_dir(dirname, **kwargs)

llnl.util.lang module

class llnl.util.lang.HashableMap

Bases: collections.abc.MutableMapping

This is a hashable, comparable dictionary. Hash is performed on a tuple of the values in the dictionary.

copy()

Type-agnostic clone method. Preserves subclass type.

class llnl.util.lang.LazyReference(ref_function)

Bases: object

Lazily evaluated reference to part of a singleton.

class llnl.util.lang.ObjectWrapper(wrapped_object)

Bases: object

Base class that wraps an object. Derived classes can add new behavior while staying undercover.

This class is modeled after the stackoverflow answer: * http://stackoverflow.com/a/1445289/771663

exception llnl.util.lang.RequiredAttributeError(message)

Bases: ValueError

class llnl.util.lang.Singleton(factory)

Bases: object

Simple wrapper for lazily initialized singleton objects.

instance
llnl.util.lang.attr_required(obj, attr_name)

Ensure that a class has a required attribute.

llnl.util.lang.attr_setdefault(obj, name, value)

Like dict.setdefault, but for objects.

llnl.util.lang.caller_locals()

This will return the locals of the parent of the caller. This allows a function to insert variables into its caller’s scope. Yes, this is some black magic, and yes it’s useful for implementing things like depends_on and provides.

llnl.util.lang.check_kwargs(kwargs, fun)

Helper for making functions with kwargs. Checks whether the kwargs are empty after all of them have been popped off. If they’re not, raises an error describing which kwargs are invalid.

Example:

def foo(self, **kwargs):
    x = kwargs.pop('x', None)
    y = kwargs.pop('y', None)
    z = kwargs.pop('z', None)
    check_kwargs(kwargs, self.foo)

# This raises a TypeError:
foo(w='bad kwarg')
class llnl.util.lang.classproperty

Bases: property

classproperty decorator: like property but for classmethods.

llnl.util.lang.dedupe(sequence)

Yields a stable de-duplication of an hashable sequence

Parameters

sequence – hashable sequence to be de-duplicated

Returns

stable de-duplication of the sequence

llnl.util.lang.get_calling_module_name()

Make sure that the caller is a class definition, and return the enclosing module’s name.

llnl.util.lang.has_method(cls, name)
llnl.util.lang.in_function(function_name)

True if the caller was called from some function with the supplied Name, False otherwise.

llnl.util.lang.index_by(objects, *funcs)

Create a hierarchy of dictionaries by splitting the supplied set of objects on unique values of the supplied functions. Values are used as keys. For example, suppose you have four objects with attributes that look like this:

a = Spec(name="boost",    compiler="gcc",   arch="bgqos_0")
b = Spec(name="mrnet",    compiler="intel", arch="chaos_5_x86_64_ib")
c = Spec(name="libelf",   compiler="xlc",   arch="bgqos_0")
d = Spec(name="libdwarf", compiler="intel", arch="chaos_5_x86_64_ib")

list_of_specs = [a,b,c,d]
index1 = index_by(list_of_specs, lambda s: s.arch,
                  lambda s: s.compiler)
index2 = index_by(list_of_specs, lambda s: s.compiler)

index1 now has two levels of dicts, with lists at the leaves, like this:

{ 'bgqos_0'           : { 'gcc' : [a], 'xlc' : [c] },
  'chaos_5_x86_64_ib' : { 'intel' : [b, d] }
}

And index2 is a single level dictionary of lists that looks like this:

{ 'gcc'    : [a],
  'intel'  : [b,d],
  'xlc'    : [c]
}

If any elemnts in funcs is a string, it is treated as the name of an attribute, and acts like getattr(object, name). So shorthand for the above two indexes would be:

index1 = index_by(list_of_specs, 'arch', 'compiler')
index2 = index_by(list_of_specs, 'compiler')

You can also index by tuples by passing tuples:

index1 = index_by(list_of_specs, ('arch', 'compiler'))

Keys in the resulting dict will look like (‘gcc’, ‘bgqos_0’).

llnl.util.lang.key_ordering(cls)

Decorates a class with extra methods that implement rich comparison operations and __hash__. The decorator assumes that the class implements a function called _cmp_key(). The rich comparison operations will compare objects using this key, and the __hash__ function will return the hash of this key.

If a class already has __eq__, __ne__, __lt__, __le__, __gt__, or __ge__ defined, this decorator will overwrite them.

Raises

TypeError – If the class does not have a _cmp_key method

llnl.util.lang.list_modules(directory, **kwargs)

Lists all of the modules, excluding __init__.py, in a particular directory. Listed packages have no particular order.

llnl.util.lang.load_module_from_file(module_name, module_path)

Loads a python module from the path of the corresponding file.

Parameters
  • module_name (str) – namespace where the python module will be loaded, e.g. foo.bar

  • module_path (str) – path of the python file containing the module

Returns

A valid module object

Raises
  • ImportError – when the module can’t be loaded

  • FileNotFoundError – when module_path doesn’t exist

llnl.util.lang.match_predicate(*args)

Utility function for making string matching predicates.

Each arg can be a: * regex * list or tuple of regexes * predicate that takes a string.

This returns a predicate that is true if: * any arg regex matches * any regex in a list or tuple of regexes matches. * any predicate in args matches.

llnl.util.lang.memoized(func)

Decorator that caches the results of a function, storing them in an attribute of that function.

llnl.util.lang.pretty_date(time, now=None)

Convert a datetime or timestamp to a pretty, relative date.

Parameters
  • time (datetime or int) – date to print prettily

  • now (datetime) – dateimte for ‘now’, i.e. the date the pretty date is relative to (default is datetime.now())

Returns

pretty string like ‘an hour ago’, ‘Yesterday’,

‘3 months ago’, ‘just now’, etc.

Return type

(str)

Adapted from https://stackoverflow.com/questions/1551382.

llnl.util.lang.pretty_string_to_date(date_str, now=None)

Parses a string representing a date and returns a datetime object.

Parameters

date_str (str) – string representing a date. This string might be in different format (like YYYY, YYYY-MM, YYYY-MM-DD, YYYY-MM-DD HH:MM, YYYY-MM-DD HH:MM:SS) or be a pretty date (like yesterday or two months ago)

Returns

datetime object corresponding to date_str

Return type

(datetime)

llnl.util.lang.union_dicts(*dicts)

Use update() to combine all dicts into one.

This builds a new dictionary, into which we update() each element of dicts in order. Items from later dictionaries will override items from earlier dictionaries.

Parameters

dicts (list) – list of dictionaries

Return: (dict): a merged dictionary containing combined keys and

values from dicts.

llnl.util.lock module

class llnl.util.lock.Lock(path, start=0, length=0, debug=False, default_timeout=None)

Bases: object

This is an implementation of a filesystem lock using Python’s lockf.

In Python, lockf actually calls fcntl, so this should work with any filesystem implementation that supports locking through the fcntl calls. This includes distributed filesystems like Lustre (when flock is enabled) and recent NFS versions.

Note that this is for managing contention over resources between processes and not for managing contention between threads in a process: the functions of this object are not thread-safe. A process also must not maintain multiple locks on the same file.

acquire_read(timeout=None)

Acquires a recursive, shared lock for reading.

Read and write locks can be acquired and released in arbitrary order, but the POSIX lock is held until all local read and write locks are released.

Returns True if it is the first acquire and actually acquires the POSIX lock, False if it is a nested transaction.

acquire_write(timeout=None)

Acquires a recursive, exclusive lock for writing.

Read and write locks can be acquired and released in arbitrary order, but the POSIX lock is held until all local read and write locks are released.

Returns True if it is the first acquire and actually acquires the POSIX lock, False if it is a nested transaction.

release_read()

Releases a read lock.

Returns True if the last recursive lock was released, False if there are still outstanding locks.

Does limited correctness checking: if a read lock is released when none are held, this will raise an assertion error.

release_write()

Releases a write lock.

Returns True if the last recursive lock was released, False if there are still outstanding locks.

Does limited correctness checking: if a read lock is released when none are held, this will raise an assertion error.

class llnl.util.lock.LockTransaction(lock, acquire_fn=None, release_fn=None, timeout=None)

Bases: object

Simple nested transaction context manager that uses a file lock.

This class can trigger actions when the lock is acquired for the first time and released for the last.

If the acquire_fn returns a value, it is used as the return value for __enter__, allowing it to be passed as the as argument of a with statement.

If acquire_fn returns a context manager, its __enter__ function will be called in __enter__ after acquire_fn, and its __exit__ funciton will be called before release_fn in __exit__, allowing you to nest a context manager to be used along with the lock.

Timeout for lock is customizable.

class llnl.util.lock.WriteTransaction(lock, acquire_fn=None, release_fn=None, timeout=None)

Bases: llnl.util.lock.LockTransaction

LockTransaction context manager that does a write and releases it.

class llnl.util.lock.ReadTransaction(lock, acquire_fn=None, release_fn=None, timeout=None)

Bases: llnl.util.lock.LockTransaction

LockTransaction context manager that does a read and releases it.

exception llnl.util.lock.LockError

Bases: Exception

Raised for any errors related to locks.

exception llnl.util.lock.LockTimeoutError

Bases: llnl.util.lock.LockError

Raised when an attempt to acquire a lock times out.

exception llnl.util.lock.LockPermissionError

Bases: llnl.util.lock.LockError

Raised when there are permission issues with a lock.

exception llnl.util.lock.LockROFileError(path)

Bases: llnl.util.lock.LockPermissionError

Tried to take an exclusive lock on a read-only file.

exception llnl.util.lock.CantCreateLockError(path)

Bases: llnl.util.lock.LockPermissionError

Attempt to create a lock in an unwritable location.

llnl.util.multiproc module

This implements a parallel map operation but it can accept more values than multiprocessing.Pool.apply() can. For example, apply() will fail to pickle functions if they’re passed indirectly as parameters.

class llnl.util.multiproc.Barrier(n, timeout=None)

Bases: object

Simple reusable semaphore barrier.

Python 2.6 doesn’t have multiprocessing barriers so we implement this.

See http://greenteapress.com/semaphores/downey08semaphores.pdf, p. 41.

wait()

Module contents