llnl.util package

Submodules

llnl.util.argparsewriter module

class llnl.util.argparsewriter.ArgparseRstWriter(out=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>, rst_levels=['=', '-', '^', '~', ':', '`'], strip_root_prog=True)

Bases: llnl.util.argparsewriter.ArgparseWriter

Write argparse output as rst sections.

begin_command(prog)
begin_optionals()
begin_positionals()
begin_subcommands(subcommands)
description(description)
line(string='')
optional(opts, help)
positional(name, help)
usage(usage)
class llnl.util.argparsewriter.ArgparseWriter

Bases: object

Analyzes an argparse ArgumentParser for easy generation of help.

begin_command(prog)
begin_optionals()
begin_positionals()
begin_subcommands(subcommands)
description(description)
end_command(prog)
end_optionals()
end_positionals()
end_subcommands(subcommands)
optional(option, help)
positional(name, help)
usage(usage)
write(parser, root=True)

Write out details about an ArgumentParser.

Parameters:
  • parser (ArgumentParser) – an argparse parser
  • root (bool or int) – if bool, whether to include the root parser; or 1 to flatten the root parser with first-level subcommands

llnl.util.filesystem module

class llnl.util.filesystem.FileFilter(*filenames)

Bases: object

Convenience class for calling filter_file a lot.

filter(regex, repl, **kwargs)
class llnl.util.filesystem.FileList(files)

Bases: collections.abc.Sequence

Sequence of absolute paths to files.

Provides a few convenience methods to manipulate file paths.

basenames

Stable de-duplication of the base-names in the list

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir3/liba.a'])
>>> l.basenames
['liba.a', 'libb.a']
>>> h = HeaderList(['/dir1/a.h', '/dir2/b.h', '/dir3/a.h'])
>>> h.basenames
['a.h', 'b.h']
Returns:A list of base-names
Return type:list of strings
directories

Stable de-duplication of the directories where the files reside.

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/libc.a'])
>>> l.directories
['/dir1', '/dir2']
>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.directories
['/dir1', '/dir2']
Returns:A list of directories
Return type:list of strings
joined(separator=' ')
class llnl.util.filesystem.HeaderList(files)

Bases: llnl.util.filesystem.FileList

Sequence of absolute paths to headers.

Provides a few convenience methods to manipulate header paths and get commonly used compiler flags or names.

add_macro(macro)

Add a macro definition

Parameters:macro (str) – The macro to add
cpp_flags

Include flags + macro definitions

>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.cpp_flags
'-I/dir1 -I/dir2'
>>> h.add_macro('-DBOOST_DYN_LINK')
>>> h.cpp_flags
'-I/dir1 -I/dir2 -DBOOST_DYN_LINK'
Returns:A joined list of include flags and macro definitions
Return type:str
headers

Stable de-duplication of the headers.

Returns:A list of header files
Return type:list of strings
include_flags

Include flags

>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.include_flags
'-I/dir1 -I/dir2'
Returns:A joined list of include flags
Return type:str
macro_definitions

Macro definitions

>>> h = HeaderList(['/dir1/a.h', '/dir1/b.h', '/dir2/c.h'])
>>> h.add_macro('-DBOOST_LIB_NAME=boost_regex')
>>> h.add_macro('-DBOOST_DYN_LINK')
>>> h.macro_definitions
'-DBOOST_LIB_NAME=boost_regex -DBOOST_DYN_LINK'
Returns:A joined list of macro definitions
Return type:str
names

Stable de-duplication of header names in the list without extensions

>>> h = HeaderList(['/dir1/a.h', '/dir2/b.h', '/dir3/a.h'])
>>> h.names
['a', 'b']
Returns:A list of files without extensions
Return type:list of strings
class llnl.util.filesystem.LibraryList(files)

Bases: llnl.util.filesystem.FileList

Sequence of absolute paths to libraries

Provides a few convenience methods to manipulate library paths and get commonly used compiler flags or names

ld_flags

Search flags + link flags

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.ld_flags
'-L/dir1 -L/dir2 -la -lb'
Returns:A joined list of search flags and link flags
Return type:str
libraries

Stable de-duplication of library files.

Returns:A list of library files
Return type:list of strings

Link flags for the libraries

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.link_flags
'-la -lb'
Returns:A joined list of link flags
Return type:str
names

Stable de-duplication of library names in the list

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir3/liba.so'])
>>> l.names
['a', 'b']
Returns:A list of library names
Return type:list of strings
search_flags

Search flags for the libraries

>>> l = LibraryList(['/dir1/liba.a', '/dir2/libb.a', '/dir1/liba.so'])
>>> l.search_flags
'-L/dir1 -L/dir2'
Returns:A joined list of search flags
Return type:str
llnl.util.filesystem.ancestor(dir, n=1)

Get the nth ancestor of a directory.

llnl.util.filesystem.can_access(file_name)

True if we have read/write access to the file.

llnl.util.filesystem.change_sed_delimiter(old_delim, new_delim, *filenames)

Find all sed search/replace commands and change the delimiter.

e.g., if the file contains seds that look like 's///', you can call change_sed_delimiter('/', '@', file) to change the delimiter to '@'.

Note that this routine will fail if the delimiter is ' or ". Handling those is left for future work.

Parameters:
  • old_delim (str) – The delimiter to search for
  • new_delim (str) – The delimiter to replace with
  • *filenames – One or more files to search and replace
llnl.util.filesystem.copy_mode(src, dest)

Set the mode of dest to that of src unless it is a link.

llnl.util.filesystem.filter_file(regex, repl, *filenames, **kwargs)

Like sed, but uses python regular expressions.

Filters every line of each file through regex and replaces the file with a filtered version. Preserves mode of filtered files.

As with re.sub, repl can be either a string or a callable. If it is a callable, it is passed the match object and should return a suitable replacement string. If it is a string, it can contain \1, \2, etc. to represent back-substitution as sed would allow.

Parameters:
  • regex (str) – The regular expression to search for
  • repl (str) – The string to replace matches with
  • *filenames – One or more files to search and replace
Keyword Arguments:
 
  • string (bool) – Treat regex as a plain string. Default it False
  • backup (bool) – Make backup file(s) suffixed with ~. Default is True
  • ignore_absent (bool) – Ignore any files that don’t exist. Default is False
llnl.util.filesystem.find(root, files, recursive=True)

Search for files starting from the root directory.

Like GNU/BSD find but written entirely in Python.

Examples:

$ find /usr -name python

is equivalent to:

>>> find('/usr', 'python')
$ find /usr/local/bin -maxdepth 1 -name python

is equivalent to:

>>> find('/usr/local/bin', 'python', recursive=False)

Accepts any glob characters accepted by fnmatch:

Pattern Meaning
matches everything
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
Parameters:
  • root (str) – The root directory to start searching from
  • files (str or collections.Sequence) – Library name(s) to search for
  • recurse (bool, optional) – if False search only root folder, if True descends top-down from the root. Defaults to True.
Returns:

The files that have been found

Return type:

list of strings

llnl.util.filesystem.find_headers(headers, root, recursive=False)

Returns an iterable object containing a list of full paths to headers if found.

Accepts any glob characters accepted by fnmatch:

Pattern Meaning
matches everything
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
Parameters:
  • headers (str or list of str) – Header name(s) to search for
  • root (str) – The root directory to start searching from
  • recursive (bool, optional) – if False search only root folder, if True descends top-down from the root. Defaults to False.
Returns:

The headers that have been found

Return type:

HeaderList

llnl.util.filesystem.find_libraries(libraries, root, shared=True, recursive=False)

Returns an iterable of full paths to libraries found in a root dir.

Accepts any glob characters accepted by fnmatch:

Pattern Meaning
matches everything
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
Parameters:
  • libraries (str or list of str) – Library name(s) to search for
  • root (str) – The root directory to start searching from
  • shared (bool, optional) – if True searches for shared libraries, otherwise for static. Defaults to True.
  • recursive (bool, optional) – if False search only root folder, if True descends top-down from the root. Defaults to False.
Returns:

The libraries that have been found

Return type:

LibraryList

llnl.util.filesystem.find_system_libraries(libraries, shared=True)

Searches the usual system library locations for libraries.

Search order is as follows:

  1. /lib64
  2. /lib
  3. /usr/lib64
  4. /usr/lib
  5. /usr/local/lib64
  6. /usr/local/lib

Accepts any glob characters accepted by fnmatch:

Pattern Meaning
matches everything
? matches any single character
[seq] matches any character in seq
[!seq] matches any character not in seq
Parameters:
  • libraries (str or list of str) – Library name(s) to search for
  • shared (bool, optional) – if True searches for shared libraries, otherwise for static. Defaults to True.
Returns:

The libraries that have been found

Return type:

LibraryList

llnl.util.filesystem.fix_darwin_install_name(path)

Fix install name of dynamic libraries on Darwin to have full path.

There are two parts of this task:

  1. Use install_name('-id', ...) to change install name of a single lib
  2. Use install_name('-change', ...) to change the cross linking between libs. The function assumes that all libraries are in one folder and currently won’t follow subfolders.
Parameters:path (str) – directory in which .dylib files are located
llnl.util.filesystem.force_remove(*paths)

Remove files without printing errors. Like rm -f, does NOT remove directories.

llnl.util.filesystem.copy(src, dest, _permissions=False)

Copies the file src to the file or directory dest.

If dest specifies a directory, the file will be copied into dest using the base filename from src.

Parameters:
  • src (str) – the file to copy
  • dest (str) – the destination file or directory
  • _permissions (bool) – for internal use only
llnl.util.filesystem.install(src, dest)

Installs the file src to the file or directory dest.

Same as copy() with the addition of setting proper permissions on the installed file.

Parameters:
  • src (str) – the file to install
  • dest (str) – the destination file or directory
llnl.util.filesystem.copy_tree(src, dest, symlinks=True, ignore=None, _permissions=False)

Recursively copy an entire directory tree rooted at src.

If the destination directory dest does not already exist, it will be created as well as missing parent directories.

If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree and the metadata of the original links will be copied as far as the platform allows; if false, the contents and metadata of the linked files are copied to the new tree.

If ignore is set, then each path relative to src will be passed to this function; the function returns whether that path should be skipped.

Parameters:
  • src (str) – the directory to copy
  • dest (str) – the destination directory
  • symlinks (bool) – whether or not to preserve symlinks
  • ignore (function) – function indicating which files to ignore
  • _permissions (bool) – for internal use only
llnl.util.filesystem.install_tree(src, dest, symlinks=True, ignore=None)

Recursively install an entire directory tree rooted at src.

Same as copy_tree() with the addition of setting proper permissions on the installed files and directories.

Parameters:
  • src (str) – the directory to install
  • dest (str) – the destination directory
  • symlinks (bool) – whether or not to preserve symlinks
  • ignore (function) – function indicating which files to ignore
llnl.util.filesystem.is_exe(path)

True if path is an executable file.

llnl.util.filesystem.join_path(prefix, *args)
llnl.util.filesystem.mkdirp(*paths, **kwargs)

Creates a directory, as well as parent directories if needed.

Removes any dead link that is present in root.

Parameters:root (str) – path where to search for dead links

Removes the argument if it is a dead link.

Parameters:path (str) – The potential dead link
llnl.util.filesystem.remove_linked_tree(path)

Removes a directory and its contents.

If the directory is a symlink, follows the link and removes the real directory before removing the link.

Parameters:path (str) – Directory to be removed
llnl.util.filesystem.set_executable(path)
llnl.util.filesystem.set_install_permissions(path)

Set appropriate permissions on the installed file.

llnl.util.filesystem.touch(path)

Creates an empty file at the specified path.

llnl.util.filesystem.touchp(path)

Like touch, but creates any parent directories needed for the file.

llnl.util.filesystem.traverse_tree(source_root, dest_root, rel_path='', **kwargs)

Traverse two filesystem trees simultaneously.

Walks the LinkTree directory in pre or post order. Yields each file in the source directory with a matching path from the dest directory, along with whether the file is a directory. e.g., for this tree:

root/
  a/
    file1
    file2
  b/
    file3

When called on dest, this yields:

('root',         'dest')
('root/a',       'dest/a')
('root/a/file1', 'dest/a/file1')
('root/a/file2', 'dest/a/file2')
('root/b',       'dest/b')
('root/b/file3', 'dest/b/file3')
Keyword Arguments:
 
  • order (str) – Whether to do pre- or post-order traversal. Accepted values are ‘pre’ and ‘post’
  • ignore (function) – function indicating which files to ignore
  • follow_nonexisting (bool) – Whether to descend into directories in src that do not exit in dest. Default is True
  • follow_links (bool) – Whether to descend into symlinks in src
llnl.util.filesystem.unset_executable_mode(path)
llnl.util.filesystem.working_dir(dirname, **kwargs)

llnl.util.lang module

class llnl.util.lang.HashableMap

Bases: collections.abc.MutableMapping

This is a hashable, comparable dictionary. Hash is performed on a tuple of the values in the dictionary.

copy()

Type-agnostic clone method. Preserves subclass type.

class llnl.util.lang.LazyReference(ref_function)

Bases: object

Lazily evaluated reference to part of a singleton.

class llnl.util.lang.ObjectWrapper(wrapped_object)

Bases: object

Base class that wraps an object. Derived classes can add new behavior while staying undercover.

This class is modeled after the stackoverflow answer: * http://stackoverflow.com/a/1445289/771663

exception llnl.util.lang.RequiredAttributeError(message)

Bases: ValueError

class llnl.util.lang.Singleton(factory)

Bases: object

Simple wrapper for lazily initialized singleton objects.

instance
llnl.util.lang.attr_required(obj, attr_name)

Ensure that a class has a required attribute.

llnl.util.lang.attr_setdefault(obj, name, value)

Like dict.setdefault, but for objects.

llnl.util.lang.caller_locals()

This will return the locals of the parent of the caller. This allows a function to insert variables into its caller’s scope. Yes, this is some black magic, and yes it’s useful for implementing things like depends_on and provides.

llnl.util.lang.check_kwargs(kwargs, fun)

Helper for making functions with kwargs. Checks whether the kwargs are empty after all of them have been popped off. If they’re not, raises an error describing which kwargs are invalid.

Example:

def foo(self, **kwargs):
    x = kwargs.pop('x', None)
    y = kwargs.pop('y', None)
    z = kwargs.pop('z', None)
    check_kwargs(kwargs, self.foo)

# This raises a TypeError:
foo(w='bad kwarg')
class llnl.util.lang.classproperty

Bases: property

classproperty decorator: like property but for classmethods.

llnl.util.lang.dedupe(sequence)

Yields a stable de-duplication of an hashable sequence

Parameters:sequence – hashable sequence to be de-duplicated
Returns:stable de-duplication of the sequence
llnl.util.lang.get_calling_module_name()

Make sure that the caller is a class definition, and return the enclosing module’s name.

llnl.util.lang.has_method(cls, name)
llnl.util.lang.in_function(function_name)

True if the caller was called from some function with the supplied Name, False otherwise.

llnl.util.lang.index_by(objects, *funcs)

Create a hierarchy of dictionaries by splitting the supplied set of objects on unique values of the supplied functions. Values are used as keys. For example, suppose you have four objects with attributes that look like this:

a = Spec(name="boost",    compiler="gcc",   arch="bgqos_0")
b = Spec(name="mrnet",    compiler="intel", arch="chaos_5_x86_64_ib")
c = Spec(name="libelf",   compiler="xlc",   arch="bgqos_0")
d = Spec(name="libdwarf", compiler="intel", arch="chaos_5_x86_64_ib")

list_of_specs = [a,b,c,d]
index1 = index_by(list_of_specs, lambda s: s.arch,
                  lambda s: s.compiler)
index2 = index_by(list_of_specs, lambda s: s.compiler)

index1 now has two levels of dicts, with lists at the leaves, like this:

{ 'bgqos_0'           : { 'gcc' : [a], 'xlc' : [c] },
  'chaos_5_x86_64_ib' : { 'intel' : [b, d] }
}

And index2 is a single level dictionary of lists that looks like this:

{ 'gcc'    : [a],
  'intel'  : [b,d],
  'xlc'    : [c]
}

If any elemnts in funcs is a string, it is treated as the name of an attribute, and acts like getattr(object, name). So shorthand for the above two indexes would be:

index1 = index_by(list_of_specs, 'arch', 'compiler')
index2 = index_by(list_of_specs, 'compiler')

You can also index by tuples by passing tuples:

index1 = index_by(list_of_specs, ('arch', 'compiler'))

Keys in the resulting dict will look like (‘gcc’, ‘bgqos_0’).

llnl.util.lang.key_ordering(cls)

Decorates a class with extra methods that implement rich comparison operations and __hash__. The decorator assumes that the class implements a function called _cmp_key(). The rich comparison operations will compare objects using this key, and the __hash__ function will return the hash of this key.

If a class already has __eq__, __ne__, __lt__, __le__, __gt__, or __ge__ defined, this decorator will overwrite them.

Raises:TypeError – If the class does not have a _cmp_key method
llnl.util.lang.list_modules(directory, **kwargs)

Lists all of the modules, excluding __init__.py, in a particular directory. Listed packages have no particular order.

llnl.util.lang.match_predicate(*args)

Utility function for making string matching predicates.

Each arg can be a: * regex * list or tuple of regexes * predicate that takes a string.

This returns a predicate that is true if: * any arg regex matches * any regex in a list or tuple of regexes matches. * any predicate in args matches.

class llnl.util.lang.memoized(func)

Bases: object

Decorator that caches the results of a function, storing them in an attribute of that function.

clear()

Expunge cache so that self.func will be called again.

llnl.util.lang.pretty_date(time, now=None)

Convert a datetime or timestamp to a pretty, relative date.

Parameters:
  • time (datetime or int) – date to print prettily
  • now (datetime) – dateimte for ‘now’, i.e. the date the pretty date is relative to (default is datetime.now())
Returns:

pretty string like ‘an hour ago’, ‘Yesterday’,

‘3 months ago’, ‘just now’, etc.

Return type:

(str)

Adapted from https://stackoverflow.com/questions/1551382.

llnl.util.lang.pretty_string_to_date(date_str, now=None)

Parses a string representing a date and returns a datetime object.

Parameters:date_str (str) – string representing a date. This string might be in different format (like YYYY, YYYY-MM, YYYY-MM-DD) or be a pretty date (like yesterday or two months ago)
Returns:datetime object corresponding to date_str
Return type:(datetime)

llnl.util.lock module

class llnl.util.lock.Lock(path, start=0, length=0, debug=False, default_timeout=None)

Bases: object

This is an implementation of a filesystem lock using Python’s lockf.

In Python, lockf actually calls fcntl, so this should work with any filesystem implementation that supports locking through the fcntl calls. This includes distributed filesystems like Lustre (when flock is enabled) and recent NFS versions.

Note that this is for managing contention over resources between processes and not for managing contention between threads in a process: the functions of this object are not thread-safe. A process also must not maintain multiple locks on the same file.

acquire_read(timeout=None)

Acquires a recursive, shared lock for reading.

Read and write locks can be acquired and released in arbitrary order, but the POSIX lock is held until all local read and write locks are released.

Returns True if it is the first acquire and actually acquires the POSIX lock, False if it is a nested transaction.

acquire_write(timeout=None)

Acquires a recursive, exclusive lock for writing.

Read and write locks can be acquired and released in arbitrary order, but the POSIX lock is held until all local read and write locks are released.

Returns True if it is the first acquire and actually acquires the POSIX lock, False if it is a nested transaction.

release_read()

Releases a read lock.

Returns True if the last recursive lock was released, False if there are still outstanding locks.

Does limited correctness checking: if a read lock is released when none are held, this will raise an assertion error.

release_write()

Releases a write lock.

Returns True if the last recursive lock was released, False if there are still outstanding locks.

Does limited correctness checking: if a read lock is released when none are held, this will raise an assertion error.

class llnl.util.lock.LockTransaction(lock, acquire_fn=None, release_fn=None, timeout=None)

Bases: object

Simple nested transaction context manager that uses a file lock.

This class can trigger actions when the lock is acquired for the first time and released for the last.

If the acquire_fn returns a value, it is used as the return value for __enter__, allowing it to be passed as the as argument of a with statement.

If acquire_fn returns a context manager, its __enter__ function will be called in __enter__ after acquire_fn, and its __exit__ funciton will be called before release_fn in __exit__, allowing you to nest a context manager to be used along with the lock.

Timeout for lock is customizable.

class llnl.util.lock.WriteTransaction(lock, acquire_fn=None, release_fn=None, timeout=None)

Bases: llnl.util.lock.LockTransaction

LockTransaction context manager that does a write and releases it.

class llnl.util.lock.ReadTransaction(lock, acquire_fn=None, release_fn=None, timeout=None)

Bases: llnl.util.lock.LockTransaction

LockTransaction context manager that does a read and releases it.

exception llnl.util.lock.LockError

Bases: Exception

Raised for any errors related to locks.

exception llnl.util.lock.LockTimeoutError

Bases: llnl.util.lock.LockError

Raised when an attempt to acquire a lock times out.

exception llnl.util.lock.LockPermissionError

Bases: llnl.util.lock.LockError

Raised when there are permission issues with a lock.

exception llnl.util.lock.LockROFileError(path)

Bases: llnl.util.lock.LockPermissionError

Tried to take an exclusive lock on a read-only file.

exception llnl.util.lock.CantCreateLockError(path)

Bases: llnl.util.lock.LockPermissionError

Attempt to create a lock in an unwritable location.

llnl.util.multiproc module

This implements a parallel map operation but it can accept more values than multiprocessing.Pool.apply() can. For example, apply() will fail to pickle functions if they’re passed indirectly as parameters.

llnl.util.multiproc.spawn(f)
llnl.util.multiproc.parmap(f, elements)
class llnl.util.multiproc.Barrier(n, timeout=None)

Bases: object

Simple reusable semaphore barrier.

Python 2.6 doesn’t have multiprocessing barriers so we implement this.

See http://greenteapress.com/semaphores/downey08semaphores.pdf, p. 41.

wait()

Module contents