Using External GPU Support

Many packages come with a +cuda or +rocm variant. With no added configuration Spack will download and install the needed components. It may be preferable to use existing system support: the following sections help with using a system installation of GPU libraries.

Using an External ROCm Installation

Spack breaks down ROCm into many separate component packages. The following is an example packages.yaml that organizes a consistent set of ROCm components for use by dependent packages:

packages:
  all:
    compiler: [rocmcc@=5.3.0]
    variants: amdgpu_target=gfx90a
  hip:
    buildable: false
    externals:
    - spec: hip@5.3.0
      prefix: /opt/rocm-5.3.0/hip
  hsa-rocr-dev:
    buildable: false
    externals:
    - spec: hsa-rocr-dev@5.3.0
      prefix: /opt/rocm-5.3.0/
  llvm-amdgpu:
    buildable: false
    externals:
    - spec: llvm-amdgpu@5.3.0
      prefix: /opt/rocm-5.3.0/llvm/
  comgr:
    buildable: false
    externals:
    - spec: comgr@5.3.0
      prefix: /opt/rocm-5.3.0/
  hipsparse:
    buildable: false
    externals:
    - spec: hipsparse@5.3.0
      prefix: /opt/rocm-5.3.0/
  hipblas:
    buildable: false
    externals:
    - spec: hipblas@5.3.0
      prefix: /opt/rocm-5.3.0/
  rocblas:
    buildable: false
    externals:
    - spec: rocblas@5.3.0
      prefix: /opt/rocm-5.3.0/
  rocprim:
    buildable: false
    externals:
    - spec: rocprim@5.3.0
      prefix: /opt/rocm-5.3.0/rocprim/

This is in combination with the following compiler definition:

compilers:
- compiler:
    spec: rocmcc@=5.3.0
    paths:
      cc: /opt/rocm-5.3.0/bin/amdclang
      cxx: /opt/rocm-5.3.0/bin/amdclang++
      f77: null
      fc: /opt/rocm-5.3.0/bin/amdflang
    operating_system: rhel8
    target: x86_64

This includes the following considerations:

Each of the listed externals specifies buildable: false to force Spack to use only the externals we defined.
spack external find can automatically locate some of the hip/rocm packages, but not all of them, and furthermore not in a manner that guarantees a complementary set if multiple ROCm installations are available.
The prefix is the same for several components, but note that others require listing one of the subdirectories as a prefix.

Using an External CUDA Installation

CUDA is split into fewer components and is simpler to specify:

packages:
  all:
    variants:
    - cuda_arch=70
  cuda:
    buildable: false
    externals:
    - spec: cuda@11.0.2
      prefix: /opt/cuda/cuda-11.0.2/

where /opt/cuda/cuda-11.0.2/lib/ contains libcudart.so.

Using an External OpenGL API

Depending on whether we have a graphics card or not, we may choose to use OSMesa or GLX to implement the OpenGL API.

If a graphics card is unavailable, OSMesa is recommended and can typically be built with Spack. However, if we prefer to utilize the system GLX tailored to our graphics card, we need to declare it as an external. Here’s how to do it:

packages:
  libglx:
    require: [opengl]
  opengl:
    buildable: false
    externals:
    - prefix: /usr/
      spec: opengl@4.6

Note that prefix has to be the root of both the libraries and the headers, using is /usr not the path the the lib. To know which spec for opengl is available use cd /usr/include/GL && grep -Ri gl_version.