Using External GPU Support

Many packages come with a +cuda or +rocm variant. With no added configuration Spack will download and install the needed components. It may be preferable to use existing system support: the following sections help with using a system installation of GPU libraries.

Using an External ROCm Installation

Spack breaks down ROCm into many separate component packages. The following is an example packages.yaml that organizes a consistent set of ROCm components for use by dependent packages:

packages:
  all:
    compiler: [rocmcc@=5.3.0]
    variants: amdgpu_target=gfx90a
  hip:
    buildable: false
    externals:
    - spec: hip@5.3.0
      prefix: /opt/rocm-5.3.0/hip
  hsa-rocr-dev:
    buildable: false
    externals:
    - spec: hsa-rocr-dev@5.3.0
      prefix: /opt/rocm-5.3.0/
  llvm-amdgpu:
    buildable: false
    externals:
    - spec: llvm-amdgpu@5.3.0
      prefix: /opt/rocm-5.3.0/llvm/
  comgr:
    buildable: false
    externals:
    - spec: comgr@5.3.0
      prefix: /opt/rocm-5.3.0/
  hipsparse:
    buildable: false
    externals:
    - spec: hipsparse@5.3.0
      prefix: /opt/rocm-5.3.0/
  hipblas:
    buildable: false
    externals:
    - spec: hipblas@5.3.0
      prefix: /opt/rocm-5.3.0/
  rocblas:
    buildable: false
    externals:
    - spec: rocblas@5.3.0
      prefix: /opt/rocm-5.3.0/
  rocprim:
    buildable: false
    externals:
    - spec: rocprim@5.3.0
      prefix: /opt/rocm-5.3.0/rocprim/

This is in combination with the following compiler definition:

compilers:
- compiler:
    spec: rocmcc@=5.3.0
    paths:
      cc: /opt/rocm-5.3.0/bin/amdclang
      cxx: /opt/rocm-5.3.0/bin/amdclang++
      f77: null
      fc: /opt/rocm-5.3.0/bin/amdflang
    operating_system: rhel8
    target: x86_64

This includes the following considerations:

• Each of the listed externals specifies buildable: false to force Spack to use only the externals we defined.

• spack external find can automatically locate some of the hip/rocm packages, but not all of them, and furthermore not in a manner that guarantees a complementary set if multiple ROCm installations are available.

• The prefix is the same for several components, but note that others require listing one of the subdirectories as a prefix.

Using an External CUDA Installation

CUDA is split into fewer components and is simpler to specify:

packages:
  all:
    variants:
    - cuda_arch=70
  cuda:
    buildable: false
    externals:
    - spec: cuda@11.0.2
      prefix: /opt/cuda/cuda-11.0.2/

where /opt/cuda/cuda-11.0.2/lib/ contains libcudart.so.