Install firedrake on CentOS 7

2. Install firedrake on CentOS 7#

These notes describe how to install Firedrake on a small HPC system running CentOS 7 without root privileges. All dependencies, including GCC and Python, are installed using Spack.

Note:

The login node has internet access, but direct access to github.com may be restricted.

If you need to access github.com, use an SSH proxy as described in the SSH proxy section.

2.1. Install spack#

We install spack on dir $HOME/opt.

Clone the repo and load spack env

git clone --depth=2 --branch=releases/v0.23 https://github.com/spack/spack.git $HOME/opt/spack

. $HOME/opt/spack/share/spack/setup-env.sh

2.2. Install dependencies#

Install required packages:

spack install gcc@9.3.0
spack install bison cmake flex git gmake openblas ninja python tmux cuda@11.2.0 py-venv py-pip py-pysocks
spack install openmpi +cuda cuda_arch=80,86 fabrics=ucx ^cuda@11.2.0 \
    ^ucx+cuda+verbs+cma+dc+dm+gdrcopy+mlx5_dv+thread_multiple cuda_arch=80,86

Create a Spack view to collect the installed tools:

spack view -d no add -i $HOME/opt/tools-view \
    gcc@9.3.0 openmpi hwloc bison cmake flex git gmake ninja python tmux openblas zlib-ng ucx \
    py-venv py-pip py-pysocks

If you want to remove a package from the view (for example, cmake), run:

spack view -d no rm $HOME/opt/tools-view cmake

Add the bin path of view to PATH

export PATH="$HOME/opt/tools-view/bin:$PATH"

2.2.1. Notes on CUDA and Kokkos#

CUDA:
Refer to the CUDA 11.2 installation guide.
The GPU nodes use Nvidia driver version 460.27.04, which is compatible with CUDA 11.2. According to the CUDA documentation, GCC versions below 10 are required.
Kokkos:
See the Kokkos requirements.
Kokkos requires a compiler version greater than 8.2.0.

Based on these requirements, GCC 9.3.0 is selected. Note that GCC 9.4.0 and 9.5.0 may cause the error:
error: identifier "__builtin_ia32_rndscalesd_round" is undefined

2.3. Install Firedrake#

Follow the official Firedrake installation instructions.

See install-firedrake-centos7.sh for an example.

2.3.1. Wrapper script for launching Python#

Create a file with the following content and save it in the bin directory of the Firedrake environment, so it can be used to start Python in the corresponding environment:

#!/usr/bin/env bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
FIRDRAKE_BASENAME=$( basename $( dirname $SCRIPT_DIR ) )  # firedrake-comple
MY_SPACK_DIR=$( dirname $( dirname $SCRIPT_DIR ) )           # /.../firedrake-env
MY_SPACK_LOG=$(spack env status)

if [ "`basename \"$VIRTUAL_ENV\"`" != "$FIRDRAKE_BASENAME" ]; then
        if [ "$VIRTURE_ENV" != "" ]; then
                deactivate
        fi
        if `echo "$MY_SPACK_LOG" | grep -v "$MY_SPACK_DIR" > /dev/null 2>&1`; then
                if `echo "$MY_SPACK_LOG" | grep -v "No" > /dev/null 2>&1`; then
                        spack env deactivate
                fi
                spack env activate $MY_SPACK_DIR -p
                unset PYTHON_PATH
        fi
        . $SCRIPT_DIR/activate
fi

export MPIR_CVAR_ENABLE_GPU=0
export OMP_NUM_THREADS=1
# pass all the parameters to python
$SCRIPT_DIR/python $@

2.3.2. Configure PETSc#

When building PETSc, you need to specify the locations of dependencies so PETSc can find them. Set the environment variable $DEPS_VIEW to your Spack view directory (e.g., $HOME/opt/tools-view). Add the following options to your PETSc configure command:

--with-hwloc-dir=$DEPS_VIEW \
--with-openblas-dir=$DEPS_VIEW \
--with-zlib-dir=$DEPS_VIEW \

For CUDA support, set $CUDA_DIR to the CUDA installation path (find it with spack location -i cuda) and add:

--with-cuda-dir=$CUDA_DIR

To enable Kokkos support, add the following options to your PETSc configure command (replace 80 with the compute capability of your GPU if different):

--with-cuda-arch=80 `#maybe not needed https://petsc.org/release/changes/316/#changes-3-16` \
--download-kokkos \
--download-kokkos-kernels

2.3.3. Notes#

Since OpenMPI is built with CUDA support, you can disable CUDA at runtime by setting the environment variable:
```
export OMPI_MCA_opal_cuda_support=0
```
If running on a system without CUDA, add the CUDA stubs directory to LD_LIBRARY_PATH to suppress warnings about missing libcuda.so:
```
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$CUDA_DIR/lib64/stubs"
```
If you encounter an error about libcuda.so.1 not being found, create a symbolic link in the stubs directory:
```
ln -s libcuda.so "$CUDA_DIR/lib64/stubs/libcuda.so.1"
```

2.4. SSH proxy#

This section describes how to set up an SSH SOCKS5 proxy on port 5000 and ensure it is automatically stopped when your session ends.

First, configure passwordless SSH access to your proxy server by adding the following to your ~/.ssh/config:

Host proxy-server
    HostName your.proxy.server.address
    User your-username

To start the proxy and set the necessary environment variables for git, curl, and pip, use the following script:

#!/bin/bash

set -e

ssh -vv -n -N -D 5000 -o ExitOnForwardFailure=yes proxy-server >ssh-proxy.log 2>&1 &
SSH_PID=$!

for i in {1..30}; do
    if ! ps -p $SSH_PID > /dev/null; then
        echo "SSH failed to start. See ssh-proxy.log for details."
        exit 1
    fi
    sleep 0.1
done

echo "Started SSH tunnel with PID $SSH_PID"

trap "echo 'Killing SSH tunnel...'; kill $SSH_PID" EXIT

export http_proxy="socks5h://localhost:5000"
export https_proxy="socks5h://localhost:5000"

# Place your commands that require proxy access below