PETSc Installation Notes | Debojyoti Ghosh

PETSc (Portable, Extensible Toolkit for Scientific Computing) is a library of scalable linear and non-linear equation solvers and ODE solvers, developed and maintained at the Mathematics & Computer Science Division at Argonne National Laboratory. The official webpage of PETSc provides the download and installation instructions, as well as extensive documentation. Most of the codes I work with use the TS module of PETsc for advanced high-order time-integration schemes and it will be useful to briefly go through the documentation for this module to understand and utilize them:

User Manual (PDF) (see section on TS: Scalable ODE and DAE Solvers)

TS Examples

To download the official or development releases of PETSc, go here.

Complete installation instructions, including all options, are available here. This page contains brief instructions that I find useful to quickly download and install PETSc to use with the codes I am working with.

Download

git clone -b release  https://gitlab.com/petsc/petsc.git petsc

Installation

The following steps should install PETSc:

Set environment variables PETSC_DIR and PETSC_ARCH (Can be set in .bashrc too)

export PETSC_DIR=/path/to/petsc/ (Eg. PETSC_DIR=/home/ghosh/petsc)

export PETSC_ARCH=(name of build) (Eg. arch-debug or arch-opt. It can be anything)

Move to the PETSc directory

cd $PETSC_DIR

Configure with options for debug (slow) or optimized (fast)
Example: If GNU compilers exist but PETSc needs to download and compile MPICH,

./configure --with-cc=gcc --with-fc=gfortran --download-mpich=1 --with-shared-libraries --with-debugging=1 (debug)

./configure --with-cc=gcc --with-fc=gfortran --download-mpich=1 --with-shared-libraries --with-debugging=0 (optimized)

Example: If MPICH already exists and PETSc needs to use that,

./configure --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx --with-shared-libraries --with-debugging=1 (debug)

./configure --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx --with-shared-libraries --with-debugging=0 (optimized)

Note: Use --with-batch for configuring on machines that use a job scheduler.

Note: PETSc can download and compile additional packages by adding --download-packagename=yes or use existing installations of those packages --with-packagename-include=/path/to/package/include and --with-packagename-lib=/path/to/package/lib or --with-packagename-dir=/path/to/package/ (use --help to see a list of these options and packages that PETSc can download). This page has examples of installing PETSc with downloading or using existing installations of HDF5, NetCDF and METIS packages.

Compilation

make all

Some machine specific configure commands

LLNL LC Dane (With Hypre)

./configure --with-batch --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx COPTFLAGS="-O2 -std=c99" FOPTFLAGS="-O2" CXXOPTFLAGS="-O2" --with-shared-libraries --with-debugging=0 --download-make --download-hypre --with-cxx-dialect=C++11

LLNL LC Dane (With ParMETIS, Hypre and SuperLU)

./configure --with-batch --with-cc=mpicc --with-fc=mpif90 --with-cxx=mpicxx COPTFLAGS="-O2 -std=c99" FOPTFLAGS="-O2" CXXOPTFLAGS="-O2" --with-shared-libraries --with-debugging=0 --download-make --download-cmake --download-hypre --download-superlu --download-superlu_dist --download-parmetis --download-metis --with-cxx-dialect=C++11

LLNL LC Matrix (With CUDA and Hypre)

module load gcc/12.1.1-magic mvapich2/2.3.7 cuda/12.2.2 cmake/3.30.5 boost/1.80.0

./configure --with-make-np=36 --with-mpiexec=srun -G4 --gpu-bind=none --with-batch --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 COPTFLAGS="-g -O3" FOPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O2" CUDAFLAGS+="-g -O3" --with-cuda=1 --with-cuda-arch=90 --with-debugging=0 --download-hypre

LLNL LC Tuolumne (With HIP)

module load cmake/3.24.2 rocm/6.4.0 rocmcc/6.4.0-cce-19.0.0d-magic

            export MPI_DIR=/usr/tce/packages/cray-mpich/cray-mpich-9.0.1-rocmcc-6.4.1-magic

./configure --with-make-np=32 --with-batch --with-mpi-dir=$MPI_DIR --with-hip-dir=$ROCM_PATH COPTFLAGS="-g -O3" FOPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O2" HIPOPTFLAGS="-g -O3" --with-cuda=0 --with-hip=1 --with-clanguage=c --with-debugging=0 --download-kokkos --download-kokkos-kernels --with-fortran-bindings=0

Note: The following environment variable should be available after loading the modules.

ROCM_PATH=/opt/rocm-6.4.0

NERSC Perlmutter (With CUDA and Hypre)

module load gpu craype craype-x86-milan craype-accel-nvidia80 cudatoolkit cmake/3.30.2

            export CRAY_ACCEL_TARGET=nvidia80

            export CXXFLAGS="-march=znver3"

            export CFLAGS="-march=znver3"

            export CC=cc

            export CXX=CC

            export FC=ftn

            export CUDACXX=$(which nvcc)

            export CUDAHOSTCXX=CC

./configure --with-make-np=8 --with-mpiexec=srun -G4 --gpu-bind=none --with-batch=0 --with-cc=cc --with-cxx=CC --with-fc=ftn COPTFLAGS="-g -O3" FOPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O2" CUDAFLAGS="-g -O3" --with-cuda=1 --with-cuda-arch=80 --with-debugging=0 --download-hypre

PETSc installation

Download

Installation

Some machine specific configure commands