Repository files navigation

ppkMHD stands for Performance Portable Kokkos for Magneto-HydroDynamics (MHD) solvers.

Here a small list of numerical schemes implementations:

• second order MUSCL-HANCOCK scheme for hydrodynamics and MHD
• high-order MOOD (hydrodynamics only)
• high-order Spectral Difference Method schemes: hydrodynamics only

All scheme are available in 2D and 3D using Kokkos+MPI implementation, and support at least the OpenMP and CUDA kokkos backends (other backends may be used, but not tested).

• Kokkos library will either be built by ppkMHD by using cmake option-DPPKMHD_BUILD_KOKKOS=ON, either be detected if you already have it installed.
• cmake with version >= 3.X (3.X is chosen to meet Kokkos own requirement for cmake; i.e. it might increase in the future)

For beginner user, wu suggest that you build ppkMHD with-DPPKMHD_BUILD_KOKKOS=ON so that Kokkos is built together, with the same flags as the main application.

A few example builds, with minimal configuration options.

If you build kokkos yourself, we advise you to always active HWLOC TPLS.

Just make sure that your env variableCMAKE_PREFIX_PATH point to the location where Kokkos where installed. More precisely if Kokkos is installed inKOKKOS_ROOT, you add$KOKKOS_ROOT/lib/cmake to yourCMAKE_PREFIX_PATH; this way kokkos will be found automagically by cmake, and the right Kokkos hardware backend will be selected.

# cd into ppkMHD toplevel sourcescmake -S. -B _build/default -DPPKMHD_BUILD_KOKKOS=OFFcmake --build _build/default -j 6

• Create a build directory, configure and make

# cd into ppkMHD toplevel sourcescmake -S. -B _build/openmp -DPPKMHD_BUILD_KOKKOS=ON -DPPKMHD_USE_MPI=OFF -DPPKMHD_BACKEND=OpenMP -DKokkos_ENABLE_HWLOC=ONcmake --build _build/openmp -j 6

Add variable CXX on the cmake command line to change the compiler (clang++, icpc, pgcc, ....).

• Create a build directory, configure and make

# cd into ppkMHD toplevel sourcescmake -S. -B _build/cuda -DPPKMHD_BUILD_KOKKOS=ON -DPPKMHD_USE_MPI=OFF -DPPKMHD_BACKEND=Cuda# note: GPU architecture will be detected when building on a host with a GPU; if you're building on a host, and running on another machine# you'll need to tell kokkos what is the target architecture, e.g. add flag like '-DKokkos_ARCH_TURING75=ON' on the cmake configure linecmake --build _build/cuda -j 6

Please make sure to use aCUDA-aware MPI implementation (OpenMPI or MVAPICH2) built with the proper flags for activating CUDA support.

It may happen that eventhough your MPI implementation is actually cuda-aware, cmake MPI detection (through a call tofind_package(MPI) macro) may fails to detect if your MPI implementation is cuda aware. In that case, you can enforce cuda awareness by turning optionPPKMHD_USE_MPI_CUDA_AWARE_ENFORCED to ON.

You don't need to use mpi compiler wrapper mpicxx, cmakeshould be able to correctly populateMPI_CXX_INCLUDE_PATH,MPI_CXX_LIBRARIES which are passed to all final targets by using the alias libraryMPI::MPI_CXX.

• Create a build directory, configure and make

mkdir build;cd buildcmake -DPPKMHD_USE_MPI=ON -DPPKMHD_BACKEND=Cuda -DKokkos_ARCH_TURING75=ON ..make -j 4

Example command line to run the application (1 GPU used per MPI task)

mpirun -np 4 ./ppkMHD ./test_implode_2D_mpi.ini

In order to activate building SDM (Spectral Difference Method) schemes, use cmake option-DPPKMHD_USE_SDM=ON.

The MOOD numerical scheme require some linear algebra (QR decomposition) on the host (not device). This is done using a Blas/Lapack implementation using the C language interface namedLapacke.

Please note that Atlas doesn't provide Lapackage.Currently (March 2017), on Ubuntu 16.04, package libatlas-dev is not compatible with package Lapacke (generate errors at link time). So please eitherNetlib/Lapack orOpenBLAS implementation.

If you want to enforce the use of OpenBLAS, just use a recent cmake (>=3.6) and add-DBLA_VENDOR on the cmake command line. This will tell the cmake build system (through the call to find_package(BLAS) ) to only look for the OpenBLAS implementation.

On a recent Ubuntu, if atlas is not installed, but OpenBLAS is, you don't need to have a bleeding edge cmake, current cmake will find OpenBLAS.

• ppkMHD Wiki
• Implementing Spectral Difference Methods (SDM) for Compressible Euler flow simulations using performance portable library kokkos
• Implementing Spectral Difference Methods (SDM) for Compressible Euler flow simulations using performance portable library kokkos (astrosim 2018)