Advanced Configuration and Build

nvcc_wrapper: Why do we have it and what does it do

In general (except when being configured with Kokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE=ON), performance-portable code depending on Kokkos identifies itself as CXX code in CMake. This implies that the compiler used by CMake to compile CXX code must be able to compile the code for the backends and architectures enabled in Kokkos.

For the CUDA backend using nvcc this implies setting CMAKE_CXX_COMPILER=nvcc during configuration. But nvcc requires separate flags for host and device compilation. This requirement of nvcc for separate flags implies that other libraries that are linked to the same target also need to adhere to do this. Other compilers like clang do not need to differentiate between host and device flags and thus do not need a wrapper script but can be directly used via CMAKE_CXX_COMPILER.

To help users with this problem, Kokkos comes with a small bash script called nvcc_wrapper located in the bin subdirectory. This script has two functions. It redirects compile and link commands to nvcc, and it sorts the given compiler and linker flags into the ones for the host and the device compiler.

To use it explicitly, set CMAKE_CXX_COMPILER accordingly. If the compiler can’t compile device code, Kokkos will use kokkos_launch_compiler (see next section) to redirect calls to nvcc_wrapper automatically.

kokkos_launch_compiler: What does it do and how to control it

In complex software projects that rely on multiple libraries, it can occur that the compiler Kokkos requires for the enabled backend can not compile one of the other libraries the project depends on. To work around these situations, Kokkos introduced another bash script called kokkos_launch_compiler. This script only redirects compiler and linker commands that compile a C++ file that uses Kokkos to a compiler that can compile Kokkos code (e.g. nvcc_wrapper for the CUDA backend). Compiler and linker commands of C++ files that don’t use Kokkos, or files in different languages will not be redirected.

This script, located in the bin subdirectory, is meant to be used like a compiler launcher in CMake. But (except when being configured with Kokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE=ON) Kokkos will try to detect if the CXX compiler that CMake uses can compile the code for the enabled backend. If the CXX compiler can not compile the backend code, Kokkos automatically uses kokkos_launch_compiler. If it redirects to nvcc_wrapper, it will set nvcc’s host compiler to the CXX compiler. The idea of this is to simplify the usage of Kokkos’ performance-portable build system.

Although this covers most usecases, Kokkos provides ways for users to request kokkos_launch_compiler to be used always or never. To always use kokkos_launch_compiler, users can ask for the launch_compiler component when calling find_package:

find_package(Kokkos REQUIRED COMPONENTS launch_compiler)

In contrast, when users want that kokkos_launch_compiler to never be used:

find_package(Kokkos REQUIRED COMPONENTS separable_compilation)

Compile in the CMake language of the backend

For the CUDA and HIP backends, that have dedicated CMake languages corresponding to them, Kokkos can be configured to only set compiler and linker flags in the respective CMake language. This effectively causes Kokkos to identify as a CUDA or HIP library to CMake (instead of identifying as a CXX library). This does allow additional host backends to be enabled (e.g. CUDA, OPENMP and SERIAL can be compiled in CUDA language mode). This implies, that files that link to Kokkos also have to identify as the respective CMake language, and need to be compiled for the architecture that is enabled in Kokkos. To enable this behavior, enable the option Kokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE. For a detailed example and explanation check the notes on the option.

Compile in multiple languages

Complex software projects might use multiple libraries that use Kokkos. If one of these uses Kokkos in the CMake language of the backend (e.g. Cuda or HIP), while other libraries use Kokkos as a CXX library, they need a Kokkos installation that can be compiled in both modes. To allow this, Kokkos provides the option Kokkos_ENABLE_MULTIPLE_CMAKE_LANGUAGES. For details and requirements check the notes on the option.