Compiling LLVM with Clang and OpenMP for Remote OpenMP Offloading

I am trying to compile LLVM with Remote OpenMP Offloading feature. 2 changes have been made to file openmp/libomptarget/plugins/remote/CMakeLists.txt to enable it.

  1. add set(LIBOMPTARGET_ENABLE_EXPERIMENTAL_REMOTE_PLUGIN "1") before if (NOT(LIBOMPTARGET_ENABLE_EXPERIMENTAL_REMOTE_PLUGIN))
  2. add set(CMAKE_PREFIX_PATH "/home/bscuser/opt/grpc/lib/cmake") before find_package(gRPC CONFIG).

The cmake enables compilation but compilation pops errors with grpc like ‘grpc_impl::internal’ has not been declared. I have tried latest tags of release 14 and 15.

cmake command:
$ cmake ../llvm/ -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=`pwd` -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_SHARED_LIBS=on -DLLVM_ENABLE_PROJECTS="clang;openmp

What is the right way to build OpenMP for Remote Offloading?

I asked the experts to reply.

Uthmnahere,

This error has to do with the version of grpc and protobuf in the system. We also struggle a lot with this, and we are trying to improve it.

We were able to build it with versions 1.46.2 of gRPC. We built it with this command:

git clone --branch v1.46.2 --depth 1 https://github.com/grpc/grpc
cd grpc
git submodule update --init --quiet --depth 1
cd ..
cmake -S grpc                               \
      -B grpc_objdir                        \
      -G Ninja                              \
      -DCMAKE_BUILD_TYPE=Release            \
      -DCMAKE_INSTALL_PREFIX="$PREFIX"      \
      -DCMAKE_C_COMPILER=gcc             \
      -DCMAKE_CXX_COMPILER=g++           \
      -DCMAKE_C_FLAGS="-O3 -march=native"             \
      -DCMAKE_CXX_FLAGS="-O3 -march=native"         \
      -DCMAKE_EXE_LINKER_FLAGS="-O3 -march=native -Wl,-O1"   \
      -DBUILD_SHARED_LIBS=ON                \
      -DCMAKE_CXX_STANDARD=17               \
      -DgRPC_INSTALL=ON                     \
      -DgRPC_BUILD_TESTS=OFF                \
      -DgRPC_BUILD_CODEGEN=ON               \
      -DgRPC_BUILD_CSHARP_EXT=OFF           \
      -DgRPC_BACKWARDS_COMPATIBILITY_MODE=OFF
ninja -C grpc_objdir -j install

There is currently an effort on upstreaming a back end that uses UCX instead of gRPC. But I am not sure what is the status of that.

It would be nice to see if the above command helps. If not, let me know so I can run some tests on my end and try to find appropriate instructions.

the changes I mentioned earlier are not required for gRPC when I use this command. It enables remote openmp offloading out of the box.

Compilation still fails. Error message(s) for tag llvmorg-14.0.6:

path/llvm-project/openmp/libomptarget/plugins/remote/server/Server.cpp: In member function ‘virtual grpc::Status RemoteOffloadImpl::InitDevice(grpc::ServerContext*, const openmp::libomptarget::remote::I32*, openmp::libomptarget::remote::I32*)’:
path/llvm-project/openmp/libomptarget/plugins/remote/server/Server.cpp:109:54: error: ‘__gnu_cxx::__alloc_traits<std::allocator<std::unique_ptr<DeviceTy> > >::value_type {aka class std::unique_ptr<DeviceTy>}’ has no member named ‘RTL’
   Reply->set_number(PM->Devices[DeviceNum->number()].RTL->init_device(

This was fixed after 14 was released, in ⚙ D119984 [Libomptarget][remote] Fix compile-time error

The following error with llvmorg-15.0.0-rc2:

/home/bscuser/dev/llvm/llvm-project/build_15_0_0_rc2/include/openmp/libomptarget/plugins/remote/openmp.grpc.pb.h:2202:26: error: ‘internal’ in namespace ‘grpc_impl’ does not name a type
         new ::grpc_impl::internal::CallbackServerStreamingHandler< ::grpc::ByteBuffer, ::grpc::ByteBuffer>(

which branch or commit should I use to get it compiling and working?

This looks like an error in the generated protobuf header. Did we make sure the GRPC version and protobuf are what we support? @josemonsalve2 and others build it with trunk recently successfully. What are your versions of gRPC and protobuf?

  • grpc v1.46.2
  • protoc v3.9.2

We used Wenbin’s fork at the end because we couldn’t figure out how to get these versions right. Here are the details.

https://github.com/lyu/llvm-project/commits/remote-offloading

These are the instructions he shared with us. Let me know if you figure out how to build the trunk version.

Installing gRPC:


#!/usr/bin/env bash

PREFIX=$HOME/opt
MAKE_JOBS=16
OPT_FLAGS="-O3"
UARCH_FLAGS="-march=native"

export TMPDIR=/dev/shm
export CC=/usr/bin/gcc
export CXX=/usr/bin/g++
export CFLAGS="$OPT_FLAGS $UARCH_FLAGS"
export CXXFLAGS="$OPT_FLAGS $UARCH_FLAGS"
export LDFLAGS="$OPT_FLAGS $UARCH_FLAGS -Wl,-O1"

git clone --branch v1.46.2 --depth 1 https://github.com/grpc/grpc
cd grpc
git submodule update --init --quiet --depth 1
cd ..
cmake -S grpc                               \
      -B grpc_objdir                        \
      -G Ninja                              \
      -DCMAKE_BUILD_TYPE=Release            \
      -DCMAKE_INSTALL_PREFIX="$PREFIX"      \
      -DCMAKE_C_COMPILER="$CC"              \
      -DCMAKE_CXX_COMPILER="$CXX"           \
      -DCMAKE_C_FLAGS="$CFLAGS"             \
      -DCMAKE_CXX_FLAGS="$CXXFLAGS"         \
      -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS"   \
      -DBUILD_SHARED_LIBS=ON                \
      -DCMAKE_CXX_STANDARD=17               \
      -DgRPC_INSTALL=ON                     \
      -DgRPC_BUILD_TESTS=OFF                \
      -DgRPC_BUILD_CODEGEN=ON               \
      -DgRPC_BUILD_CSHARP_EXT=OFF           \
      -DgRPC_BACKWARDS_COMPATIBILITY_MODE=OFF
ninja -C grpc_objdir -j $MAKE_JOBS install
rm -rf grpc*

Installing UCX:


#!/usr/bin/env bash

PREFIX=$HOME/opt
OPT_FLAGS="-O3"
UARCH_FLAGS="-march=native"

export CFLAGS="$OPT_FLAGS $UARCH_FLAGS"
export CXXFLAGS="$OPT_FLAGS $UARCH_FLAGS"
export LDFLAGS="$OPT_FLAGS $UARCH_FLAGS -Wl,-O1"


git clone --branch v1.13.x --depth 1 https://github.com/openucx/ucx
cd ucx
./autogen.sh
./configure --prefix=$PREFIX                \
            --enable-mt                     \
            --enable-cma                    \
            --enable-numa                   \
            --enable-silent-rules           \
            --enable-optimizations          \
            --enable-builtin-memcpy         \
            --enable-compiler-opt=3         \
            --enable-option-checking        \
            --disable-debug                 \
            --disable-gtest                 \
            --disable-stats                 \
            --disable-static                \
            --disable-tuning                \
            --disable-logging               \
            --disable-examples              \
            --disable-profiling             \
            --disable-assertions            \
            --disable-debug-data            \
            --disable-params-check          \
            --disable-fault-injection       \
            --disable-frame-pointer         \
            --disable-dependency-tracking   \
            --without-go                    \
            --without-bfd                   \
            --without-mpi                   \
            --without-knem                  \
            --without-java                  \
            --without-valgrind              \
            --with-cuda=$CUDA_PATH
make -j && make install
cd ..
rm -rf ucx

Installing LLVM:


#!/usr/bin/env bash

# LIBOMPTARGET_RPC_ADDRESS={IP}:{PORT}
# LIBOMPTARGET_RPC_TRANSPORT=gRPC/UCX


PREFIX=$HOME/offload_llvm
MAKE_JOBS=40

export LDFLAGS="-Wl,-O1"

git clone --depth 1 git@github.com:lyu/llvm-project
cmake -S llvm-project/llvm                          \
      -B llvm_objdir                                \
      -G Ninja                                      \
      -DCMAKE_INSTALL_PREFIX="$PREFIX"              \
      -DCMAKE_BUILD_TYPE=Release                    \
      -DCMAKE_C_COMPILER="/usr/bin/gcc"             \
      -DCMAKE_CXX_COMPILER="/usr/bin/g++"           \
      -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS"           \
      -DLLVM_BUILD_UTILS=OFF                        \
      -DLLVM_ENABLE_PROJECTS="clang"                \
      -DLLVM_TARGETS_TO_BUILD="host;NVPTX"          \
      -DGCC_INSTALL_PREFIX="/usr"                   \
      -DCLANG_ENABLE_ARCMT=OFF                      \
      -DCLANG_ENABLE_STATIC_ANALYZER=OFF            \
      -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_70
ninja -C llvm_objdir -j $MAKE_JOBS install

rm -rf llvm_objdir


cmake -S llvm-project/openmp                        \
      -B llvm_objdir                                \
      -G Ninja                                      \
      -DLLVM_ROOT="$PREFIX"                         \
      -DCMAKE_INSTALL_PREFIX="$PREFIX"              \
      -DCMAKE_BUILD_TYPE=Release                    \
      -DCMAKE_C_COMPILER="$PREFIX/bin/clang"        \
      -DCMAKE_CXX_COMPILER="$PREFIX/bin/clang++"    \
      -DCMAKE_EXE_LINKER_FLAGS="$LDFLAGS"           \
      -DCMAKE_EXPORT_COMPILE_COMMANDS=ON            \
      -DUCX_DIR="$HOME/opt"                         \
      -DOPENMP_ENABLE_LIBOMPTARGET_PROFILING=OFF    \
      -DLIBOMPTARGET_ENABLE_EXPERIMENTAL_REMOTE_PLUGIN=ON   \
      -DLIBOMP_HAVE_OMPT_SUPPORT=OFF                \
      -DLIBOMP_INSTALL_ALIASES=OFF                  \
      -DLIBOMP_USE_HWLOC=ON                         \
      -DLIBOMP_HWLOC_INSTALL_DIR="$HOME/.local"     \
      -DLIBOMPTARGET_ENABLE_DEBUG=OFF               \
      -DLIBOMPTARGET_BUILD_AMDGPU_PLUGIN=OFF        \
      -DLIBOMPTARGET_AMDGCN_GFXLIST=""              \
      -DLIBOMPTARGET_BUILD_NVPTX_BCLIB=ON           \
      -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=70  \
      -DLIBOMPTARGET_NVPTX_ALTERNATE_HOST_COMPILER="/usr/bin/gcc"
ninja -C llvm_objdir -j $MAKE_JOBS install

mv llvm_objdir/libomptarget/openmp-offloading-server $PREFIX/bin

rm -rf llvm_objdir

I have tried these commands twice and I am still getting a very basic error while compiling for OpenMP. It says:

FAILED: libomptarget/plugins/remote/src/CMakeFiles/omptarget.rtl.remote.dir/__/transports/ucx/Base.cpp.o
llvm-project/openmp/libomptarget/plugins/remote/transports/ucx/Utils.h:55:13: error: no template named 'vector' in namespace 'std'
extern std::vector<std::string> MessageKindToString;
       ~~~~~^
llvm-project/openmp/libomptarget/plugins/remote/transports/ucx/Utils.h:67:8: error: no template named 'vector' in namespace 'std' 
  std::vector<ConnectionConfigTy> ConnectionConfigs;
  ~~~~~^

I was able to build the different projects.After talking to @uthmnahere.It seems to be due to the difference in compiler versions we had installed on our machine.
Here are the versions
gcc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
clang version 14.0.0-1ubuntu1