Cross-compilation of MLIR runner libraries to ARM on X86 host

Hi,

I am trying to cross-compile MLIR runner libraries such as libmlir_runner_utils.so to ARM on a X86_64 host.

My clang/llvm/mlir build commands are as follows:

cmake -GNinja \
  "-H$LLVM_SRC_DIR/llvm" \
  "-B$build_dir$suffix" \
  -DCMAKE_C_COMPILER=clang-10 \
  -DCMAKE_CXX_COMPILER=clang++-10 \
  -DCMAKE_INSTALL_PREFIX=$install_dir$suffix  \
  -DLLVM_INSTALL_UTILS=ON   \
  -DLLVM_ENABLE_LLD=ON   \
  -DLLVM_ENABLE_PROJECTS="clang;llvm;mlir"   \
  -DCMAKE_CROSSCOMPILING=True \
  -DLLVM_TARGET_ARCH="ARM" \
  -DLLVM_TARGETS_TO_BUILD="ARM;X86"   \
  -DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabihf \
  -DLLVM_INCLUDE_TOOLS=ON   \
  -DLLVM_BUILD_TOOLS=OFF   \
  -DLLVM_INCLUDE_TESTS=ON   \
  -DMLIR_INCLUDE_TESTS=ON   \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DLLVM_ENABLE_ASSERTIONS=On \
  -DLLVM_BUILD_EXAMPLES=OFF

cmake --build "$build_dir$suffix" --target clang opt mlir-opt mlir-translate mlir-cpu-runner FileCheck --target install

With this setup and qemu-user installed, I can AOT compile mlir code into both x86 or ARM object files, and execute ARM binaries with the emulator.

Now I would like to link ARM object files with the runner libraries, however these were only compiled for the x86 target (verified by objectdump -f <file.so>). I believe the same thing happened when I only have -DLLVM_TARGETS_TO_BUILD="ARM" set.

This in development patch is sort of related to what I want to do, but I cant tell which CMAKE flags to set so that MLIR runner libraries get compiled for the LLVM_TARGET_ARCH="ARM"

TLDR:
How can I configure the project so that libmlir_runner_utils.so gets compiled to an ARM target?

Linker command used:

/working_dir/builds/llvm-project/build-cross/bin/clang --target=arm-linux-gnueabihf /working_dir/builds/llvm-project/build-cross/lib/libmlir_runner_utils.so -o main main.o
/working_dir/builds/llvm-project/build-cross/lib/libmlir_runner_utils.so: file not recognized: File format not recognized
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

Cross-compilation of the runtime isn’t really supported right now, this is an unfortunate consequence of the runtime growing within the project itself without distinction.

This is something I’m trying to fix with [RFC] Restructuring of the MLIR repo by introducing a runtime/ folder that would be compiled for the “target” instead of the “host” (where the compiler runs).

The only way right now (I think) is to build separately the project for ARM (basically cross-compile the compiler itself for a ARM “host”) and use the objects from there.

I’m confused by this. When I cross-compile MLIR, I get:

$ file lib/libmlir_runner_utils.so.14git 
lib/libmlir_runner_utils.so.14git: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, not stripped

The only major diffference I see is we target AArch64, not 32-bit ARM.

Usually when cross-compiling the tricky part is the bits that need to get compiled and run on the host, like tablegen. Is it possible you’re build line is just not actually building that library?

There are 3 platforms involved in cross-compiling:

  • The “build”: the one you run cmake and ninja on (let’s say X86)
  • The host machine: the one the compiler will run on (let’s say Arm64)
  • The target machine: the one the compiled code will run on (let’s say RISC-V)

Right now you can cross-compile MLIR on a X86 machine and get a compiler that runs on a Arm64 platform. But if you want to then to use this compiler to build RISC-V executable, you only have an Arm64 runtime.

So what is possible right now is only the case where “host” == “target”, regardless of “build”.
I assumed that the question was about “host” != “target”, but maybe I misunderstood the original description?

2 Likes

Yes, this is a problem. One way to canadian cross like this (if this is what is desired) is to straight-cross build as well to get the right runtime libraries. Essentially this is a special case of “Most of what I’m building is going to run on the host machine”, but some things (namely the compiler that generates code for the target) also need to be able to run on the build machine.

1 Like

Thank you for your comments. I really appreciate it.

I started following the thread. Thanks!

Using mehdi’s scheme, I have a x86 build machine, a x86 host machine, and an ARM target machine. So “host” != “target”

I have set DLLVM_INCLUDE_TOOLS=ON during build, with the build line below, which allowed me to complete a build without separate tablegen binaries. It generated x86 runner libraries. Tablegen gets build as first task of the build process. I will try to specify a separate tablegen path using -DLLVM_TABLEGEN=....

cmake -GNinja \
  "-H$LLVM_SRC_DIR/llvm" \
  "-B$build_dir$suffix" \
  -DCMAKE_C_COMPILER=clang-10 \
  -DCMAKE_CXX_COMPILER=clang++-10 \
  -DCMAKE_INSTALL_PREFIX=$install_dir$suffix  \
  -DLLVM_INSTALL_UTILS=ON  \
  -DLLVM_ENABLE_LLD=ON   \
  -DLLVM_ENABLE_PROJECTS="mlir;llvm" \
  -DCMAKE_CROSSCOMPILING=True \
  -DLLVM_TARGET_ARCH=ARM \
  -DLLVM_TARGETS_TO_BUILD=ARM   \
  -DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabihf \
  -DLLVM_INCLUDE_TOOLS=ON   \
  -DLLVM_BUILD_TOOLS=OFF   \
  -DLLVM_INCLUDE_TESTS=OFF   \
  -DMLIR_INCLUDE_TESTS=OFF   \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=OFF \
  -DLLVM_BUILD_EXAMPLES=OFF

It is building the library for the build (or host) system

file /working_dir/builds/llvm-project/build-cross/lib/libmlir_runner_utils.so.14git
/working_dir/builds/llvm-project/build-cross/lib/libmlir_runner_utils.so.14git: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[xxHash]=d3d72e055a38ce4c, with debebug_info, not stripped

@stephenneuendorffer Would you mind sharing your build line that enabled you to cross-compile the runner libraries to the AArch64 target?

I will iterate over my build line to see if I can achieve the same results.

Humm, after building llvm a couple of times trying to generate AArch64 host binaries, as suggested by Stephen, but getting x86 executables, I found this article:

It appears that Ubuntu’s apt-installed clang++10 can only target x86 (clang --print-targets). I was also missing -DCMAKE_CXX_FLAGS='-march=armv8-a -mtune=cortex-a72' flags.

I have started a new compilation following instructions from the post and will update here with the outcome.

For Julia we do a lot of cross-compilation including MLIR, our build setup uses binarybuilder.org, but you can find our build-scripts here Yggdrasil/common.jl at 36f194120bc4b18f9eafc25960ce84ba05f21ad9 · JuliaPackaging/Yggdrasil · GitHub

2 Likes

Thank you for sharing this. Your scripts put me on the right path, namely getting the right compiler during cross compilation of the relevant bits. I will post an update soon on how I achieved what I wanted.

1 Like

Finally arrived on a solution.
I wanted to run armv7 binaries using qemu-arm (on x86). These binaries derive from MLIR files that depend on the runner library, hence the following is necessary.

  • Cross-compile the mlir runner libraries, on a x86 host/builder to target arm.
  • Compile the mlir code into a dynamically linked binary for ARM
  • Link at runtime binary, runner libraries, additional decencies for qemu

For that, a couple of builds must be performed, thus I followed the steps below:

  1. On your x86 host, enter development container / install needed libraries
    • With qemu-user, crossbuild-essential-armhf, libz-dev, clang-10 installed
  2. Compile bootstrap mlir-tblgen binaries (for x86)
    • required when cross compiling for arm
    • can be integrated with 3, but I found best to keep it separate
  3. Compile clang, llvm, mlir for x86 with the required targets
    • I am particularly interested on the x86 and ARM targets
    • mlir runner libraries will be compiled only for x86 in this step
  4. Cross-compile mlir-runner libraries for ARM
    • this requires arm libs (from step 1) and llvm-project/cmake targeting cross-compilation
    • needs mlir-tblgen from step 2
  5. Compile and link desired application for arm, run with qemu
    • Using compiler from step 3, libraries from step 4, and pointing qemu to look for the right libraries

Sketch of relevant commands:

# 1. On your x86 host, enter development container / install needed libraries
apt update && apt install -y clang-10 crossbuild-essential-armhf libz-dev qemu-user

# 2. Compile bootstrap mlir-tblgen binaries
suffix=-bootstrap-tblgen
build_dir=build
install_dir=install
cmake -GNinja \
  "-H$LLVM_SRC_DIR/llvm" \
  "-B$build_dir$suffix" \
  -DCMAKE_C_COMPILER=clang-10 \
  -DCMAKE_CXX_COMPILER=clang++-10 \
  -DCMAKE_INSTALL_PREFIX=$install_dir$suffix  \
  -DLLVM_INSTALL_UTILS=ON   \
  -DLLVM_ENABLE_LLD=ON   \
  -DLLVM_ENABLE_PROJECTS="clang;llvm;mlir"   \
  -DLLVM_TARGET_ARCH="host" \
  -DLLVM_INCLUDE_TOOLS=ON   \
  -DLLVM_BUILD_TOOLS=OFF   \
  -DLLVM_INCLUDE_TESTS=OFF   \
  -DMLIR_INCLUDE_TESTS=OFF   \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_ASSERTIONS=OFF \
  -DLLVM_BUILD_EXAMPLES=OFF

cmake --build "$build_dir$suffix" --target clang-tblgen llvm-tblgen mlir-tblgen

# 3. Compile clang, llvm, mlir for x86 with the required targets
# This will take a while (20min using 16c/32t)
suffix=-x86
build_dir=build
install_dir=install
cmake -GNinja \
  "-H$LLVM_SRC_DIR/llvm" \
  "-B$build_dir$suffix" \
  -DCMAKE_C_COMPILER=clang-10 \
  -DCMAKE_CXX_COMPILER=clang++-10 \
  -DCMAKE_INSTALL_PREFIX=$install_dir$suffix  \
  -DLLVM_INSTALL_UTILS=ON   \
  -DLLVM_ENABLE_LLD=ON   \
  -DLLVM_ENABLE_PROJECTS="clang;llvm;mlir"   \
  -DLLVM_TARGET_ARCH="host" \
  -DLLVM_TARGETS_TO_BUILD="host;ARM;AArch64"   \
  -DLLVM_INCLUDE_TOOLS=ON   \
  -DLLVM_BUILD_TOOLS=OFF   \
  -DLLVM_INCLUDE_TESTS=ON   \
  -DMLIR_INCLUDE_TESTS=ON   \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DLLVM_ENABLE_ASSERTIONS=On \
  -DLLVM_BUILD_EXAMPLES=OFF

cmake --build "$build_dir$suffix" --target clang opt mlir-opt mlir-translate mlir-cpu-runner FileCheck --target install


# 4. Cross-compile mlir-runner libraries for ARM 
# Options to target armv7
  # -DCMAKE_C_COMPILER=arm-linux-gnueabihf-gcc-7 \
  # -DCMAKE_CXX_COMPILER=arm-linux-gnueabihf-g++-7 \
  # -DCMAKE_CROSSCOMPILING=True \
  # -DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabihf \
  # -DLLVM_TARGET_ARCH=ARM \
  # -DLLVM_TARGETS_TO_BUILD=ARM  \
  # -DCMAKE_CXX_FLAGS='-fcompare-debug-second' \

# Options to target AArch64
  #   -DCMAKE_C_COMPILER=aarch64-linux-gnu-gcc \
  # -DCMAKE_CXX_COMPILER=aarch64-linux-gnu-g++ \
  # -DCMAKE_CROSSCOMPILING=True \
  # -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-linux-gnu \
  # -DLLVM_TARGET_ARCH=AArch64 \
  # -DLLVM_TARGETS_TO_BUILD=AArch64  \
  
# Options required if compiling clang or llvm
  # -DCLANG_TABLEGEN=$clang_tablegen \
  # -DLLVM_TABLEGEN=$llvm_tablegen \

suffix=-runner-arm
build_dir=build
install_dir=install
mlir_tablegen=build-bootstrap-tblgen/bin/mlir-tblgen
cmake -GNinja \
  "-H$LLVM_SRC_DIR/llvm" \
  "-B$build_dir$suffix" \
  -DCMAKE_INSTALL_PREFIX=$install_dir$suffix  \
  -DMLIR_TABLEGEN=$mlir_tablegen \
  -DLLVM_INSTALL_UTILS=OFF  \
  -DLLVM_ENABLE_LLD=OFF   \
  -DLLVM_ENABLE_PROJECTS="mlir" \
  -DCMAKE_C_COMPILER=arm-linux-gnueabihf-gcc-7 \
  -DCMAKE_CXX_COMPILER=arm-linux-gnueabihf-g++-7 \
  -DCMAKE_CXX_FLAGS='-fcompare-debug-second' \
  -DCMAKE_CROSSCOMPILING=True \
  -DLLVM_DEFAULT_TARGET_TRIPLE=arm-linux-gnueabihf \
  -DLLVM_TARGET_ARCH=ARM \
  -DLLVM_TARGETS_TO_BUILD=ARM  \
  -DLLVM_BUILD_TOOLS=OFF   \
  -DLLVM_INCLUDE_TESTS=OFF   \
  -DMLIR_INCLUDE_TESTS=OFF   \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DLLVM_ENABLE_ASSERTIONS=ON \
  -DLLVM_BUILD_EXAMPLES=OFF

cmake --build "$build_dir$suffix" --target mlir_c_runner_utils mlir_runner_utils

# 5. Compile and link desired application for arm, run with qemu

# Or for individual steps on AOT compiling app.mlir into an arm binary
cp llvm-project/mlir/test/Integration/Dialect/Linalg/CPU/test-conv-1d-nwc-wcf-call.mlir app.mlir
/working_dir/builds/llvm-project/build-x86/bin/mlir-opt -convert-linalg-to-loops -convert-scf-to-std -convert-linalg-to-llvm -lower-affine -convert-scf-to-std --convert-memref-to-llvm -convert-std-to-llvm -reconcile-unrealized-casts -o app-llvm.mlir app.mlir
/working_dir/builds/llvm-project/build-x86/bin/mlir-translate -mlir-to-llvmir -o app.ll app-llvm.mlir
/working_dir/builds/llvm-project/build-x86/bin/clang --target=arm-linux-gnueabihf -march=armv7-a -marm -mfloat-abi=hard -c -o app.o app.ll
/working_dir/builds/llvm-project/build-x86/bin/clang -o app app.o --target=arm-linux-gnueabihf -Wl,-rpath=/working_dir/builds/llvm-project/build-runner-arm/lib -L/working_dir/builds/llvm-project/build-runner-arm/lib -lmlir_runner_utils
# Running with qemu
qemu-arm 

Some relevant clang compilation flags to use when compiling the object files:

# Target triple
ARM_TARGET_TRIPLE=arm-linux-gnueabihf
X86_TARGET_TRIPLE=x86_64-linux-gnu
AARCH64_TARGET_TRIPLE=aarch64-linux-gnu

# Flags
ARM_FLAGS=--target=$(ARM_TARGET_TRIPLE) -march=armv7-a -marm -mfloat-abi=hard
AARCH64_FLAGS=--target=$(AARCH64_TARGET_TRIPLE) -march=armv8-a -marm
X86_FLAGS=--target=$(X86_TARGET_TRIPLE)

# Linker (pass this so that no LD_LIBRARY_PATH is needed)
MLIRLIB=/working_dir/builds/llvm-project/build-runner-arm/lib
RUNNER_FLAGS=-L$(MLIRLIB) -lmlir_runner_utils
LDFLAGS=--target=$(TARGET_TRIPLE) -Wl,-rpath=$(MLIRLIB) $(RUNNER_FLAGS)

Thank so much for the replies and help. Hope this can help someone in the future.

2 Likes