Trying to create a pure LLVM toolchain on musl based distribution

Hello,

I'm trying to create a pure LLVM toolchain (that will not depend on GNU and produce GNU-free code too) on a musl based distribution.

For now, I use gcc to bootstrap and build all LLVM components. I do it individually because I was running out of space and memory trying to build all using LLVM_ENABLE_PROJECTS. Also, I don't want to create a all-in-one package. Then, once I'm able to build program with clang, I'll rebuild all using clang instead of gcc.

# LLVM

I've built LLVM using the following configuration:

     -DCMAKE_BUILD_TYPE=Release
     -DBUILD_SHARED_LIBS=On
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLLVM_HOST_TRIPLE=x86_64-linux-musl
     -DLLVM_ENABLE_RTTI=On

# compiler-rt

I've needed to disable xray/sanitizers otherwise it didn't build on musl.

     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DCOMPILER_RT_BUILD_BUILTINS=On
     -DCOMPILER_RT_BUILD_SANITIZERS=Off
     -DCOMPILER_RT_BUILD_XRAY=Off
     -DCOMPILER_RT_USE_BUILTINS_LIBRARY=On

# libc++

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLIBCXX_HAS_MUSL_LIBC=On
     -DLIBCXX_HAS_GCC_S_LIB=Off
     -DLIBCXX_ENABLE_STATIC=Off

# libc++abi

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLIBCXXABI_ENABLE_STATIC=Off

# libunwind

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DLIBUNWIND_ENABLE_STATIC=Off
     -DLIBUNWIND_USE_COMPILER_RT=On

# clang

And finally clang:

     -DBUILD_SHARED_LIBS=On
     -DCMAKE_BUILD_TYPE=Release
     -DCMAKE_INSTALL_PREFIX=/usr
     -DCLANG_DEFAULT_CXX_STDLIB=libc++
     -DCLANG_DEFAULT_LINKER=lld
     -DCLANG_DEFAULT_RTLIB=compiler-rt
     -DCLANG_VENDOR="Vanilla Linux"

Everything built fine, now the problem is the location of compiler-rt libraries which are not the correct place where clang/ld.lld searches. As the following output shows:

clang -v /tmp/test.c
Vanilla Linux clang version 8.0.0 (tags/RELEASE_800/final) (based on LLVM 8.0.0)
Target: x86_64-unknown-linux-musl
Thread model: posix
InstalledDir: /usr/bin
  "/usr/bin/clang-8" -cc1 -triple x86_64-unknown-linux-musl -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name test.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-array -target-cpu x86-64 -dwarf-column-info -debugger-tuning=gdb -v -resource-dir /usr/lib/clang/8.0.0 -internal-isystem /usr/local/include -internal-isystem /usr/lib/clang/8.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /usr/src/vanilla -ferror-limit 19 -fmessage-length 184 -fobjc-runtime=gcc -fdiagnostics-show-option -o /tmp/test-1d99ce.o -x c /tmp/test.c -faddrsig
clang -cc1 version 8.0.0 based upon LLVM 8.0.0 default target x86_64-linux-musl
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
  /usr/local/include
  /usr/lib/clang/8.0.0/include
  /usr/include
End of search list.
  "/usr/bin/ld.lld" -pie --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib/ld-musl-x86_64.so.1 -o a.out /usr/bin/../lib/Scrt1.o /usr/bin/../lib/crti.o crtbeginS.o -L/usr/bin/../lib -L/lib -L/usr/lib /tmp/test-1d99ce.o /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a -lc /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a crtendS.o /usr/bin/../lib/crtn.o
ld.lld: error: cannot open crtbeginS.o: No such file or directory
ld.lld: error: cannot open /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a: No such file or directory
ld.lld: error: cannot open /usr/lib/clang/8.0.0/lib/linux/libclang_rt.builtins-x86_64.a: No such file or directory
ld.lld: error: cannot open crtendS.o: No such file or directory
clang-8: error: linker command failed with exit code 1 (use -v to see invocation)

compiler-rt installed files under the following directories hierarchy instead:

usr/lib/linux/libclang_rt.builtins-x86_64.a
usr/lib/linux/libclang_rt.fuzzer-x86_64.a
usr/lib/linux/libclang_rt.fuzzer_no_main-x86_64.a
usr/lib/linux/libclang_rt.profile-x86_64.a

What did I misconfigured? Is it clang/lld or compiler-rt? Also do I need to do something else regarding crtbeginS.o and crtendS.o that are not provided by musl?

Regards,

Hello David,

I don't know much about the specifics of Musl, so I'm responding generally.

As I understand it, clang expects to find the compiler-rt libraries
relative to the resource directory, which you can find out the
location of with clang --print-resource-dir . By default it is
lib/clang/9.0.0 assuming you are building from master. I think that
-DCMAKE_INSTALL_PREFIX=/usr has broken that assumption. I think that
you would either need to take out the CMAKE_INSTALL_PREFIX or change
the location of the resource directory, which I think that you can
alter at build time.

I don't think that there is a way of preventing clang adding
crtbeginS.o and crtendS.o without --nostdlib or --nostartfiles but
using these options will also not add the other crt*.o files that you
may be expecting.

There is a line in lib/Driver/Toolchains/Gnu.cpp which the Linux driver uses:
const bool HasCRTBeginEndFiles =
      ToolChain.getTriple().hasEnvironment() ||
      (ToolChain.getTriple().getVendor() != llvm::Triple::MipsTechnologies);
It looks like Musl may need to be included there?

Peter

Hello David,

I don't know much about the specifics of Musl, so I'm responding generally.

As I understand it, clang expects to find the compiler-rt libraries
relative to the resource directory, which you can find out the
location of with clang --print-resource-dir . By default it is
lib/clang/9.0.0 assuming you are building from master. I think that
-DCMAKE_INSTALL_PREFIX=/usr has broken that assumption. I think that
you would either need to take out the CMAKE_INSTALL_PREFIX or change
the location of the resource directory, which I think that you can
alter at build time.

You're right, I've just checked both packages in Alpine and Arch and they indeed move the /usr/lib/linux directory under the clang's one. I've fixed that it worked, now I still have to find the solution regarding crtbeginS.so and crtendS.so.

Thanks :slight_smile:

I don't think that there is a way of preventing clang adding
crtbeginS.o and crtendS.o without --nostdlib or --nostartfiles but
using these options will also not add the other crt*.o files that you
may be expecting.

There is a line in lib/Driver/Toolchains/Gnu.cpp which the Linux driver uses:
const bool HasCRTBeginEndFiles =
       ToolChain.getTriple().hasEnvironment() ||
       (ToolChain.getTriple().getVendor() != llvm::Triple::MipsTechnologies);
It looks like Musl may need to be included there?

I'll have a look, I don't know much the lowest part of the toolchain build process so I don't understand very much what's going on there. Hopefully someone that has better low-level knowledges and use musl will be able to guide me for this.

What can I tell is that musl provide those files:

usr/lib/Scrt1.o
usr/lib/crt1.o
usr/lib/crti.o
usr/lib/crtn.o
usr/lib/ld-musl-x86_64.so.1

Regards,

Hello David,

I don’t know much about the specifics of Musl, so I’m responding generally.

As I understand it, clang expects to find the compiler-rt libraries
relative to the resource directory, which you can find out the
location of with clang --print-resource-dir . By default it is
lib/clang/9.0.0 assuming you are building from master. I think that
-DCMAKE_INSTALL_PREFIX=/usr has broken that assumption. I think that
you would either need to take out the CMAKE_INSTALL_PREFIX or change
the location of the resource directory, which I think that you can
alter at build time.

You’re right, I’ve just checked both packages in Alpine and Arch and
they indeed move the /usr/lib/linux directory under the clang’s one.
I’ve fixed that it worked, now I still have to find the solution
regarding crtbeginS.so and crtendS.so.

https://reviews.llvm.org/D28791 should address this. We’re still discussing some related aspects on https://reviews.llvm.org/D59264, but I’m hoping to reach a conclusion and land these changes this week.

> Le 25/03/2019 à 14:41, Peter Smith a écrit :
> > Hello David,
> >
> > I don't know much about the specifics of Musl, so I'm responding
> generally.
> >
> > As I understand it, clang expects to find the compiler-rt libraries
> > relative to the resource directory, which you can find out the
> > location of with clang --print-resource-dir . By default it is
> > lib/clang/9.0.0 assuming you are building from master. I think that
> > -DCMAKE_INSTALL_PREFIX=/usr has broken that assumption. I think that
> > you would either need to take out the CMAKE_INSTALL_PREFIX or change
> > the location of the resource directory, which I think that you can
> > alter at build time.
> >
>
> You're right, I've just checked both packages in Alpine and Arch and
> they indeed move the /usr/lib/linux directory under the clang's one.
> I've fixed that it worked, now I still have to find the solution
> regarding crtbeginS.so and crtendS.so.
>

https://reviews.llvm.org/D28791 should address this. We're still discussing
some related aspects on https://reviews.llvm.org/D59264, but I'm hoping to
reach a conclusion and land these changes this week.

Nice, thanks. I've applied D28791 in compiler-rt 8.0.0 without problems. Howerver D59264 does not apply correctly on clang 8.0.0. Will there be an intermediate version before 9.0.0? Otherwise I'll try to update the patch for 8.0.0.

> Thanks :slight_smile:
>
> > I don't think that there is a way of preventing clang adding
> > crtbeginS.o and crtendS.o without --nostdlib or --nostartfiles but
> > using these options will also not add the other crt*.o files that you
> > may be expecting.
> >
> > There is a line in lib/Driver/Toolchains/Gnu.cpp which the Linux driver
> uses:
> > const bool HasCRTBeginEndFiles =
> > ToolChain.getTriple().hasEnvironment() ||
> > (ToolChain.getTriple().getVendor() !=
> llvm::Triple::MipsTechnologies);
> > It looks like Musl may need to be included there?
>
> I'll have a look, I don't know much the lowest part of the toolchain
> build process so I don't understand very much what's going on there.
> Hopefully someone that has better low-level knowledges and use musl will
> be able to guide me for this.
>
> What can I tell is that musl provide those files:
>
> usr/lib/Scrt1.o
> usr/lib/crt1.o
> usr/lib/crti.o
> usr/lib/crtn.o
> usr/lib/ld-musl-x86_64.so.1
>
> Regards,
>
> --
> David
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

Regards,

David,

You may be interested to know that Zig (https://ziglang.org/) ships with
such a toolchain already. It cross compiles components (musl, libunwind,
etc) for the target lazily from source, and supports these libc targets:

  aarch64_be-linux-gnu
  aarch64_be-linux-musl
  aarch64-linux-gnu
  aarch64-linux-musleabi
  armeb-linux-gnueabi
  armeb-linux-gnueabihf
  armeb-linux-musleabi
  armeb-linux-musleabihf
  arm-linux-gnueabi
  arm-linux-gnueabihf
  arm-linux-musleabi
  arm-linux-musleabihf
  i386-linux-gnu
  i386-linux-musl
  mips64el-linux-gnuabi64
  mips64el-linux-gnuabin32
  mips64el-linux-musl
  mips64-linux-gnuabi64
  mips64-linux-gnuabin32
  mips64-linux-musl
  mipsel-linux-gnu
  mipsel-linux-musl
  mips-linux-gnu
  mips-linux-musl
  powerpc64le-linux-gnu
  powerpc64le-linux-musl
  powerpc64-linux-gnu
  powerpc64-linux-musl
  powerpc-linux-gnu
  powerpc-linux-musl
  riscv32-linux-musl
  riscv64-linux-gnu
  riscv64-linux-musl
  s390x-linux-gnu
  s390x-linux-musl
  sparc-linux-gnu
  sparcv9-linux-gnu
  x86_64-linux-gnu
  x86_64-linux-gnux32
  x86_64-linux-musl

Here's an example of using Zig to compile hello-world.c for
aarch64-linux-gnu:

andy@xps ~/m/z/hello> cat hello.c
#include <stdio.h>

int main(int argc, char **argv) {
    printf("what up dawg\n");
    return 0;
}
andy@xps ~/m/z/hello> zig build-exe --c-source hello.c --library c
-target aarch64v8-linux-gnu
andy@xps ~/m/z/hello> file hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV),
dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for
GNU/Linux 2.0.0, with debug_info, not stripped

And now for x86_64-linux-musl:

andy@xps ~/m/z/hello> zig build-exe --c-source hello.c --library c
-target x86_64-linux-musl
andy@xps ~/m/z/hello> ./hello
what up dawg
andy@xps ~/m/z/hello> ldd ./hello
  not a dynamic executable

This works on any OS including Windows and macOS since it has no system
dependencies.

This feature is new and not well tested yet, but if it piques your
interest, you can play with this functionality using pre-built binaries
available on https://ziglang.org/download/ or you can build Zig master
branch from source.

Regards,
Andrew