Cross compiling C++ program

Project Infrastructure LLVM Dev List Archives
1

Hello,

As a newcomer to the libcxx implementation, I'll probably ask a silly question, but why following example can be compiled natively, but not cross compiled?

#include <cstdlib>
using ::atof;
int main() {
  return 0;
}

Native compile: clang++ -o example example.cpp
Cross compile: clang++ -fno-builtin -fno-exceptions -fcheck-new -nostdlib -ffreestanding -target arm-none-eabi -march=armv7-m -mcpu=cortex-m4 -Wall -Wshadow -Wundef -g -Os -fno-strict-aliasing -fomit-frame-pointer -mcpu=cortex-m4 -mthumb -mfloat-abi=soft -stdlib=libc++ -o example example.cpp

Error I get:
In file included from example.cpp:1:
In file included from /usr/include/c++/v1/cstdlib:85:
/usr/include/c++/v1/__config:938:4: error: "No thread API"
# error "No thread API"
   ^
1 error generated.

Without -stdlib=libc++ I get this error:
example.cpp:1:10: fatal error: 'cstdlib' file not found
#include <cstdlib>
         ^~~~~~~~~
1 error generated.

My OS is FreeBSD 11.1 and Clang version is 4.0.0. What am I not understanding well enough and could you recommend reading something on the topic? Any help is welcome! Thank you!

Regards,
meka

2

Hello,

As a newcomer to the libcxx implementation, I'll probably ask a silly question, but why following example can be compiled natively, but not cross compiled?

#include <cstdlib>
using ::atof;
int main() {
   return 0;
}

Native compile: clang++ -o example example.cpp
Cross compile: clang++ -fno-builtin -fno-exceptions -fcheck-new -nostdlib -ffreestanding -target arm-none-eabi -march=armv7-m -mcpu=cortex-m4 -Wall -Wshadow -Wundef -g -Os -fno-strict-aliasing -fomit-frame-pointer -mcpu=cortex-m4 -mthumb -mfloat-abi=soft -stdlib=libc++ -o example example.cpp

Error I get:
In file included from example.cpp:1:
In file included from /usr/include/c++/v1/cstdlib:85:

I strongly recommend against using the host's headers when cross compiling. You need to either find or build an arm-none-eabi sysroot, and use the --sysroot= flag.

Jon

3

OK, that's a start. I found https://www.llvm.org/docs/HowToCrossCompileLLVM.html. Is it enough for --sysroot? In case it is, what should be the value of -DLLVM_DEFAULT_TARGET_TRIPLE if I want to use FreeBSD as host and Cortex-M4 as sysroot? Thanx!

Regards,
meka

4

I strongly recommend against using the host's headers when cross compiling.
You need to either find or build an arm-none-eabi sysroot, and use the
--sysroot= flag.

Jon

OK, that's a start. I found How To Cross-Compile Clang/LLVM using Clang/LLVM — LLVM 18.0.0git documentation. Is it enough for --sysroot? In case it is, what should be the value of -DLLVM_DEFAULT_TARGET_TRIPLE if I want to use FreeBSD as host and Cortex-M4 as sysroot? Thanx!

IIUC, you don't want to cross compile llvm itself (which is what those instructions are for), but instead you want to *use* llvm to cross compile things.

To build your sysroot, you'll need to cross-build:

   1) A libc. Good choices for that for baremetal are: newlib or musl.
   2) Builtins. In llvm, that's provided in the compiler-rt repo. Otherwise you can use libgcc from an arm-none-eabi-gcc toolchain.
   3) If you want c++, you'll also need:
      3A) c++ abi library. llvm provides this as libcxxabi. There's also libsupc++, and libcxxrt, depending on what licenses you're comfortable with.
      3B) An unwinder. llvm provides libunwind. There's also one packaged in libgcc_s (depending on how it was built), and another from the old HP libunwind project (different codebase, but same name as the llvm one).
      3C) A c++ standard library. llvm provides this as libcxx. There's also libstdc++.

To start with, I recommend using the cmake cache in: $clang_src/cmake/caches/BaremetalARM.cmake. You'll want the stuff in it that references the armv7em-none-eabi triple. To use it, do something like:

$ cmake -G <build system> -DBAREMETAL_ARMV7EM_SYSROOT=path/to/your/v7em/sysroot -C path/to/BaremetalARM.cmake [other CMake Options]

Assuming your sysroot has libc headers in it, that should get you a clang that's capable of compiling basic things, which you can use to build all the libraries above.

Jon

5

Hello,

Your input was very valuable, but I must be too noob so I have to ask. I tried to read as much as I could about musl, compiler-rt and libunwind to understand what are those libs and I decided to go with a "relexed" licenced alternatives. :o) If I got it right, the order of repos I need to compile is:

- musl
- compiler-rt
- libcxxabi
- libunwind
- libcxx
- clang (with the suggestion about baremetal cache)

If you know any doc on how to get clang++ compile for any bare metal arm, I'll be glad to RTFM. :o)

6

IIUC, you don't want to cross compile llvm itself (which is what those
instructions are for), but instead you want to *use* llvm to cross compile
things.

To build your sysroot, you'll need to cross-build:

   1) A libc. Good choices for that for baremetal are: newlib or musl.
   2) Builtins. In llvm, that's provided in the compiler-rt repo. Otherwise
you can use libgcc from an arm-none-eabi-gcc toolchain.
   3) If you want c++, you'll also need:
      3A) c++ abi library. llvm provides this as libcxxabi. There's also
libsupc++, and libcxxrt, depending on what licenses you're comfortable with.
      3B) An unwinder. llvm provides libunwind. There's also one packaged in
libgcc_s (depending on how it was built), and another from the old HP
libunwind project (different codebase, but same name as the llvm one).
      3C) A c++ standard library. llvm provides this as libcxx. There's also
libstdc++.

To start with, I recommend using the cmake cache in:
$clang_src/cmake/caches/BaremetalARM.cmake. You'll want the stuff in it that
references the armv7em-none-eabi triple. To use it, do something like:

$ cmake -G <build system>
-DBAREMETAL_ARMV7EM_SYSROOT=path/to/your/v7em/sysroot -C
path/to/BaremetalARM.cmake [other CMake Options]

Assuming your sysroot has libc headers in it, that should get you a clang
that's capable of compiling basic things, which you can use to build all the
libraries above.

Jon

Hello,

Your input was very valuable, but I must be too noob so I have to ask. I tried to read as much as I could about musl, compiler-rt and libunwind to understand what are those libs and I decided to go with a "relexed" licenced alternatives. :o) If I got it right, the order of repos I need to compile is:

- musl
- compiler-rt
- libcxxabi
- libunwind
- libcxx
- clang (with the suggestion about baremetal cache)

Not quite. You'll need clang built in order to build the other things. (I guess you could use your existing toolchain to do that, but IIUC, you want to use clang for all of it & bootstrap a full clang-based toolchain). For that, the usual order is to:

1) Pick a directory as your sysroot
2) Install the libc headers there (without having built it)
3) Build clang w/ baremetal cache, pointing it at the partially built sysroot.
4) Build your libc.
4A) Test your libc.
4B) Install your libc in your sysroot.
5) Build libcxxabi. (it's build needs to know where the source for libcxx lives)
5A) Test libcxxabi. (testing it requires knowing where source for libcxx lives. use remote executors & linker scripts to run on your board, or in qemu)
5B) Install libcxxabi in your sysroot.
6) Build libunwind.
6A) Test libunwind. (testing it requires knowing where source for libcxx lives. use remote executors & linker scripts to run on your board, or in qemu)
6B) Install libunwind in your sysroot.
7) Build libcxx.
7A) Test libcxx. (use remote executors & linker scripts to run on your board, or in qemu)
7B) Install libcxx in your sysroot.

A note on linker scripts: support for them in lld is still a bit rough around the edges, so you may want to use the linker that comes with your board (presumably binutils ld).

If you know any doc on how to get clang++ compile for any bare metal arm, I'll be glad to RTFM. :o)

Sorry, I don't have a doc for that. We're getting closer to making it work "out of the tin", but it's not there yet. We (wearing my CodeSourcery hat) are working on it, but it's going to take some time before it "just works" upstream.

Jon

7

You may also take a look at the ELLCC project (www.ellcc.org). As far as I understand it they produce and package cross toolchains for a number of popular targets.

  • Matthias
8

AFAIU, ELLCC doesn’t support baremetal. It only has linux targets. Jon

9

I am talking with him on IRC to try to make ELLCC run on FreeBSD via Linux emulation and maybe get started with that.

10

> > IIUC, you don't want to cross compile llvm itself (which is what those
> > instructions are for), but instead you want to *use* llvm to cross compile
> > things.
> >
> > To build your sysroot, you'll need to cross-build:
> >
> > 1) A libc. Good choices for that for baremetal are: newlib or musl.
> > 2) Builtins. In llvm, that's provided in the compiler-rt repo. Otherwise
> > you can use libgcc from an arm-none-eabi-gcc toolchain.
> > 3) If you want c++, you'll also need:
> > 3A) c++ abi library. llvm provides this as libcxxabi. There's also
> > libsupc++, and libcxxrt, depending on what licenses you're comfortable with.
> > 3B) An unwinder. llvm provides libunwind. There's also one packaged in
> > libgcc_s (depending on how it was built), and another from the old HP
> > libunwind project (different codebase, but same name as the llvm one).
> > 3C) A c++ standard library. llvm provides this as libcxx. There's also
> > libstdc++.
> >
> >
> > To start with, I recommend using the cmake cache in:
> > $clang_src/cmake/caches/BaremetalARM.cmake. You'll want the stuff in it that
> > references the armv7em-none-eabi triple. To use it, do something like:
> >
> > $ cmake -G <build system>
> > -DBAREMETAL_ARMV7EM_SYSROOT=path/to/your/v7em/sysroot -C
> > path/to/BaremetalARM.cmake [other CMake Options]
> >
> > Assuming your sysroot has libc headers in it, that should get you a clang
> > that's capable of compiling basic things, which you can use to build all the
> > libraries above.
> >
> >
> > Jon
>
> Hello,
>
> Your input was very valuable, but I must be too noob so I have to ask. I tried to read as much as I could about musl, compiler-rt and libunwind to understand what are those libs and I decided to go with a "relexed" licenced alternatives. :o) If I got it right, the order of repos I need to compile is:
>
> - musl
> - compiler-rt
> - libcxxabi
> - libunwind
> - libcxx
> - clang (with the suggestion about baremetal cache)

Not quite. You'll need clang built in order to build the other things. (I
guess you could use your existing toolchain to do that, but IIUC, you want
to use clang for all of it & bootstrap a full clang-based toolchain). For
that, the usual order is to:

1) Pick a directory as your sysroot

~/arm-sysroot

2) Install the libc headers there (without having built it)

What I did was mv musl-*/include ~/arm-sysroot/usr/include.

3) Build clang w/ baremetal cache, pointing it at the partially built
sysroot.

cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -C ../../code/clang/cmake/caches/BaremetalARM.cmake ../../code/clang
loading initial cache file ../../code/clang/cmake/caches/BaremetalARM.cmake
-- Found LLVM_CONFIG as /usr/local/bin/llvm-config
-- Building with -fPIC
-- Could NOT find Z3 (missing: Z3_LIBRARIES Z3_INCLUDE_DIR) (Required is at least version "4.5")
-- Clang version: 4.0.1
CMake Error at /usr/local/llvm40/lib/cmake/llvm/LLVM-Config.cmake:203 (message):
  Library `BinaryFormat' not found in list of llvm libraries.
Call Stack (most recent call first):
  /usr/local/llvm40/lib/cmake/llvm/AddLLVM.cmake:508 (llvm_map_components_to_libnames)
  cmake/modules/AddClang.cmake:86 (llvm_add_library)
  lib/AST/CMakeLists.txt:6 (add_clang_library)

-- Configuring incomplete, errors occurred!
See also "/usr/home/meka/repos/clang/build/clang/CMakeFiles/CMakeOutput.log".

A note on linker scripts: support for them in lld is still a bit rough
around the edges, so you may want to use the linker that comes with your
board (presumably binutils ld).

Thank you!

Sorry, I don't have a doc for that. We're getting closer to making it work
"out of the tin", but it's not there yet. We (wearing my CodeSourcery hat)
are working on it, but it's going to take some time before it "just works"
upstream.

I will test what ever you throw my way regarding this (or write a doc).

11

IIUC, you don't want to cross compile llvm itself (which is what those
instructions are for), but instead you want to *use* llvm to cross compile
things.

To build your sysroot, you'll need to cross-build:

    1) A libc. Good choices for that for baremetal are: newlib or musl.
    2) Builtins. In llvm, that's provided in the compiler-rt repo. Otherwise
you can use libgcc from an arm-none-eabi-gcc toolchain.
    3) If you want c++, you'll also need:
       3A) c++ abi library. llvm provides this as libcxxabi. There's also
libsupc++, and libcxxrt, depending on what licenses you're comfortable with.
       3B) An unwinder. llvm provides libunwind. There's also one packaged in
libgcc_s (depending on how it was built), and another from the old HP
libunwind project (different codebase, but same name as the llvm one).
       3C) A c++ standard library. llvm provides this as libcxx. There's also
libstdc++.

To start with, I recommend using the cmake cache in:
$clang_src/cmake/caches/BaremetalARM.cmake. You'll want the stuff in it that
references the armv7em-none-eabi triple. To use it, do something like:

$ cmake -G <build system>
-DBAREMETAL_ARMV7EM_SYSROOT=path/to/your/v7em/sysroot -C
path/to/BaremetalARM.cmake [other CMake Options]

Assuming your sysroot has libc headers in it, that should get you a clang
that's capable of compiling basic things, which you can use to build all the
libraries above.

Jon

Hello,

Your input was very valuable, but I must be too noob so I have to ask. I tried to read as much as I could about musl, compiler-rt and libunwind to understand what are those libs and I decided to go with a "relexed" licenced alternatives. :o) If I got it right, the order of repos I need to compile is:

- musl
- compiler-rt
- libcxxabi
- libunwind
- libcxx
- clang (with the suggestion about baremetal cache)

Not quite. You'll need clang built in order to build the other things. (I
guess you could use your existing toolchain to do that, but IIUC, you want
to use clang for all of it & bootstrap a full clang-based toolchain). For
that, the usual order is to:

1) Pick a directory as your sysroot

~/arm-sysroot

2) Install the libc headers there (without having built it)

What I did was mv musl-*/include ~/arm-sysroot/usr/include.

I think you need to copy them, not move them.

3) Build clang w/ baremetal cache, pointing it at the partially built
sysroot.

  cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -C ../../code/clang/cmake/caches/BaremetalARM.cmake ../../code/clang

Sorry, I wasn't explicit enough. You need to define all of the variables that the cache is looking for.

loading initial cache file ../../code/clang/cmake/caches/BaremetalARM.cmake
-- Found LLVM_CONFIG as /usr/local/bin/llvm-config
-- Building with -fPIC
-- Could NOT find Z3 (missing: Z3_LIBRARIES Z3_INCLUDE_DIR) (Required is at least version "4.5")
-- Clang version: 4.0.1
CMake Error at /usr/local/llvm40/lib/cmake/llvm/LLVM-Config.cmake:203 (message):
   Library `BinaryFormat' not found in list of llvm libraries.
Call Stack (most recent call first):
   /usr/local/llvm40/lib/cmake/llvm/AddLLVM.cmake:508 (llvm_map_components_to_libnames)
   cmake/modules/AddClang.cmake:86 (llvm_add_library)
   lib/AST/CMakeLists.txt:6 (add_clang_library)

-- Configuring incomplete, errors occurred!
See also "/usr/home/meka/repos/clang/build/clang/CMakeFiles/CMakeOutput.log".

I think it found an installed llvm, and is trying to use that to build clang against. First, the clang and llvm's versions have to match. Second, when I mentioned building clang, I really meant building llvm+clang... sorry.

$ cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -C path/to/copy/of/edited/BaremetalARM.cmake path/to/llvm

Where your path/to/llvm has the symlinks:

   * path/to/llvm/tools/clang -> path/to/clang
   * path/to/llvm/runtimes/compiler_rt path/to/compiler_rt

And by "edited BaremetalARM.cmake" I mean you can strip out the other targets in there that you don't need.

Jon

12

> > 1) Pick a directory as your sysroot
> ~/arm-sysroot
>
> > 2) Install the libc headers there (without having built it)
> What I did was mv musl-*/include ~/arm-sysroot/usr/include.

I think you need to copy them, not move them.

On a new directories and fresh untar of musl, I don't think there's a difference

> > 3) Build clang w/ baremetal cache, pointing it at the partially built
> > sysroot.
> cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -C ../../code/clang/cmake/caches/BaremetalARM.cmake ../../code/clang

Sorry, I wasn't explicit enough. You need to define all of the variables
that the cache is looking for.

> loading initial cache file ../../code/clang/cmake/caches/BaremetalARM.cmake
> -- Found LLVM_CONFIG as /usr/local/bin/llvm-config
> -- Building with -fPIC
> -- Could NOT find Z3 (missing: Z3_LIBRARIES Z3_INCLUDE_DIR) (Required is at least version "4.5")
> -- Clang version: 4.0.1
> CMake Error at /usr/local/llvm40/lib/cmake/llvm/LLVM-Config.cmake:203 (message):
> Library `BinaryFormat' not found in list of llvm libraries.
> Call Stack (most recent call first):
> /usr/local/llvm40/lib/cmake/llvm/AddLLVM.cmake:508 (llvm_map_components_to_libnames)
> cmake/modules/AddClang.cmake:86 (llvm_add_library)
> lib/AST/CMakeLists.txt:6 (add_clang_library)
>
> -- Configuring incomplete, errors occurred!
> See also "/usr/home/meka/repos/clang/build/clang/CMakeFiles/CMakeOutput.log".

I think it found an installed llvm, and is trying to use that to build clang
against. First, the clang and llvm's versions have to match. Second, when I
mentioned building clang, I really meant building llvm+clang... sorry.

$ cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -C
path/to/copy/of/edited/BaremetalARM.cmake path/to/llvm

Where your path/to/llvm has the symlinks:

  * path/to/llvm/tools/clang -> path/to/clang

ln -s `which clang40` path/to/llvm/tools/clang

  * path/to/llvm/runtimes/compiler_rt path/to/compiler_rt

ln -s path/to/llvm/projects/compiler-rt path/to/llvm/runtimes/compiler_rt

Is that right?

And by "edited BaremetalARM.cmake" I mean you can strip out the other
targets in there that you don't need.

So I have copied BaremetalARM.cmake into llvm/cmake/caches/BaremetalARM.cmake to be able to alter it, I've put all other repos in llvm/projects and the command is:

# cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -DBAREMETAL_ARMV6M_SYSROOT=~/arm-sysroot -DBAREMETAL_ARMV7M_SYSROOT=~/arm-sysroot -C ../../code/llvm/cmake/caches/BaremetalARM.cmake ../../code/llvm

Before adding all repos into llvm/projects, it complained about 'Specified distribution component 'clang' doesn't have a target' (just an example). Now I have 'Specified distribution component 'builtins-armv6m-none-eabi' doesn't have a target' as error.

13

1) Pick a directory as your sysroot

~/arm-sysroot

2) Install the libc headers there (without having built it)

What I did was mv musl-*/include ~/arm-sysroot/usr/include.

I think you need to copy them, not move them.

On a new directories and fresh untar of musl, I don't think there's a difference

I was assuming you'd want to build it from the same sources later...

3) Build clang w/ baremetal cache, pointing it at the partially built
sysroot.

   cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -C ../../code/clang/cmake/caches/BaremetalARM.cmake ../../code/clang

Sorry, I wasn't explicit enough. You need to define all of the variables
that the cache is looking for.

loading initial cache file ../../code/clang/cmake/caches/BaremetalARM.cmake
-- Found LLVM_CONFIG as /usr/local/bin/llvm-config
-- Building with -fPIC
-- Could NOT find Z3 (missing: Z3_LIBRARIES Z3_INCLUDE_DIR) (Required is at least version "4.5")
-- Clang version: 4.0.1
CMake Error at /usr/local/llvm40/lib/cmake/llvm/LLVM-Config.cmake:203 (message):
    Library `BinaryFormat' not found in list of llvm libraries.
Call Stack (most recent call first):
    /usr/local/llvm40/lib/cmake/llvm/AddLLVM.cmake:508 (llvm_map_components_to_libnames)
    cmake/modules/AddClang.cmake:86 (llvm_add_library)
    lib/AST/CMakeLists.txt:6 (add_clang_library)

-- Configuring incomplete, errors occurred!
See also "/usr/home/meka/repos/clang/build/clang/CMakeFiles/CMakeOutput.log".

I think it found an installed llvm, and is trying to use that to build clang
against. First, the clang and llvm's versions have to match. Second, when I
mentioned building clang, I really meant building llvm+clang... sorry.

$ cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -C
path/to/copy/of/edited/BaremetalARM.cmake path/to/llvm

Where your path/to/llvm has the symlinks:

   * path/to/llvm/tools/clang -> path/to/clang

ln -s `which clang40` path/to/llvm/tools/clang

No, not the binary, the source.

The closer to top of trunk you pull the source, the easier it'll be to get it working. A lot of the upstreaming of baremetal support has been *very* recent.... "bleeding edge" recent.

   * path/to/llvm/runtimes/compiler_rt path/to/compiler_rt

ln -s path/to/llvm/projects/compiler-rt path/to/llvm/runtimes/compiler_rt

Is that right?

Sorry, it should be spelled `compiler-rt`, not `compiler_rt`.

And by "edited BaremetalARM.cmake" I mean you can strip out the other
targets in there that you don't need.

So I have copied BaremetalARM.cmake into llvm/cmake/caches/BaremetalARM.cmake to be able to alter it, I've put all other repos in llvm/projects and the command is:

Leave all the others out for now. An in-tree build of them is just going to give you host binaries, which isn't helpful in your case. You'll have to build them "out of tree".

Might be a helpful exercise for you to try building vanilla clang with runtimes for the host, before worrying about how to build everything for your baremetal case: Getting Started with the LLVM System — LLVM 18.0.0git documentation

# cmake -G 'Unix Makefiles' -DBAREMETAL_ARMV7EM_SYSROOT=~/arm-sysroot -DBAREMETAL_ARMV6M_SYSROOT=~/arm-sysroot -DBAREMETAL_ARMV7M_SYSROOT=~/arm-sysroot -C ../../code/llvm/cmake/caches/BaremetalARM.cmake ../../code/llvm

Before adding all repos into llvm/projects, it complained about 'Specified distribution component 'clang' doesn't have a target' (just an example). Now I have 'Specified distribution component 'builtins-armv6m-none-eabi' doesn't have a target' as error.

I think that error happened because you symlinked the clang binary in, not the source.

Jon

14

I think I'll start with that, first :o)

15

ELLCC has been used for baremetal stuff, but it takes some work. I’m looking at adding NuttX support, which will make baremetal much easier. -Rich

16

So I did. :o) It was fun realizing how much Release type reduces LD memory usage :o)

Anyway, just to be clear what "I did" means:
- llvm sources from trunk and projects/tools repos checkout under llvm
- clang-6.0 compiles "Hello World"
- make check-all fails (not sure I'm not causing it)
- make install creates everything under PREFIX

# cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_INSTALL_PREFIX=/tmp/x86-sysroot ../../source/llvm
# make -j4
# make install

It was fun learning how to manualy compile compiler for the first time. Now how to cross compile for bare? Given that I have musl-1.1.16 directory and that I'm preseeding sysroot with:

# rm -rf /tmp/barearm-sysroot
# mkdir -p /tmp/barearm-sysroot
# cp -r musl-1.1.16/include /tmp/barearm-sysroot/ # Thanx for the tip!

So I tried to use it, and clang/llvm build, but compiler-rt fails:

# cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE=Release -DBAREMETAL_ARMV7EM_SYSROOT=/tmp/barearm-sysroot -DCMAKE_INSTALL_PREFIX=/tmp/barearm-sysroot -C ../../source/llvm/cmake/caches/BaremetalARM.cmake ../../source/llvm
# make -j4
https://gist.github.com/anonymous/85cd5081a6d46a63731d970ee95b20b4

This is the BaremetalARM.cmake:
https://gist.github.com/anonymous/ff3248f08ddc99361dcc009e9f4c1020

17

It is slightly tricky to compile compile musl and compiler-rt. The bits
directory is populated when you install musl, but musl needs compiler-rt
built if you want to build libc.so.
The solution that works is to first do a "make install-headers" on musl,
then build compiler-rt, then build musl. If you look at the ELLCC
make-libraries and build-musl scripts, you'll see how ELLCC does it. Or
you can disable building the musl shared library.

The other problem you'll run into is that musl is nowhere near bare
metal. It's whole purpose in life is to turn the C standard library
calls into Linux system calls. While a baremetal version of musl would
be very nice to have, it is a non-trivial project. Baremetal is what
something like NuttX's C library provides.

-Rich

18

>
> So I tried to use it, and clang/llvm build, but compiler-rt fails:
>
> # cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE=Release -DBAREMETAL_ARMV7EM_SYSROOT=/tmp/barearm-sysroot -DCMAKE_INSTALL_PREFIX=/tmp/barearm-sysroot -C ../../source/llvm/cmake/caches/BaremetalARM.cmake ../../source/llvm
> # make -j4
> gist:85cd5081a6d46a63731d970ee95b20b4 · GitHub
>
> This is the BaremetalARM.cmake:
> gist:ff3248f08ddc99361dcc009e9f4c1020 · GitHub
It is slightly tricky to compile compile musl and compiler-rt. The bits
directory is populated when you install musl, but musl needs compiler-rt
built if you want to build libc.so.
The solution that works is to first do a "make install-headers" on musl,
then build compiler-rt, then build musl. If you look at the ELLCC
make-libraries and build-musl scripts, you'll see how ELLCC does it. Or
you can disable building the musl shared library.

Thanx for this. I got to libunwind (libcxxabi expects it as dependency) and this is the error I get:

# set -xg CC /tmp/barearm-sysroot/bin/clang-6.0
# set -xg CXX /tmp/barearm-sysroot/bin/clang-6.0
# cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE=Release -DBAREMETAL_ARMV7EM_SYSROOT=/tmp/barearm-sysroot -DCMAKE_INSTALL_PREFIX=/tmp/barearm-sysroot -C ../../source/llvm/cmake/caches/BaremetalARM.cmake ../../source/libunwind
# make -j4 VERBOSE=1

The other problem you'll run into is that musl is nowhere near bare
metal. It's whole purpose in life is to turn the C standard library
calls into Linux system calls. While a baremetal version of musl would
be very nice to have, it is a non-trivial project. Baremetal is what
something like NuttX's C library provides.

I know, but I though that this is how ELLCC is built, too. Baremetal clang build with musl not having much to with libc that NuttX uses. Also, I though this is a way to create sysroot. Am I wrong?

19

Might be a helpful exercise for you to try building vanilla clang with
runtimes for the host, before worrying about how to build everything for
your baremetal case:
Getting Started with the LLVM System — LLVM 18.0.0git documentation

So I did. :o) It was fun realizing how much Release type reduces LD memory usage :o)

Anyway, just to be clear what "I did" means:
- llvm sources from trunk and projects/tools repos checkout under llvm
- clang-6.0 compiles "Hello World"
- make check-all fails (not sure I'm not causing it)
- make install creates everything under PREFIX

# cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_INSTALL_PREFIX=/tmp/x86-sysroot ../../source/llvm
# make -j4
# make install

It was fun learning how to manualy compile compiler for the first time. Now how to cross compile for bare? Given that I have musl-1.1.16 directory and that I'm preseeding sysroot with:

# rm -rf /tmp/barearm-sysroot
# mkdir -p /tmp/barearm-sysroot
# cp -r musl-1.1.16/include /tmp/barearm-sysroot/ # Thanx for the tip!

Your sysroot needs to be laid out like:

$sysroot/usr/include/<libc headers here>
$sysroot/usr/include/c++/v1/<libcxx headers here>
$sysroot/usr/lib/<all target *.a's here, except for the builtins>

with the builtins placed in:

$PREFIX/lib/clang/$version/lib/baremetal/libclang_rt.builtins-armv7em.a
$BUILD/lib/clang/$version/lib/baremetal/libclang_rt.builtins-armv7em.a

Jon

20

# set -xg CC /tmp/barearm-sysroot/bin/clang-6.0
# set -xg CXX /tmp/barearm-sysroot/bin/clang-6.0

So I think I need to clarify what a sysroot actually is, since that ^ suggests you think it's like $prefix, which it's not.

A sysroot is a collection of headers, and runtime bits, all of which are designed to run on a $target. Your clang however, was built to run on $host. For non-baremetal targets, the headers & libraries are a snapshot of how they'd be laid out on the actual target, i.e. if your target has `/usr/lib/libc.a`, then your sysroot should have `$sysroot/usr/lib/libc.a`. Likewise for all the relevant headers, and $target binaries.

Usually, when you build a packaged toolchain, the sysroots live inside $prefix next to the compiler installation like so:

$prefix/armv7em-none-eabi/usr/include/stdio.h
$prefix/armv7em-none-eabi/usr/lib/libc.a
$prefix/bin/clang

Then when compiling things with that clang, you do:

> $prefix/bin/clang++ -target armv7em-none-eabi --sysroot=$prefix/armv7em-none-eabi -T path/to/some/ldscript.ld hello_world.cpp -o hw.out

Jon