ARM Linux libc++ / libunwind — Exceptions not being caught

I have built the llvm/clang 7.0.0 rc2, including libc++/libc++abi/libunwind for my Ubuntu 16.04 Linux 32-bit ARM7 hf target. A trivial c++17 test program with just this try/catch block:

try { auto result = stoul(“foo”); } catch (...) { std::cerr << “caught!” << std::endl; }

fails to print “caught!” Instead it just aborts. If no exceptions are thrown, programs built with this toolchain behave as expected.

I have tried numerous variations of the toolchain, compile and link steps (including alternatives to the cxxabi), however they all run into various errors or ultimately give the same runtime abort.

Any suggestions for how to get a functioning Linux armhf clang/libc++ 7.0.0 toolchain?

Are you building your test program with -fno-rtti? I’ve seen a problem where if libc++ is built with RTTI and exceptions enabled, and a program is built with exceptions enabled BUT -fno-rtti, exceptions thrown from the program will not be caught.

Louis

No, I have rtti enabled. I am continuing to try and build a version of the toolchain that works, but many combinations simply don’t build and I have yet to hit upon one that catches exceptions properly.

It would be most helpful if the was a known good combination of libraries and settings that would net a working toolchain for ARM7hf Ubuntu 16.04, with CLang 7 and libc++. I’ve been trying libc++abi and libunwind mostly (and compiler-rt in the last day or two) as I had build problems with the various gcc-based paths. What is the recommended path to follow though? A lot of options are provided but clearly it’s a minefield of broken paths.

Honestly, I’m in the same situation as you are. I just started working on libc++ and I’m trying to figure out what configurations are supported — it’s not obvious to me at all. Marshall, Eric or other long date contributors may know better — I’m not the best resource. For now I’m focused on getting LLVM 7 out the door, but in the future I would like to document a matrix of supported configurations:

Linux:

libc++ | ABI library | Unwind library

Hello Andrew,

I've not built libc++ precisely with the v7 release candidate as I
don't have that to hand but I've managed to get your example to print
"caught!" when building natively on an Arm Ubuntu 16.04 machine and
using --stdlib=libc++. I built shared libraries and didn't use
compiler-rt this time to keep it simple. I followed a similar recipe
to the buildbot
http://lab.llvm.org:8011/builders/libcxx-libcxxabi-libunwind-armv7-linux
which is using clang 5.0

My steps were:
1.) Build trunk clang to use for building libc++ etc.
2.) Using cmake

cmake -GNinja\
      /path/to/monorepo/llvm \
      -DLLVM_ENABLE_PROJECTS="libcxxabi;libcxx;libunwind" \
      -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_ENABLE_ASSERTIONS=true\
      -DCMAKE_C_FLAGS='-mcpu=cortex-a57'\
      -DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'\
      -DLLVM_TARGETS_TO_BUILD='ARM'\
      -DLIBCXXABI_USE_LLVM_UNWINDER=On
3.) build
ninja libunwind
ninja cxxabi
ninja cxx

I was then able to run the example with:
clang++ --stdlib=libc++ -I ./include/c++/v1 t.cpp -o t.axf -L ./lib -v
--std=c++17
LD_LIBRARY_PATH=lib ./t.axf./t.axf
caught!

Some things you might want to try:
- Use trunk clang rather than the 7.0 release. You may have found a
release blocking bug if it is only a problem on the 7.0 branch.
- What happens to your example when you use libstd++ and the gcc
unwinding implementation? I'm trying to rule out whether clang has not
produced correct exception handling information for the program.
- If you are able to build libunwind with debug then I've had some
success with the logging functions that are enabled when certain
environment variables are set, such as LIBUNWIND_PRINT_UNWINDING. This
might help determine if the abort is because the callsite of the throw
isn't found in the .ARM.exidx exception table.
- It may also be worth simplifying your example so that it is your
executable that throws the exception and not a function in the libc++
library itself, which in this case I believe would be in the libc++
shared object itself.

I'm making the assumption you are building natively here. I have
cross-compiled from x86_64 to AArch64 before but I haven't done it for
Arm. The solution I found needed standalone builds of the libraries.
If you can share a bit more about how you are trying to build we may
be able to spot something wrong?

Peter

Hi Peter, thanks for the detailed reply...

My steps were:
1.) Build trunk clang to use for building libc++ etc.
2.) Using cmake

cmake -GNinja\
     /path/to/monorepo/llvm \
     -DLLVM_ENABLE_PROJECTS="libcxxabi;libcxx;libunwind" \
     -DCMAKE_BUILD_TYPE=Release \
     -DLLVM_ENABLE_ASSERTIONS=true\
     -DCMAKE_C_FLAGS='-mcpu=cortex-a57'\
     -DCMAKE_CXX_FLAGS='-mcpu=cortex-a57'\
     -DLLVM_TARGETS_TO_BUILD='ARM'\
     -DLIBCXXABI_USE_LLVM_UNWINDER=On
3.) build
ninja libunwind
ninja cxxabi
ninja cxx

I will attempt to emulate your steps. I just switched to ninja from makefiles (for the toolchain build) a few days ago because almost everyone uses ninja for their examples and I’m trying to minimize differences… unfortunately that seemed to cause issues with how I was file sharing (my weak ARM SBC Linux host mounts a share from an OSX machine as that is better and faster than abusing the SDcards of the SBCs). I think I have that sorted now, so hopefully the builds proceed more smoothly now. Building the toolchain on such a weak machine isn’t the fastest thing in the world. Can’t even use all the cores since the compiler’s memory footprint pushes the limits of the host’s 2GB.

Some things you might want to try:
- Use trunk clang rather than the 7.0 release. You may have found a
release blocking bug if it is only a problem on the 7.0 branch.

Well I initially inadvertently was on tip of tree (i.e. 8.0.0) until I figured out the whole svn tags/7.0.0/rc2 scheme. It had the same issue.

- What happens to your example when you use libstd++ and the gcc
unwinding implementation? I'm trying to rule out whether clang has not
produced correct exception handling information for the program.

It works fine when I build my little repro case like this (some arguments may be superfluous, they are there for historical reasons):

  clang++ -funwind-tables -stdlib=libstdc++ -std=gnu++1z -I/usr/lib/arm-linux-gnueabihf/../include -pthread -c test.cpp -o test.o
  clang++ -funwind-tables -stdlib=libstdc++ -L/usr/local/lib -lpthread test.o -o test

It fails when I build it like this:

  clang++ -funwind-tables -stdlib=libc++ -std=gnu++1z -I/usr/lib/arm-linux-gnueabihf/../include -pthread -c test.cpp -o test.o
  clang++ -funwind-tables -stdlib=libc++ -L/usr/local/lib -lunwind -lc++ -lc++abi -lpthread test.o -o test

I’ve lost track of how many ways I’ve tried to build libunwind/libc++/libc++abi. I’ve tried static vs shared libraries. I’ve tried the supc++ route and some other variations — most failed to build. There are quite a few options and configuration parameters, and its all rather hit-and-miss when you aren’t steeped in the mythos of the linux / llvm toolchain.

And no, using libstdc++ is not an attractive option.

- If you are able to build libunwind with debug then I've had some
success with the logging functions that are enabled when certain
environment variables are set, such as LIBUNWIND_PRINT_UNWINDING. This
might help determine if the abort is because the callsite of the throw
isn't found in the .ARM.exidx exception table.

Okay, that sounds like a path of last resort! :slight_smile: I did catch one variation of the problem (i.e. an earlier toolchain build) in gdb and it seemed to be stuck in an endless unwind loop with one of the unwinder’s C functions repeating. That version bailed in a less pleasant manner than the current abort.

- It may also be worth simplifying your example so that it is your
executable that throws the exception and not a function in the libc++
library itself, which in this case I believe would be in the libc++
shared object itself.

Indeed, this was a useful thing to try — if I throw invalid_argument directly, it is caught properly! That explains why my actual project hadn’t exhibited this problem before, as I was fairly certain it was throwing at least a handful of exceptions but not aborting (until I had a stoul call throw, which I have coded around… but having a broken toolchain is less than ideal).

So the problem is when libc++ throws…? That implies I’m building the library wrong?

I'm making the assumption you are building natively here. I have
cross-compiled from x86_64 to AArch64 before but I haven't done it for
Arm. The solution I found needed standalone builds of the libraries.
If you can share a bit more about how you are trying to build we may
be able to spot something wrong?

I am building natively on my ODROID HC-1 (8 32-bit cores) running Ubuntu 16.04. I also have a Wandboard Quad (4 32-bit cores) running Ubuntu 16.04. Both exhibit the same problem, but that’s not terribly surprising since I followed the same build path for the toolchain on each host. The programs are cmake projects with makefiles, but given the simple repro case above, that seems unrelated. The programs (my full app and the test case) work fine on my OSX host using the Xcode toolchain.

I followed these steps with a couple of caveats:
  * used the (mostly) working clang toolchain I have in place (7.0.0rc2)
  * removed the -mcpu options (don’t know what valid list is)
  * had to prefix sub-project names with llvm/projects/*
  * created build directory in llvm/ (don’t normally like to do that)
  * ninja unwind (no lib prefix)
  * also tried using install-* target; built test program with paths to build dir as well as to the installed locations

Still doesn’t catch the stoul exception.

I followed these steps with a couple of caveats:
  * used the (mostly) working clang toolchain I have in place (7.0.0rc2)
  * removed the -mcpu options (don’t know what valid list is)
  * had to prefix sub-project names with llvm/projects/*
  * created build directory in llvm/ (don’t normally like to do that)
  * ninja unwind (no lib prefix)
  * also tried using install-* target; built test program with paths to build dir as well as to the installed locations

Still doesn’t catch the stoul exception.

I would suggest that you:

1. Use ldd to see exactly what shared objects are loaded by your program, and confirm that those are the ones you expect.
2. Run `nm --demangle` on libc++, libc++abi, libunwind and your program and look for the typeinfo of your exception class.
3. When building libcxxabi and libunwind, pass `-v` to ninja to see all the compilation commands, and then modify a command to preprocess `-E` the file instead and see exactly which headers are used to build. Confirm that you’re building libc++abi and libunwind against the correct headers.

Maybe you’ll find out that the exception class has typeinfo in more than one place, which could lead to the exception not being caught (if it’s thrown with a typeinfo different from the one used to catch). I’ve debugged very similar problems like that in the past. Another option is that your libunwind/libc++abi are somehow built against different headers or in a different configuration than you’re expecting, and that can lead to trouble (we’ve had one like that during the LLVM 7 release).

Louis

I already checked ldd and the paths are as I expect... but I was surprised to see gcc_s in the list. Is that expected?

It is only the std exceptions thrown from libc++ that seem to have a problem, so doesn’t that imply that the lib is internally inconsistent (which an include path problem couldn’t cause?).

Okay, we can mark this one as solved, but I’d like to generate a pair of requests/suggestions based on it. First though, the fix...

Once I realized that the libc++ library itself seemed to not be throwing/catching properly, I went back and looked at the build log for it. I was newly armed with the knowledge that clang and gcc generate different exception related information. I realized that the environment I had used to build the libraries was the same one I had used to build clang from scratch… using gcc. The $CC/$CXX env vars were unset and therefore the cmake/ninja build system picked up that gcc should be used, and as a result libc++ was being built with a compiler that generates incompatible unwind tables (or whatever the relevant data structures are called). Once I rebuilt the libraries with clang the repro case and my project started catching exceptions thrown from libc++.

Thanks for your help, Louis and Peter. It would have taken me a long time to figure that out without your replies.

Now, my suggestions…

1) Clearly the situation around the creation of unwind-tables, et al. is something of a muddle. Am I mistaken in thinking that libraries built with different compilers are going to have this problem? Seems like a subtle change in the ABI. Expecting that it should get fixed seems like asking for a lot, but it would be *really* nice if something in the environment could examine all of the various modules involved in launching a process, and determine if they have incompatible exception handling schemes, and emit a warning. Ideally this would be the linker or the runtime itself. Perhaps only in debug builds. Using ldd to examine library dependencies is pretty common, so that would be an effective place too (but much slower to roll out).

2) Building the toolchain seems fraught with peril, and the documentation provides some guidance but with too many options and nothing definitive. I understand that the deployment matrix for this whole ecosystem is impressively enormous, but some kind of map like Louis alluded to would be most helpful. What he suggested was filling out the whole matrix, but I don’t think that’s necessary. Instead, some sort of official page where known working paths to success can be recorded and maintained — ideally in the form of scripts that can be downloaded, tweaked if needed, and run. If in the form of scripts then there is the potential to do automated testing (admittedly challenging given the purpose of covering a wide array of platforms). Weird hacky solutions would *not* be listed there, just proper/intended uses of the toolchain build system to get to a working toolchain. I wasted too much time on stackoverflow/etc trolling through various people’s attempts and hacks to get the last ~8 years worth of versions running.

Okay, we can mark this one as solved, but I’d like to generate a pair of requests/suggestions based on it. First though, the fix...

Once I realized that the libc++ library itself seemed to not be throwing/catching properly, I went back and looked at the build log for it. I was newly armed with the knowledge that clang and gcc generate different exception related information. I realized that the environment I had used to build the libraries was the same one I had used to build clang from scratch… using gcc. The $CC/$CXX env vars were unset and therefore the cmake/ninja build system picked up that gcc should be used, and as a result libc++ was being built with a compiler that generates incompatible unwind tables (or whatever the relevant data structures are called). Once I rebuilt the libraries with clang the repro case and my project started catching exceptions thrown from libc++.

Thanks for your help, Louis and Peter. It would have taken me a long time to figure that out without your replies.

That's good to hear.

Now, my suggestions…

1) Clearly the situation around the creation of unwind-tables, et al. is something of a muddle. Am I mistaken in thinking that libraries built with different compilers are going to have this problem? Seems like a subtle change in the ABI. Expecting that it should get fixed seems like asking for a lot, but it would be *really* nice if something in the environment could examine all of the various modules involved in launching a process, and determine if they have incompatible exception handling schemes, and emit a warning. Ideally this would be the linker or the runtime itself. Perhaps only in debug builds. Using ldd to examine library dependencies is pretty common, so that would be an effective place too (but much slower to roll out).

The Arm unwinding tables themselves should be compatible between
compilers [*] so it should be possible to throw an exception through
libraries that have been compiled by different compilers. I'm aware
that there is a incompatibility between the unwinder and the c++ abi
library i.e. you need to have one combination of (libc++abi,
libunwind) and (libsupc++, libgcc-s) in the program. I guess it could
be possible to detect these incompatibilities with cmake, but that is
way beyond my cmake ability.

[*] The Arm exception tables are standardised in

2) Building the toolchain seems fraught with peril, and the documentation provides some guidance but with too many options and nothing definitive. I understand that the deployment matrix for this whole ecosystem is impressively enormous, but some kind of map like Louis alluded to would be most helpful. What he suggested was filling out the whole matrix, but I don’t think that’s necessary. Instead, some sort of official page where known working paths to success can be recorded and maintained — ideally in the form of scripts that can be downloaded, tweaked if needed, and run. If in the form of scripts then there is the potential to do automated testing (admittedly challenging given the purpose of covering a wide array of platforms). Weird hacky solutions would *not* be listed there, just proper/intended uses of the toolchain build system to get to a working toolchain. I wasted too much time on stackoverflow/etc trolling through various people’s attempts and hacks to get the last ~8 years worth of versions running.

I share your pain here. With me it was trying to cross-compile and
test compiler-rt. However when I tried to write it up [**] it was way
more difficult that I thought. I found that I could precisely describe
something that worked on the OS I was using, the Arm gcc toolchain I
had downloaded, the version of clang etc. it wouldn't have been much
use to someone starting from a different place and in a different
time. I think the result was probably a bit too general and hand wavy.

Perhaps a LLVM build recipes and gotchas page would be useful.

[**] https://llvm.org/docs/HowToCrossCompileBuiltinsOnArm.html

Peter