llvm-toolchain-3.8 on lower arm targets, specifically Debian armel and Raspbian.

llvm-toolchain-3.8 seems to have problems on debian armel and raspbian.

On raspbian it builds but our armv7 contamination checker blocked it from entering the repo. Further investigation showed that "compiler-rt" was being built with -march=armv7 . I was able to remove the -march with some build-system hacker but then I got a failure on
projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S

[ 8%] Building ASM object projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins-armhf.dir/arm/sync_fetch_and_add_4.S.o
cd /llvm-toolchain-3.8-3.8/build-llvm/projects/compiler-rt/lib/builtins && /usr/bin/gcc-5 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/llvm-toolchain-3.8-3.8/build-llvm/projects/compiler-rt/lib/builtins -I/llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins -I/llvm-toolchain-3.8-3.8/build-llvm/include -I/llvm-toolchain-3.8-3.8/include -fPIC -O2 -g -DNDEBUG -mfloat-abi=hard -o CMakeFiles/clang_rt.builtins-armhf.dir/arm/sync_fetch_and_add_4.S.o -c /llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S
/llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S: Assembler messages:
/llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S:20: Error: selected processor does not support `dmb' in Thumb mode
/llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S:20: Error: selected processor does not support `ldrex r0,[r12]' in Thumb mode
/llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S:20: Error: cannot honor width suffix -- `add r2,r0,r1'
/llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S:20: Error: selected processor does not support `strex r3,r2,[r12]' in Thumb mode
/llvm-toolchain-3.8-3.8/projects/compiler-rt/lib/builtins/arm/sync_fetch_and_add_4.S:20: Error: selected processor does not support `dmb' in Thumb mode
projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins-armhf.dir/build.make:1097: recipe for target 'projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins-armhf.dir/arm/sync_fetch_and_add_4.S.o' failed
make[4]: *** [projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins-armhf.dir/arm/sync_fetch_and_add_4.S.o] Error 1
make[4]: Leaving directory '/llvm-toolchain-3.8-3.8/build-llvm'
CMakeFiles/Makefile2:12146: recipe for target 'projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins-armhf.dir/all' failed
make[3]: *** [projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins-armhf.dir/all] Error 2
make[3]: *** Waiting for unfinished jobs...

I traced this back to code in compiler-rt/lib/builtins/arm/sync-ops.h . The comments talk about both arm mode and thumb2 mode code but I only actually see thumb2 code in the file. I'm kinda stuck here.

I expect this would be a problem for armel too (though it may go unnoticed due to the fact the armel autobuilders now run on armv7 hardware and afaict Debian armel unlike raspbian doesn't have any contamination checking in place) but the build doesn't even get that far. It fails earlier with.

Hi Peter,

I traced this back to code in compiler-rt/lib/builtins/arm/sync-ops.h . The
comments talk about both arm mode and thumb2 mode code but I only actually
see thumb2 code in the file. I'm kinda stuck here.

Yes, it looks like we'd need to conditionally compile these functions
in ARM mode and use the v6 barrier instead of dmb ("mcr p15, #0, r0,
c7, c10, #5" I believe) to support the ARM1176JZF-S in RPi. You'd
probably also want the build system to use an explicit -march=armv6 or
something so you're not at the mercy of how the host compiler was
configured.

I suspect you're the first person to try and get compiler-rt going for
pre-v7 hardware, which most of us prefer to pretend never happened
most of the time.

I expect this would be a problem for armel too (though it may go unnoticed
due to the fact the armel autobuilders now run on armv7 hardware and afaict
Debian armel unlike raspbian doesn't have any contamination checking in
place) but the build doesn't even get that far. It fails earlier with.

I'm not even sure where we'd begin for compiler-rt on armel, which
looks like it's supposed to work back to armv4t. That version doesn't
even *have* basic synchronization primitives, though I think I heard
the Linux kernel emulates them at some special restartable address?

The first step would probably be to find out where that is and make
compiler-rt (and/or LLVM directly) use it.

/«PKGBUILDDIR»/include/llvm/Support/ThreadPool.h: In member function
'std::shared_future<void> llvm::ThreadPool::async(Function&&, Args&&
...)':

I think I encountered similar recently: as I recall libstdc++ just
doesn't support the C++11 thread features LLVM needs on such an old
architecture. I think you'd have to add support before you could build
LLVM (or maybe bootstrap libc++ support). It's likely to be a hard
road either way.

Cheers.

Tim.

Hi Peter,

I traced this back to code in compiler-rt/lib/builtins/arm/sync-ops.h . The
comments talk about both arm mode and thumb2 mode code but I only actually
see thumb2 code in the file. I'm kinda stuck here.

Yes, it looks like we'd need to conditionally compile these functions
in ARM mode and use the v6 barrier instead of dmb ("mcr p15, #0, r0,
c7, c10, #5" I believe) to support the ARM1176JZF-S in RPi. You'd
probably also want the build system to use an explicit -march=armv6 or
something so you're not at the mercy of how the host compiler was
configured.

I suspect you're the first person to try and get compiler-rt going for
pre-v7 hardware, which most of us prefer to pretend never happened
most of the time.

I am bad, and I should feel bad... I had it working for v4/v4t and v6m, but then never set up a buildbot to make sure it stayed working :confused: (also may have failed to upstream the (admittedly simple) patches for it).

I expect this would be a problem for armel too (though it may go unnoticed
due to the fact the armel autobuilders now run on armv7 hardware and afaict
Debian armel unlike raspbian doesn't have any contamination checking in
place) but the build doesn't even get that far. It fails earlier with.

I'm not even sure where we'd begin for compiler-rt on armel, which
looks like it's supposed to work back to armv4t. That version doesn't
even *have* basic synchronization primitives, though I think I heard
the Linux kernel emulates them at some special restartable address?

Yes, it does:

   typedef int (__kernel_cmpxchg_t) (int old, int new, int *ptr);
   #define __kernel_cmpxchg (*(__kernel_cmpxchg_t *) 0xffff0fc0)

   typedef void (__kernel_dmb_t) (void);
   #define __kernel_dmb (*(__kernel_dmb_t *) 0xffff0fa0)

Though there is still an open question of what to do for baremetal, which of course doesn't have a kernel (though that's a separate issue from Peter's problems here).

Jon

Thank you very much for your reply.

Can you clarify what exactly compiler-rt is and how important it is to the stack as a whole. It seems that llvm-toolchain-3.7 built successfully on raspbian without building compiler-rt while llvm-toolchain-3.8 built an armv7 compiler-rt (and hence got blocked from entering the raspbian repos).

Can you clarify what exactly compiler-rt is and how important it is to the
stack as a whole. It seems that llvm-toolchain-3.7 built successfully on
raspbian without building compiler-rt while llvm-toolchain-3.8 built an
armv7 compiler-rt (and hence got blocked from entering the raspbian repos).

Compiler-rt is the equivalent of libgcc, and Clang can use the
existing host's libgcc quite happily so it's really not that important
unless you're trying to build a GNU-free environment for whatever
reason.

Cheers.

Tim.

Thanks

Can you tell me how I would go about telling the cmake buildsystem (which is what the Debian packaging currently seems to use) that I want it to build without compiler-rt?

If you don't need/want the various Sanitizer runtimes (e.g. you don't
support sanitizers or already have versions provided with GCC) then
it's as easy as not downloading compiler-rt or removing it from the
projects/ directory before running CMake. The build should carry on
quite happily without it.

If you do need the sanitizers from compiler-rt, you can disable the
bits that are causing problems right now with "cmake
-DCOMPILER_RT_BUILD_BUILTINS=OFF <other options>". Of course, the
sanitizers themselves may suffer from similar issues -- you just won't
know until you get there.

Cheers.

Tim.

Thanks, -DCOMPILER_RT_BUILD_BUILTINS=OFF did the trick and I got a successful build for raspbian that passed my armv7 contamination checker script (which is not 100% foolproof but it's the best we have) and uploaded it to raspbian stretch.