Cross compiling with clang (and sysroot support)

Clang Frontend
1

Hi there,

I'm investigating adding sysroot functionality to cmake

http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/6613/focus=6702

Does anyone know what version of clang the --sysroot flag was introduced or
if it was 'a really long time ago'?

I tried to figure out how to cross compile something with clang.

http://clang.llvm.org/UniversalDriver.html

says that clang is inherently a cross compiler, so I guess that means I
don't need a separate binary, as I do with g++ based cross compilers?

I tried using clang with the arguments in this page:

LLVM Cross-Compiler - OSDev Wiki

without success:

stephen@hal:~/tmp$ cat main.c

int main(int argc, char **argv)
{
  return 0;
}

stephen@hal:~/tmp$ clang -target armv7--eabi -mcpu=cortex-a9 main.c
gcc: warning: ‘-mcpu=’ is deprecated; use ‘-mtune=’ or ‘-march=’ instead
/tmp/main-EitGh6.s: Assembler messages:
/tmp/main-EitGh6.s:1: Error: unknown pseudo-op: `.syntax'
/tmp/main-EitGh6.s:2: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:3: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:4: Error: unknown pseudo-op: `.fpu'
/tmp/main-EitGh6.s:5: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:6: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:7: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:8: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:9: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:10: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:11: Error: unknown pseudo-op: `.eabi_attribute'
/tmp/main-EitGh6.s:18: Error: too many memory references for `sub'
/tmp/main-EitGh6.s:19: Error: expecting operand after ','; got nothing
/tmp/main-EitGh6.s:20: Error: invalid char '[' beginning operand 2 `[sp'
/tmp/main-EitGh6.s:21: Error: invalid char '[' beginning operand 2 `[sp'
/tmp/main-EitGh6.s:22: Error: invalid char '[' beginning operand 2 `[sp'
/tmp/main-EitGh6.s:23: Error: invalid char '[' beginning operand 2 `[sp'
/tmp/main-EitGh6.s:24: Error: too many memory references for `mov'
/tmp/main-EitGh6.s:25: Error: invalid char '[' beginning operand 2 `[sp]'
/tmp/main-EitGh6.s:26: Error: too many memory references for `add'
/tmp/main-EitGh6.s:27: Error: no such instruction: `bx lr'
clang: error: assembler (via gcc) command failed with exit code 1 (use -v to
see invocation)
                                                                                                                                                                                                                   
stephen@hal:~/tmp$ clang --version
Ubuntu clang version 3.2-1~exp5ubuntu2 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: x86_64-pc-linux-gnu
Thread model: posix

Am I missing something?

Thanks,

Steve.

2

Hi Stephen,

I'm afraid I'm not sure when Clang started accepting sysroot; someone
else will probably have to answer that.

stephen@hal:~/tmp$ clang -target armv7--eabi -mcpu=cortex-a9 main.c
gcc: warning: ‘-mcpu=’ is deprecated; use ‘-mtune=’ or ‘-march=’ instead
/tmp/main-EitGh6.s: Assembler messages:
/tmp/main-EitGh6.s:1: Error: unknown pseudo-op: `.syntax'

This is because Clang has decided to call (probably) the host's
assembler, /usr/bin/as, to assemble the ARM code it just produced.
That's obviously not going to work very well if your host is x86.

It would actually use the --sysroot option (amongst other things) to
try and find appropriate headers/assembler/linker/... if it was
present. Whether it could actually find them or not is another matter,
of course. The code is rather heuristic.

Tim.

3

Tim Northover wrote:

Hi Stephen,

I'm afraid I'm not sure when Clang started accepting sysroot; someone
else will probably have to answer that.

stephen@hal:~/tmp$ clang -target armv7--eabi -mcpu=cortex-a9 main.c
gcc: warning: ‘-mcpu=’ is deprecated; use ‘-mtune=’ or ‘-march=’ instead
/tmp/main-EitGh6.s: Assembler messages:
/tmp/main-EitGh6.s:1: Error: unknown pseudo-op: `.syntax'

This is because Clang has decided to call (probably) the host's
assembler, /usr/bin/as, to assemble the ARM code it just produced.
That's obviously not going to work very well if your host is x86.

It would actually use the --sysroot option (amongst other things) to
try and find appropriate headers/assembler/linker/... if it was
present. Whether it could actually find them or not is another matter,
of course. The code is rather heuristic.

Thanks for the information. I don't really know where to find an appropriate
assembler to point clang to. I have several gcc based toolchains on this
computer, so I guess I should point to the as executable in one of those?
How would I specify it?

/usr/bin/arm-linux-gnueabi-gcc-4.7 main.c

'Just Works'. /usr/bin/arm-linux-gnueabi-as also exists. Should I use that?
All I need is some way to cross compile something for some platform, so if I
can invoke clang in a way that it will use that cross-as, that would be
great.

Thanks,

Steve.

4

Hi Steve,

/usr/bin/arm-linux-gnueabi-gcc-4.7 main.c

'Just Works'. /usr/bin/arm-linux-gnueabi-as also exists.

In that case, I'd suggest using "-target arm-linux-gnueabi" as a
starting-point. Clang does use this target in its search for an
appropriate assembler and linker. I think I concluded that it'll find
it if arm-linux-gnueabi-as is on the path, but I'm not certain.

Other than that, the key point is where the ARM libraries and headers
are. There's a chance Clang will find them by itself with the correct
-target option. If not, the --sysroot option is what you'll have to
use to give it a hint.

All I need is some way to cross compile something for some platform, so if I
can invoke clang in a way that it will use that cross-as, that would be
great.

A parallel option would be to add "-integrated-as" as an option. Then
Clang does the assembly step itself. I didn't mention it because you'd
still need the linker to produce a working binary.

Cheers.

Tim.

5

Tim Northover wrote:

Hi Steve,

/usr/bin/arm-linux-gnueabi-gcc-4.7 main.c

'Just Works'. /usr/bin/arm-linux-gnueabi-as also exists.

In that case, I'd suggest using "-target arm-linux-gnueabi" as a
starting-point. Clang does use this target in its search for an
appropriate assembler and linker. I think I concluded that it'll find
it if arm-linux-gnueabi-as is on the path, but I'm not certain.

Other than that, the key point is where the ARM libraries and headers
are. There's a chance Clang will find them by itself with the correct
-target option. If not, the --sysroot option is what you'll have to
use to give it a hint.

Thanks, that is progress at least:

stephen@hal:/tmp$ clang -target arm-linux-gnueabi main.c
/usr/bin/arm-linux-gnueabi-ld: cannot find crt1.o: No such file or
directory
/usr/bin/arm-linux-gnueabi-ld: cannot find crti.o: No such file or
directory
/usr/bin/arm-linux-gnueabi-ld: cannot find crtbegin.o: No such file or
directory
/usr/bin/arm-linux-gnueabi-ld: cannot find -lgcc
/usr/bin/arm-linux-gnueabi-ld: cannot find -lgcc_s
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

Running

locate crtbegin.o | grep arm

reveals

/usr/lib/gcc-cross/arm-linux-gnueabi/4.7/crtbegin.o

which looks appropriate. But what should I pass in the --sysroot?

I tried using /usr/lib/gcc-cross/arm-linux-gnueabi/4.7/ and also all parent
directories of it, with the same result as above each time.

All I need is some way to cross compile something for some platform, so
if I can invoke clang in a way that it will use that cross-as, that would
be great.

A parallel option would be to add "-integrated-as" as an option. Then
Clang does the assembly step itself. I didn't mention it because you'd
still need the linker to produce a working binary.

Right. Using that I get a linker error because it uses the wrong linker:

stephen@hal:/tmp$ clang -target armv7--eabi -integrated-as main.c
/usr/bin/ld: unknown architecture of input file `/tmp/main-CFNTHI.o' is
incompatible with i386:x86-64 output
collect2: error: ld returned 1 exit status
clang: error: linker (via gcc) command failed with exit code 1 (use -v to
see invocation)

Thanks,

Steve.

6

Hi Steve,

/usr/lib/gcc-cross/arm-linux-gnueabi/4.7/crtbegin.o

which looks appropriate. But what should I pass in the --sysroot?

Unfortunately, it looks like this is odd enough that Clang won't
detect it by itself no matter what sysroot you specify. In particular
it doesn't know anything about "gcc-cross" as a possible location.

Four options come to mind:
+ Fix clang so that it does try to look in gcc-cross.
+ Mess around with your filesystem so that it resembles what Clang is
searching for (e.g. symlink /usr/lib/gcc-cross/arm-linux-gnueabi to
/usr/lib/gcc/arm-linux-gnueabi; other things may be needed too but
there's not enough information to say yet).
+ Use Clang for the compile step ("clang -target ... -c") and gcc for
the link step ("arm-linux-gnueabi-gcc main.o -omain"). This is still
dodgy w.r.t. headers, if clang is trying to use the ones from
/usr/include, but has a good chance of working anyway.
+ Pass these special options and files yourself on the Clang command-line.

Tim.

7

Tim Northover wrote:

Hi Steve,

/usr/lib/gcc-cross/arm-linux-gnueabi/4.7/crtbegin.o

which looks appropriate. But what should I pass in the --sysroot?

Unfortunately, it looks like this is odd enough that Clang won't
detect it by itself no matter what sysroot you specify. In particular
it doesn't know anything about "gcc-cross" as a possible location.

Four options come to mind:
+ Fix clang so that it does try to look in gcc-cross.
+ Mess around with your filesystem so that it resembles what Clang is
searching for (e.g. symlink /usr/lib/gcc-cross/arm-linux-gnueabi to
/usr/lib/gcc/arm-linux-gnueabi; other things may be needed too but
there's not enough information to say yet).
+ Use Clang for the compile step ("clang -target ... -c") and gcc for
the link step ("arm-linux-gnueabi-gcc main.o -omain"). This is still
dodgy w.r.t. headers, if clang is trying to use the ones from
/usr/include, but has a good chance of working anyway.
+ Pass these special options and files yourself on the Clang command-line.

Option 5: Use a different toolchain :slight_smile:

I have two other toolchains lying around. locate crtbegin.o finds a bunch of
stuff including this stuff from a raspberry pi toolchain:

/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-linux-
gnueabihf/4.7.2/crtbegin.o
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-linux-
gnueabihf/4.7.2/arm-linux-gnueabi/crtbegin.o

I using it:

stephen@hal:/tmp$ clang -v -target arm-linux-gnueabi --sysroot
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/ main.c
Ubuntu clang version 3.2-1~exp5ubuntu2 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: arm--linux-gnueabi
Thread model: posix
"/usr/bin/clang" -cc1 -triple armv4t--linux-gnueabi -S -disable-free -
disable-llvm-verifier -main-file-name main.c -mrelocation-model static -
mdisable-fp-elim -fmath-errno -mconstructor-aliases -target-abi aapcs-linux
-target-cpu arm7tdmi -mfloat-abi soft -target-feature +soft-float-abi -
target-linker-version 2.23.1 -momit-leaf-frame-pointer -v -resource-dir
/usr/bin/../lib/clang/3.2 -isysroot /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/ -fmodule-cache-path /var/tmp/clang-module-cache -internal-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//usr/local/include -internal-
isystem /usr/bin/../lib/clang/3.2/include -internal-isystem
/usr/include/clang/3.2/include/ -internal-externc-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//usr/include/x86_64-linux-gnu
-internal-externc-isystem /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf//usr/include/x86_64-linux-gnu -internal-externc-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//usr/include -fno-dwarf-
directory-asm -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length
211 -mstackrealign -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-
option -fcolor-diagnostics -o /tmp/main-eBabK4.s -x c main.c
clang -cc1 version 3.2 based upon LLVM 3.2svn default target x86_64-pc-
linux-gnu
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf//usr/local/include"
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf//usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf//usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf//usr/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/include/clang/3.2/include
End of search list.
"/usr/bin/arm-linux-gnueabi-as" -mfloat-abi=softfp -o /tmp/main-AvS3Kc.o
/tmp/main-eBabK4.s
"/usr/bin/arm-linux-gnueabi-ld" --sysroot=/home/stephen/rpi/gcc-4.7-linaro-
rpi-gnueabihf/ -z relro -X --hash-style=gnu --build-id --eh-frame-hdr -m
armelf_linux_eabi -dynamic-linker /lib/ld-linux.so.3 -o a.out crt1.o crti.o
crtbegin.o -L/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//lib /tmp/main-
AvS3Kc.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -
lgcc_s --no-as-needed crtend.o crtn.o
/usr/bin/arm-linux-gnueabi-ld: cannot find crt1.o: No such file or directory
/usr/bin/arm-linux-gnueabi-ld: cannot find crti.o: No such file or directory
/usr/bin/arm-linux-gnueabi-ld: cannot find crtbegin.o: No such file or
directory
/usr/bin/arm-linux-gnueabi-ld: cannot find -lgcc
/usr/bin/arm-linux-gnueabi-ld: cannot find -lgcc_s
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

I find it odd that it considers

/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//usr/include/x86_64-linux-
gnu

before finding it is non-existant. It's mixing triples.

Thanks,

Steve.

8

Stephen Kelly <steveire@...> writes:

I find it odd that it considers

/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//usr/include/x86_64-linux-
gnu

before finding it is non-existant. It's mixing triples.

Ping? Any further help on this issue?

http://thread.gmane.org/gmane.comp.compilers.clang.devel/28960/focus=29153

Thanks,

Steve.

9

Stephen Kelly wrote:

Stephen Kelly <steveire@...> writes:

I find it odd that it considers

/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//usr/include/x86_64-

linux-

gnu

before finding it is non-existant. It's mixing triples.

Ping? Any further help on this issue?

http://thread.gmane.org/gmane.comp.compilers.clang.devel/28960/focus=29153

I've seen commits relating to cross compiling, so I'm sure others are
somehow able to cross compile using clang.

Can anyone else help here to get a trivial program cross compiled?

Thanks,

Steve.

10

Stephen Kelly wrote:

Stephen Kelly wrote:

Stephen Kelly <steveire@...> writes:

I find it odd that it considers

/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf//usr/include/x86_64-

linux-

gnu

before finding it is non-existant. It's mixing triples.

Ping? Any further help on this issue?

http://thread.gmane.org/gmane.comp.compilers.clang.devel/28960/focus=29153

I've seen commits relating to cross compiling, so I'm sure others are
somehow able to cross compile using clang.

Can anyone else help here to get a trivial program cross compiled?

I found that if I specify paths with -B I can make it find the crti.o etc
files. I didn't find documentation for -B, so I don't really know what it
does. I just found it here:

http://blog.boceto.fr/2012/05/03/the-universal-compiler-used-with-cmake/

I have to specify it several times. I think I'm just missing the right
option to clang to tell it where to look for stuff. After specifying all
those paths, I also have to specify some -L paths for gcc, and then I get a
fresh error (which might be only a result of hacking around with these
paths):

stephen@hal:/tmp$ clang -v -target arm-linux-gnueabi -
B/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-linux-
gnueabihf/4.7.2/ -B/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/arm-linux-
gnueabihf/libc/usr/lib/arm-linux-gnueabihf/ -L/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/arm-linux-gnueabihf/lib -L /home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/ main.c
Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: arm--linux-gnueabi
Thread model: posix
"/usr/bin/clang" -cc1 -triple armv4t--linux-gnueabi -S -disable-free -
disable-llvm-verifier -main-file-name main.c -mrelocation-model static -
mdisable-fp-elim -fmath-errno -mconstructor-aliases -target-abi aapcs-linux
-target-cpu arm7tdmi -mfloat-abi soft -target-feature +soft-float-abi -
target-linker-version 2.23.2 -momit-leaf-frame-pointer -v -resource-dir
/usr/bin/../lib/clang/3.2 -fmodule-cache-path /var/tmp/clang-module-cache -
internal-isystem /usr/local/include -internal-isystem
/usr/bin/../lib/clang/3.2/include -internal-isystem
/usr/include/clang/3.2/include/ -internal-externc-isystem
/usr/include/x86_64-linux-gnu -internal-externc-isystem /usr/include/x86_64-
linux-gnu -internal-externc-isystem /usr/include -fno-dwarf-directory-asm -
fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length 211 -
mstackrealign -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-option
-fcolor-diagnostics -o /tmp/main-87BC7C.s -x c main.c
clang -cc1 version 3.2 based upon LLVM 3.2svn default target x86_64-pc-
linux-gnu
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/usr/include/clang/3.2/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.
"/usr/bin/arm-linux-gnueabi-as" -mfloat-abi=softfp -o /tmp/main-1pJbtV.o
/tmp/main-87BC7C.s
"/usr/bin/arm-linux-gnueabi-ld" -z relro -X --hash-style=gnu --build-id --
eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /lib/ld-linux.so.3 -o
a.out /home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/arm-linux-
gnueabihf/libc/usr/lib/arm-linux-gnueabihf/crt1.o /home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/arm-linux-
gnueabihf/crti.o /home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-
linux-gnueabihf/4.7.2/crtbegin.o -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/arm-linux-gnueabihf/lib -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/ -L/usr/lib/../lib32 -L/lib -
L/usr/lib /tmp/main-1pJbtV.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -
lgcc --as-needed -lgcc_s --no-as-needed /home/stephen/rpi/gcc-4.7-linaro-
rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/crtend.o
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/arm-linux-
gnueabihf/libc/usr/lib/arm-linux-gnueabihf/crtn.o
/usr/bin/arm-linux-gnueabi-ld: error: a.out uses VFP register arguments,
/tmp/main-1pJbtV.o does not
/usr/bin/arm-linux-gnueabi-ld: failed to merge target specific data of file
/tmp/main-1pJbtV.o
/usr/bin/arm-linux-gnueabi-ld: error: a.out uses VFP register arguments,
/usr/arm-linux-gnueabi/lib/libc_nonshared.a(elf-init.oS) does not
/usr/bin/arm-linux-gnueabi-ld: failed to merge target specific data of file
/usr/arm-linux-gnueabi/lib/libc_nonshared.a(elf-init.oS)
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

Thanks,

Steve.

11

clang looks for as and ld helper programs in the paths specified with -B
followed by $PATH. It is first looking for an executable file named
"${triple}-as" / "${triple-ld}" and then for "as" / "ld". So in your
case, it should just work, if you invoke clang with -target
arm-linux-gnueabi or as arm-linux-gnueabi-clang.

The --sysroot option is used for finding header files and libraries. For
linking, it depends on ld implementing sysroot support itself.

Joerg

12

Joerg Sonnenberger wrote:

Thanks for the information. I don't really know where to find an
appropriate assembler to point clang to. I have several gcc based
toolchains on this computer, so I guess I should point to the as
executable in one of those? How would I specify it?

/usr/bin/arm-linux-gnueabi-gcc-4.7 main.c

'Just Works'. /usr/bin/arm-linux-gnueabi-as also exists. Should I use
that? All I need is some way to cross compile something for some
platform, so if I can invoke clang in a way that it will use that
cross-as, that would be great.

clang looks for as and ld helper programs in the paths specified with -B
followed by $PATH. It is first looking for an executable file named
"${triple}-as" / "${triple-ld}" and then for "as" / "ld". So in your
case, it should just work, if you invoke clang with -target
arm-linux-gnueabi or as arm-linux-gnueabi-clang.

Thanks, your mail made me double-check my triples. In

http://thread.gmane.org/gmane.comp.compilers.clang.devel/28960/focus=29153

I was mixing ...hf with non-hf triples.

So, now I get a little further:

stephen@hal:/tmp$ clang -v -target arm-linux-gnueabihf --sysroot
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf main.c
Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: arm--linux-gnueabihf
Thread model: posix
"/usr/bin/clang" -cc1 -triple armv4t--linux-gnueabihf -S -disable-free -
disable-llvm-verifier -main-file-name main.c -mrelocation-model static -
mdisable-fp-elim -fmath-errno -mconstructor-aliases -fuse-init-array -
target-abi aapcs-linux -target-cpu cortex-a8 -mfloat-abi hard -target-
feature +vfp3 -target-feature +d16 -target-feature -neon -target-linker-
version 2.23.2 -momit-leaf-frame-pointer -v -resource-dir
/usr/bin/../lib/clang/3.2 -isysroot /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf -fmodule-cache-path /var/tmp/clang-module-cache -internal-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/usr/local/include -internal-
isystem /usr/bin/../lib/clang/3.2/include -internal-isystem
/usr/include/clang/3.2/include/ -internal-externc-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/usr/include/x86_64-linux-gnu
-internal-externc-isystem /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include/x86_64-linux-gnu -internal-externc-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/usr/include -fno-dwarf-
directory-asm -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length
211 -mstackrealign -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-
option -fcolor-diagnostics -o /tmp/main-ohNsrR.s -x c main.c
clang -cc1 version 3.2 based upon LLVM 3.2svn default target x86_64-pc-
linux-gnu
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/local/include"
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/include/clang/3.2/include
End of search list.
"/usr/bin/arm-linux-gnueabihf-as" -mfloat-abi=hard -o /tmp/main-m4RQDy.o
/tmp/main-ohNsrR.s
"/usr/bin/arm-linux-gnueabihf-ld" --sysroot=/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf -z relro -X --hash-style=gnu --build-id --eh-frame-hdr
-m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o a.out
crt1.o crti.o /home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-
linux-gnueabihf/4.7.2/crtbegin.o -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2 -L/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-
linux-gnueabihf/lib/../lib32 -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-linux-
gnueabihf/lib -L/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-
linux-gnueabihf/4.7.2/../../.. -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib /tmp/main-m4RQDy.o -lgcc --as-needed -lgcc_s --no-as-needed -
lc -lgcc --as-needed -lgcc_s --no-as-needed /home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/crtend.o crtn.o
/usr/bin/arm-linux-gnueabihf-ld: cannot find crt1.o: No such file or
directory
/usr/bin/arm-linux-gnueabihf-ld: cannot find crti.o: No such file or
directory
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

And if I specify the sysroot *and* the -B option, I can make it succeed:
stephen@hal:/tmp$ clang -v -target arm-linux-gnueabihf --sysroot
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf -B /usr/arm-linux-
gnueabihf/lib/ main.c
Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: arm--linux-gnueabihf
Thread model: posix
"/usr/bin/clang" -cc1 -triple armv4t--linux-gnueabihf -S -disable-free -
disable-llvm-verifier -main-file-name main.c -mrelocation-model static -
mdisable-fp-elim -fmath-errno -mconstructor-aliases -fuse-init-array -
target-abi aapcs-linux -target-cpu cortex-a8 -mfloat-abi hard -target-
feature +vfp3 -target-feature +d16 -target-feature -neon -target-linker-
version 2.23.2 -momit-leaf-frame-pointer -v -resource-dir
/usr/bin/../lib/clang/3.2 -isysroot /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf -fmodule-cache-path /var/tmp/clang-module-cache -internal-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/usr/local/include -internal-
isystem /usr/bin/../lib/clang/3.2/include -internal-isystem
/usr/include/clang/3.2/include/ -internal-externc-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/usr/include/x86_64-linux-gnu
-internal-externc-isystem /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include/x86_64-linux-gnu -internal-externc-isystem
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/usr/include -fno-dwarf-
directory-asm -fdebug-compilation-dir /tmp -ferror-limit 19 -fmessage-length
211 -mstackrealign -fno-signed-char -fobjc-runtime=gcc -fdiagnostics-show-
option -fcolor-diagnostics -o /tmp/main-x95tmT.s -x c main.c
clang -cc1 version 3.2 based upon LLVM 3.2svn default target x86_64-pc-
linux-gnu
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/local/include"
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/include/clang/3.2/include
End of search list.
"/usr/bin/arm-linux-gnueabihf-as" -mfloat-abi=hard -o /tmp/main-duFLkY.o
/tmp/main-x95tmT.s
"/usr/bin/arm-linux-gnueabihf-ld" --sysroot=/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf -z relro -X --hash-style=gnu --build-id --eh-frame-hdr
-m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o a.out
/usr/arm-linux-gnueabihf/lib/crt1.o /usr/arm-linux-gnueabihf/lib/crti.o
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-linux-
gnueabihf/4.7.2/crtbegin.o -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2 -L/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-
linux-gnueabihf/lib/../lib32 -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-linux-
gnueabihf/lib -L/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-
linux-gnueabihf/4.7.2/../../.. -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib /tmp/main-duFLkY.o -lgcc --as-needed -lgcc_s --no-as-needed -
lc -lgcc --as-needed -lgcc_s --no-as-needed /home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/crtend.o /usr/arm-
linux-gnueabihf/lib/crtn.o

With gcc it works with just the sysroot specified:
stephen@hal:/tmp$ /usr/bin/arm-linux-gnueabihf-gcc-4.7 -v --sysroot
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf main.c
Using built-in specs.
COLLECT_GCC=/usr/bin/arm-linux-gnueabihf-gcc-4.7
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro
4.7.3-1ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --
enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-
suffix=-4.7 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --with-gxx-include-
dir=/usr/arm-linux-gnueabihf/include/c++/4.7.3 --libdir=/usr/lib --enable-
nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-
libstdcxx-time=yes --enable-gnu-unique-object --disable-libitm --enable-
plugin --with-system-zlib --enable-objc-gc --with-cloog --enable-cloog-
backend=ppl --disable-cloog-version-check --disable-ppl-version-check --
enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-
arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --
disable-werror --enable-checking=release --build=x86_64-linux-gnu --
host=x86_64-linux-gnu --target=arm-linux-gnueabihf --program-prefix=arm-
linux-gnueabihf- --includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix
gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1)
COLLECT_GCC_OPTIONS='-v' '-march=armv7-a' '-mfloat-abi=hard' '-mfpu=vfpv3-
d16' '-mthumb' '-mtls-dialect=gnu'
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/cc1 -quiet -v -imultilib . -
imultiarch arm-linux-gnueabihf -isysroot /home/stephen/rpi/gcc-4.7-linaro-
rpi-gnueabihf main.c -quiet -dumpbase main.c -march=armv7-a -mfloat-abi=hard
-mfpu=vfpv3-d16 -mthumb -mtls-dialect=gnu -auxbase main -version -fstack-
protector -o /tmp/cczGygs4.s
GNU C (Ubuntu/Linaro 4.7.3-1ubuntu1) version 4.7.3 (arm-linux-gnueabihf)
        compiled by GNU C version 4.7.3, GMP version 5.0.5, MPFR version
3.1.1-p2, MPC version 0.9
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/local/include/arm-linux-gnueabihf"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include/arm-linux-gnueabihf"
ignoring nonexistent directory "/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/usr/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include-fixed
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/include
End of search list.
GNU C (Ubuntu/Linaro 4.7.3-1ubuntu1) version 4.7.3 (arm-linux-gnueabihf)
        compiled by GNU C version 4.7.3, GMP version 5.0.5, MPFR version
3.1.1-p2, MPC version 0.9
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: e45f40a8311fe21566e2c30ee2f2b3a6
COLLECT_GCC_OPTIONS='-v' '-march=armv7-a' '-mfloat-abi=hard' '-mfpu=vfpv3-
d16' '-mthumb' '-mtls-dialect=gnu'
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/bin/as -v -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -meabi=5
-o /tmp/cclqE2KN.o /tmp/cczGygs4.s
GNU assembler version 2.23.2 (arm-linux-gnueabihf) using BFD version (GNU
Binutils for Ubuntu) 2.23.2
COMPILER_PATH=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/:/usr/lib/gcc-
cross/arm-linux-gnueabihf/4.7/:/usr/lib/gcc-cross/arm-linux-
gnueabihf/:/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/:/usr/lib/gcc-
cross/arm-linux-gnueabihf/:/usr/lib/gcc-cross/arm-linux-
gnueabihf/4.7/../../../../arm-linux-gnueabihf/bin/
LIBRARY_PATH=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/:/usr/lib/gcc-
cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/lib/../lib/:/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/../lib/:/usr/lib/gcc-cross/arm-linux-
gnueabihf/4.7/../../../../arm-linux-
gnueabihf/lib/:/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/
COLLECT_GCC_OPTIONS='-v' '-march=armv7-a' '-mfloat-abi=hard' '-mfpu=vfpv3-
d16' '-mthumb' '-mtls-dialect=gnu'
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/collect2 --
sysroot=/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf --build-id --eh-
frame-hdr -dynamic-linker /lib/ld-linux-armhf.so.3 -X --hash-style=gnu --as-
needed -m armelf_linux_eabi -z relro /usr/lib/gcc-cross/arm-linux-
gnueabihf/4.7/../../../../arm-linux-gnueabihf/lib/../lib/crt1.o
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/lib/../lib/crti.o /usr/lib/gcc-cross/arm-linux-
gnueabihf/4.7/crtbegin.o -L/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7 -
L/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/lib/../lib -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/../lib -L/usr/lib/gcc-cross/arm-linux-
gnueabihf/4.7/../../../../arm-linux-gnueabihf/lib -
L/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib /tmp/cclqE2KN.o -lgcc --
as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-
needed /usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/crtend.o /usr/lib/gcc-
cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/lib/../lib/crtn.o

Should clang be looking in more places for the crtX.o files?

Thanks,

Steve.

13

It doesn't know the path of your GCC toolchain, so it doesn't find the
files. GCC has that path compiled in.

Joerg

14

Joerg Sonnenberger wrote:

Should clang be looking in more places for the crtX.o files?

It doesn't know the path of your GCC toolchain, so it doesn't find the
files. GCC has that path compiled in.

Ok, I guess that makes sense.

However, there's still a problem. It seems that the term 'sysroot' means
different things to clang and gcc.

When I use gcc --sysroot, the specified path is prepended to the header
search path and library search path.

stephen@hal:/tmp$ cat main.c

#include <zlib.h>

int main(int argc, char **argv)
{
   int compressionLevel;
   int nbytes;
   const unsigned char* data;
   unsigned long len;

   unsigned char* bazip;

   int res = compress2(bazip, &len, data, nbytes, compressionLevel);

   return 0;
}

$ /usr/bin/arm-linux-gnueabihf-gcc-4.7 -v --sysroot /home/stephen/rpi/rasp-
pi-rootfs/ -c main.c

<snip>
ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs/usr/local/include/arm-linux-gnueabihf"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include-fixed
/usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/include
/home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf
/home/stephen/rpi/rasp-pi-rootfs/usr/include
End of search list.
<snip>

I gather from this thread, that because gcc knows where to find the crtX.o
files etc, it doesn't need to be told where to find them. However, as clang
does not know where those files are, it needs to be told.

I also gather that the way to tell clang where those crtX.o files etc are is
by using the --sysroot option.

However, the problem is that when I invoked gcc above with the sysroot, the
path I passed was to a mounted raspbian image, which contains headers and
libraries for the raspberry pi. By specifying that, I can compile my main.c
which now needs to find the zlib header, and I can link to the correct zlib
library in the image.

If I have to use the sysroot option to tell clang where to find crtX.o
files, then I can't use it to tell clang where to find the zlib headers and
libraries. They are not in the same location as the crti.o etc.

Clang also does not search in the arch-specific include dir as gcc does, so
it doesn't find sys/types:

$ clang -v -target arm-linux-gnueabihf --sysroot /home/stephen/rpi/rasp-pi-
rootfs/ -B /usr/arm-linux-gnueabihf/lib/ main.c
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
#include "..." search starts here:
#include <...> search starts here:
/home/stephen/rpi/rasp-pi-rootfs//usr/local/include
/usr/include/clang/3.2/include
/home/stephen/rpi/rasp-pi-rootfs//usr/include
End of search list.
In file included from main.c:2:
In file included from /home/stephen/rpi/rasp-pi-
rootfs//usr/include/zlib.h:34:
/home/stephen/rpi/rasp-pi-rootfs//usr/include/zconf.h:424:14: fatal error:
'sys/types.h' file not found

If I help-out by manually supplying that extra include, it compiles, but the
linking step fails:

stephen@hal:/tmp$ clang -v -target arm-linux-gnueabihf --sysroot
/home/stephen/rpi/rasp-pi-rootfs/ -B /usr/arm-linux-gnueabihf/lib/ -
I/home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf/ main.c
Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: arm--linux-gnueabihf
Thread model: posix
"/usr/bin/clang" -cc1 -triple armv4t--linux-gnueabihf -S -disable-free -
disable-llvm-verifier -main-file-name main.c -mrelocation-model static -
mdisable-fp-elim -fmath-errno -mconstructor-aliases -target-abi aapcs-linux
-target-cpu cortex-a8 -mfloat-abi hard -target-feature +vfp3 -target-feature
+d16 -target-feature -neon -target-linker-version 2.23.2 -momit-leaf-frame-
pointer -v -resource-dir /usr/bin/../lib/clang/3.2 -I
/home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf/ -isysroot
/home/stephen/rpi/rasp-pi-rootfs/ -fmodule-cache-path /var/tmp/clang-module-
cache -internal-isystem /home/stephen/rpi/rasp-pi-rootfs//usr/local/include
-internal-isystem /usr/bin/../lib/clang/3.2/include -internal-isystem
/usr/include/clang/3.2/include/ -internal-externc-isystem
/home/stephen/rpi/rasp-pi-rootfs//usr/include/x86_64-linux-gnu -internal-
externc-isystem /home/stephen/rpi/rasp-pi-rootfs//usr/include/x86_64-linux-
gnu -internal-externc-isystem /home/stephen/rpi/rasp-pi-rootfs//usr/include
-fno-dwarf-directory-asm -fdebug-compilation-dir /tmp -ferror-limit 19 -
fmessage-length 211 -mstackrealign -fno-signed-char -fobjc-runtime=gcc -
fdiagnostics-show-option -fcolor-diagnostics -o /tmp/main-rK3Mvv.s -x c
main.c
clang -cc1 version 3.2 based upon LLVM 3.2svn default target x86_64-pc-
linux-gnu
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
#include "..." search starts here:
#include <...> search starts here:
/home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf
/home/stephen/rpi/rasp-pi-rootfs//usr/local/include
/usr/include/clang/3.2/include
/home/stephen/rpi/rasp-pi-rootfs//usr/include
End of search list.
"/usr/bin/arm-linux-gnueabihf-as" -mfloat-abi=hard -o /tmp/main-Bdilph.o
/tmp/main-rK3Mvv.s
"/usr/bin/arm-linux-gnueabihf-ld" --sysroot=/home/stephen/rpi/rasp-pi-
rootfs/ -z relro -X --hash-style=gnu --build-id --eh-frame-hdr -m
armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o a.out
/usr/arm-linux-gnueabihf/lib/crt1.o /usr/arm-linux-gnueabihf/lib/crti.o
/home/stephen/rpi/rasp-pi-rootfs//usr/lib/gcc/arm-linux-
gnueabihf/4.6/crtbegin.o -L/home/stephen/rpi/rasp-pi-
rootfs//usr/lib/gcc/arm-linux-gnueabihf/4.6 -L/home/stephen/rpi/rasp-pi-
rootfs//usr/lib/gcc/arm-linux-gnueabihf/4.6/../../../arm-linux-gnueabihf -
L/home/stephen/rpi/rasp-pi-rootfs//lib/arm-linux-gnueabihf -
L/home/stephen/rpi/rasp-pi-rootfs//usr/lib/arm-linux-gnueabihf -
L/home/stephen/rpi/rasp-pi-rootfs//usr/lib/gcc/arm-linux-
gnueabihf/4.6/../../.. -L/home/stephen/rpi/rasp-pi-rootfs//lib -
L/home/stephen/rpi/rasp-pi-rootfs//usr/lib /tmp/main-Bdilph.o -lgcc --as-
needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed
/home/stephen/rpi/rasp-pi-rootfs//usr/lib/gcc/arm-linux-
gnueabihf/4.6/crtend.o /usr/arm-linux-gnueabihf/lib/crtn.o
/usr/bin/arm-linux-gnueabihf-ld: cannot find /lib/arm-linux-
gnueabihf/libc.so.6 inside
/usr/bin/arm-linux-gnueabihf-ld: cannot find /usr/lib/arm-linux-
gnueabihf/libc_nonshared.a inside
/usr/bin/arm-linux-gnueabihf-ld: cannot find /lib/arm-linux-gnueabihf/ld-
linux-armhf.so.3 inside
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

If I supply the 'sysroot' clang requires to find the crti.o etc for linking,
then of course it fails to find libz.so:

stephen@hal:/tmp$ clang -v -target arm-linux-gnueabihf --sysroot
"/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf" -B /usr/arm-linux-
gnueabihf/lib/ main.o
Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: arm--linux-gnueabihf
Thread model: posix
"/usr/bin/arm-linux-gnueabihf-ld" --sysroot=/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf -z relro -X --hash-style=gnu --build-id --eh-frame-hdr
-m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o a.out
/usr/arm-linux-gnueabihf/lib/crt1.o /usr/arm-linux-gnueabihf/lib/crti.o
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-linux-
gnueabihf/4.7.2/crtbegin.o -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2 -L/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-
linux-gnueabihf/lib/../lib32 -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-linux-
gnueabihf/lib -L/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-
linux-gnueabihf/4.7.2/../../.. -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib main.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --
as-needed -lgcc_s --no-as-needed /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/crtend.o /usr/arm-linux-
gnueabihf/lib/crtn.o
main.o: In function `main':
main.c:(.text+0x54): undefined reference to `compress2'
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

I can manually specify the path to libz.so to make it build:

clang -v -target arm-linux-gnueabihf --sysroot "/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf" -B /usr/arm-linux-gnueabihf/lib/
/home/stephen/rpi/rasp-pi-rootfs/lib/arm-linux-gnueabihf/libz.so.1.2.7
main.o

However, hopefully you can see the difference between what clang uses
'sysroot' for and what gcc uses it for, and hopefully you can see why that's
not a good thing. Maybe what is really needed is for clang to learn a --
toolchainroot command line option to find those crti.o etc files?

Thanks,

Steve.

15

However, there's still a problem. It seems that the term 'sysroot' means
different things to clang and gcc.

Clang uses it to specify a system root. Basically, prepare a directory
you would be able to chroot it into. The GNU toolchain is pretty
inconsistent in what it does with the option, so yes, it does mean a
number of different things.

This is made worse by the hundred different approaches for Linux
distributions to set up GCC. I really think we should just start
shipping our own set of the CRT files for Linux...

If I have to use the sysroot option to tell clang where to find crtX.o
files, then I can't use it to tell clang where to find the zlib headers and
libraries. They are not in the same location as the crti.o etc.

No, that should be done with -B.

Clang also does not search in the arch-specific include dir as gcc does, so
it doesn't find sys/types:
$ clang -v -target arm-linux-gnueabihf --sysroot /home/stephen/rpi/rasp-pi-
rootfs/ -B /usr/arm-linux-gnueabihf/lib/ main.c
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"

This is a bug in the Debian/Ubuntu workaround. It seems to be looking
for the host triple and not the target triple.

Joerg

16

Joerg Sonnenberger wrote:

However, there's still a problem. It seems that the term 'sysroot' means
different things to clang and gcc.

Clang uses it to specify a system root. Basically, prepare a directory
you would be able to chroot it into.

Clang sees only one 'root'. GCC sees two. One is for the toolchain (which
does not get deployed to the target) and one is for the rootfs (which does
get deployed to the target). Typically I get both separately from a customer
and use them in the way I've been describing here. The customer typically
gets the toolchain from a third party. The rootfs is either made by the
customer or also comes from a different third party.

I think you are suggesting that I copy files around from the toolchain to
the rootfs, and use that as the sysroot? I find the suggestion to do that
without tracking, without versioning and without conspiring with upstreams
odd. It seems like a very bad idea. Have I understood you correctly?

Does anyone else on this list agree that copying files around is the way
cross compiling should work with clang?

The GNU toolchain is pretty
inconsistent in what it does with the option, so yes, it does mean a
number of different things.

Please be more specific.

If I have to use the sysroot option to tell clang where to find crtX.o
files, then I can't use it to tell clang where to find the zlib headers
and libraries. They are not in the same location as the crti.o etc.

No, that should be done with -B.

Your response is ambiguous. What are you saying should be specified with -B?
The location of the crti.o or the sysroot? How does this response relate to
the suggestion to copy files around?

Clang also does not search in the arch-specific include dir as gcc does,
so it doesn't find sys/types:
$ clang -v -target arm-linux-gnueabihf --sysroot
/home/stephen/rpi/rasp-pi-
rootfs/ -B /usr/arm-linux-gnueabihf/lib/ main.c
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"

This is a bug in the Debian/Ubuntu workaround. It seems to be looking
for the host triple and not the target triple.

I don't know what the word 'workaround' is supposed to mean in that
sentence, unless it's meant in a derogatory way.

In any case, you're right. I noticed the same but I didn't realize it might
be a distro bug.

http://thread.gmane.org/gmane.comp.compilers.clang.devel/28960/focus=29153

Bug #1181493 “debian patch uses wrong triple for arch-includes” : Bugs : clang package : Ubuntu

Thanks,

Steve.

17

I have now also tried to cross compile to windows with clang.

$ clang -v -target i386-pc-win32 -c main.c

doesn't have any built in include paths, but I can supply them as copied
from the output of i686-w64-mingw32-gcc -v -c main.c.

$ clang -v -target i386-pc-win32 -I/usr/lib/gcc/i686-w64-
mingw32/4.6/include -I/usr/lib/gcc/i686-w64-mingw32/4.6/include-fixed -
I/usr/lib/gcc/i686-w64-mingw32/4.6/../../../../i686-w64-mingw32/include -c
main.c
ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
ignoring nonexistent directory "C:/Program Files/Microsoft Visual Studio
10.0/VC/include"
ignoring nonexistent directory "C:/Program Files/Microsoft Visual Studio
9.0/VC/include"
ignoring nonexistent directory "C:/Program Files/Microsoft Visual Studio
9.0/VC/PlatformSDK/Include"
ignoring nonexistent directory "C:/Program Files/Microsoft Visual Studio
8/VC/include"
ignoring nonexistent directory "C:/Program Files/Microsoft Visual Studio
8/VC/PlatformSDK/Include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/gcc/i686-w64-mingw32/4.6/include
/usr/lib/gcc/i686-w64-mingw32/4.6/include-fixed
/usr/lib/gcc/i686-w64-mingw32/4.6/../../../../i686-w64-mingw32/include
End of search list.

seems to work fine. Should clang not add the windows paths to the search
list if host != Windows? It tries to use link.exe to link too, which
obviously doesn't work:

$ clang -v -target i386-pc-win32 main.o
Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on
LLVM 3.2)
Target: i386-pc-win32
Thread model: posix
"link.exe" -out:a.out -defaultlib:libcmt -nologo main.o
clang: error: unable to execute command: No such file or directory
clang: error: linker command failed due to signal (use -v to see
invocation)

however, I can link it with i686-w64-mingw32-gcc main.o and execute it in
wine.

Next up, a dll:

stephen@hal:/tmp$ cat lib.h

#ifdef BUILDING_EXAMPLE_DLL
#define EXAMPLE_DLL __declspec(dllexport)
#else
#define EXAMPLE_DLL __declspec(dllimport)
#endif

int EXAMPLE_DLL myveryeasymethod(void);

stephen@hal:/tmp$ cat lib.c

#include "lib.h"

int myveryeasymethod(void)
{ return 42; }

Compiling with -DBUILDING_EXAMPLE_DLL works fine.

$ i686-w64-mingw32-gcc -shared lib.o -Wl,--out-implib,libexample_dll.a -o
example_dll.dll
Warning: .drectve `/EXPORT:_myveryeasymethod' unrecognized

That warning does not appear when the file is compiled with gcc. There is a
similar warning when building c++:

$ i686-w64-mingw32-g++ -shared lib.o -Wl,--out-implib,libexample_dll.a -o
example_dll.dll
Warning: .drectve `/EXPORT:myveryeasymethod()' unrecognized

What do these mean? Should I file a bug report?

Thanks,

Steve.

18

Since you’re using the mingw linker, you should probably target mingw. Try using i386-pc-mingw32 as the triple. I think using win32 as the os assumes you want the Visual Studio toolchain by default, and I don’t recall what else controls it.

19

Reid Kleckner wrote:

Since you're using the mingw linker, you should probably target mingw.
Try
using i386-pc-mingw32 as the triple. I think using win32 as the os
assumes you want the Visual Studio toolchain by default, and I don't
recall what else controls it.

Thanks for the pointer. That indeed does not warn when linking, but it also
doesn't link at all:

stephen@hal:/tmp$ i686-w64-mingw32-gcc -v -shared lib.o -o example_dll.dll -
Wl,--out-implib,libexample_dll.a
Using built-in specs.
COLLECT_GCC=i686-w64-mingw32-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/i686-w64-mingw32/4.6/lto-wrapper
Target: i686-w64-mingw32
Configured with: ../../src/configure --build=x86_64-linux-gnu --prefix=/usr
--includedir='/usr/include' --mandir='/usr/share/man' --
infodir='/usr/share/info' --sysconfdir=/etc --localstatedir=/var --
libexecdir='/usr/lib/gcc-mingw-w64' --disable-maintainer-mode --disable-
dependency-tracking --prefix=/usr --enable-shared --enable-static --disable-
multilib --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib
--without-included-gettext --libdir=/usr/lib --enable-libstdcxx-time=yes --
with-tune=generic --enable-version-specific-runtime-libs --enable-
threads=win32 --enable-fully-dynamic-string --enable-sjlj-exceptions --
enable-languages=c,c++,fortran,objc,obj-c++,ada --enable-lto --with-plugin-
ld --target=i686-w64-mingw32 --with-gxx-include-dir=/usr/include/c++/4.6 --
with-as=/usr/bin/i686-w64-mingw32-as --with-ld=/usr/bin/i686-w64-mingw32-ld
Thread model: win32
gcc version 4.6.3 (GCC)
COMPILER_PATH=/usr/lib/gcc/i686-w64-mingw32/4.6/:/usr/lib/gcc/i686-w64-
mingw32/4.6/:/usr/lib/gcc/i686-w64-mingw32/:/usr/lib/gcc/i686-w64-
mingw32/4.6/:/usr/lib/gcc/i686-w64-mingw32/
LIBRARY_PATH=/usr/lib/gcc/i686-w64-mingw32/4.6/:/usr/lib/gcc/i686-w64-
mingw32/4.6/../../../../i686-w64-mingw32/lib/../lib/:/usr/lib/gcc/i686-w64-
mingw32/4.6/../../../../i686-w64-mingw32/lib/
COLLECT_GCC_OPTIONS='-v' '-shared' '-o' 'example_dll.dll' '-mtune=generic'
'-march=pentiumpro'
/usr/lib/gcc/i686-w64-mingw32/4.6/collect2 -m i386pe --shared -Bdynamic -e
_DllMainCRTStartup@12 --enable-auto-image-base -o example_dll.dll
/usr/lib/gcc/i686-w64-mingw32/4.6/../../../../i686-w64-
mingw32/lib/../lib/dllcrt2.o /usr/lib/gcc/i686-w64-mingw32/4.6/crtbegin.o -
L/usr/lib/gcc/i686-w64-mingw32/4.6 -L/usr/lib/gcc/i686-w64-
mingw32/4.6/../../../../i686-w64-mingw32/lib/../lib -L/usr/lib/gcc/i686-w64-
mingw32/4.6/../../../../i686-w64-mingw32/lib lib.o --out-implib
libexample_dll.a -lmingw32 -lgcc_eh -lgcc -lmoldname -lmingwex -lmsvcrt -
ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_eh -lgcc -lmoldname
-lmingwex -lmsvcrt /usr/lib/gcc/i686-w64-mingw32/4.6/crtend.o
Cannot export myveryeasymethod(): symbol not found
collect2: ld returned 1 exit status

stephen@hal:/tmp$ cat lib.o
Lݘ�Q�.textd P`.drectvej�*� -export:__Z16myveryeasymethodv.text.drectve
_Z16myveryeasymethodv
stephen@hal:/tmp$ cat lib.o | i686-w64-mingw32-c++filt
Lݘ�Q� `.drectve�*� -export:myveryeasymethod() [clone .text]

However, if I remove the BUILDING_EXAMPLE_DLL from in front of
myveryeasymethod() it does successfully link, which is the opposite of what
I expect...

Thanks,

Steve.

20

Looking at the export directive, it seems like something is off in name mangling.
-export:__Z16myveryeasymethodv

The double underscore seems wrong, which suggests we’re applying both C and C++ mangling. Use a version of nm that works on COFF to make sure that the .o has the same name as the -export.

If it’s broken, try building TOT clang, and if that doesn’t work, please file a bug. There was a recent bugfix for a similar issue which had to do with stdcall mangling, but I’m not sure if it’s the same as this one.