distributed thinlto usage

Project Infrastructure LLVM Dev List Archives
1

I am trying to work through the usage of thinlto for distributed builds.

Here is the simple thinlto usage, just add -flto=thin everywhere, easy:

clang++ -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp

clang++ -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp

clang++ -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp

clang++ -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp

clang++ -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp

clang++ -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp

clang++ -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp

clang++ -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp

clang++ -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp

clang++ -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp

clang++ -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp

clang++ -flto=thin -O3 -o astar.thin CreateWay_.o Places_.o RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o Region_.o RegWay_.o Way_.o

My understanding is for the distributed, you need to do the inital compiles as above,

then a special link to get the thinlto index files, then compile again to get

native files, and then a final link. This looks like the following but

it does not work:

clang++ -flto=thin -O3 -c -o CreateWay_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else CreateWay_.cpp

clang++ -flto=thin -O3 -c -o Places_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Places_.cpp

clang++ -flto=thin -O3 -c -o RegBounds_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegBounds_.cpp

clang++ -flto=thin -O3 -c -o RegMng_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegMng_.cpp

clang++ -flto=thin -O3 -c -o Way2_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way2_.cpp

clang++ -flto=thin -O3 -c -o WayInit_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else WayInit_.cpp

clang++ -flto=thin -O3 -c -o Library.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Library.cpp

clang++ -flto=thin -O3 -c -o Random.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Random.cpp

clang++ -flto=thin -O3 -c -o Region_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Region_.cpp

clang++ -flto=thin -O3 -c -o RegWay_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else RegWay_.cpp

clang++ -flto=thin -O3 -c -o Way_.o -DSPEC_CPU -DNDEBUG -DSPEC_CPU_LITTLE_ENDIAN -Wno-dangling-else Way_.cpp

clang++ -flto=thin -O3 -o thinlto.objects -Wl,-plugin-opt,thinlto-index-only=thinlto.objects CreateWay_.o Places_.o RegBounds_.o RegMng_.o Way2_.o WayInit_.o Library.o Random.o Region_.o RegWay_.o Way_.o

clang++ -c -x ir CreateWay_.o -O3 -o CreateWay_-native.o -fthinlto-index=CreateWay_.o.thinlto.bc

clang++ -c -x ir Places_.o -O3 -o Places_-native.o -fthinlto-index=Places_.o.thinlto.bc

clang++ -c -x ir RegBounds_.o -O3 -o RegBounds_-native.o -fthinlto-index=RegBounds_.o.thinlto.bc

clang++ -c -x ir RegMng_.o -O3 -o RegMng_-native.o -fthinlto-index=RegMng_.o.thinlto.bc

clang++ -c -x ir Way2_.o -O3 -o Way2_-native.o -fthinlto-index=Way2_.o.thinlto.bc

clang++ -c -x ir WayInit_.o -O3 -o WayInit_-native.o -fthinlto-index=WayInit_.o.thinlto.bc

clang++ -c -x ir Library.o -O3 -o Library-native.o -fthinlto-index=Library.o.thinlto.bc

clang++ -c -x ir Random.o -O3 -o Random-native.o -fthinlto-index=Random.o.thinlto.bc

clang++ -c -x ir Region_.o -O3 -o Region_-native.o -fthinlto-index=Region_.o.thinlto.bc

clang++ -c -x ir RegWay_.o -O3 -o RegWay_-native.o -fthinlto-index=RegWay_.o.thinlto.bc

clang++ -c -x ir Way_.o -O3 -o Way_-native.o -fthinlto-index=Way_.o.thinlto.bc

clang++ -o astar CreateWay_-native.o Places_-native.o RegBounds_-native.o RegMng_-native.o Way2_-native.o WayInit_-native.o Library-native.o Random-native.o Region_-native.o RegWay_-native.o Way_-native.o

Library-native.o:(.data+0x0): undefined reference to `vtable for regwayobj’

clang-8: error: linker command failed with exit code 1 (use -v to see invocation)

make: *** [astar] Error 1

Did I miss a step?

2

Hi David,

That looks basically right, so I am not sure offhand what is wrong. Looks like this is the astar spec binary, so I will try to repro it tomorrow morning. What linker are you using?

Note that if your link involves archives of .a files, you will want to ensure that the final native link includes only those files selected by the linker. That is output in the file given to the “-Wl,-plugin-opt,thinlto-index-only=files.out” option. What we do is use that in combination with the ‘-Wl,-plugin-opt,thinlto-prefix-replace=path1;path2’ option to simplify the final link invocation. Specifically, if you do something like:
“-Wl,-plugin-opt,thinlto-index-only=files.out -Wl,-plugin-opt,thinlto-prefix-replace=path1;path2”
for the indexing link step, the output files from the thin link will be written to a directory tree with the original path substring “path1” replaced with “path2”, and the object names in the file emitted by thinlto-index-only= (“files.out” in the above example), will have their path names adjusted to the new path as well - if your native .o files use the new path then this is a simple way of getting a list of input files for the final link, which can be passed via @files.out to gold (note the gold manpage does not document “@” but the gnu ld manpage does).

I.e. something like:

clang++ -flto=thin -O3 -c -o path1/foo.o foo.cpp
…
clang++ -flto=thin -O3 -Wl,-plugin-opt,thinlto-index-only=files.out -Wl,-plugin-opt,thinlto-prefix-replace=path1;path2" foo.o …
clang++ -c -x ir path1/foo.o -O3 -o path2/foo.o -fthinlto-index=path2/foo.o.thinlto.bc
…
clang++ -flto=thin -O3 -o a.out @files.out

But again, that should only come into play if you are linking archives of bitcode files…

Teresa

3

Thanks Teresa

Yes it is astar, happen to send a tar of the sources but they are just copies from the spec distribution

The ld command is:

GNU ld (GNU Binutils) 2.29.1

Thanks for the guidance on path names. The prefix-replace just effects the string written to the object files right? So we could post-process that file with other tools as well, correct?

Thanks again

–david

4

Thanks Teresa

Yes it is astar, happen to send a tar of the sources but they are just copies from the spec distribution

The ld command is:

GNU ld (GNU Binutils) 2.29.1

Could you try with either a recent gold or lld to see if it still reproduces? We don’t test llvm LTO with GNU ld, so I can’t be sure there are no issues there.

Thanks for the guidance on path names. The prefix-replace just effects the string written to the object files right? So we could post-process that file with other tools as well, correct?

Correct.
Teresa

5

Fails with gold too:

Library-native.o:Library.cpp:regway: error: undefined reference to ‘vtable for regwayobj’

/home/dcallahan/fbsource/fbcode/third-party-buck/platform007/tools/binutils/bin/gold/ld: the vtable symbol may be undefined because the class is missing its key function

clang-8: error: linker command failed with exit code 1 (use -v to see invocation)

6

Ok thanks for checking. I’ll try to repro tomorrow morning probably and get back. Teresa

7

Mystery solved - the short answer is to add “-Xclang -fno-lto-unit” to your compile steps (-c -flto=thin). I had forgotten that we add this for our internal distributed builds. Another option would be to add “-Wl,-plugin-opt,obj-path=astar.o” to the thin link step, and add the resulting (native) astar.o file to the final link.

I am working on getting the default changed (D53890/D53891), since this is only needed for CFI [1], but currently by default each module is split into two - a regular LTO module containing vtable defs and a thin LTO module containing the rest. This is required by CFI so that the vtables can be globally laid out via regular LTO. If you don’t disable this splitting (via “-Xclang -fno-lto-unit”), then you need to capture the result of this regular LTO portion via the obj-path plugin option and link it as well. However, unless you plan to do CFI this is just giving your builds extra unnecessary overhead (which is why the default will change).

[1] Also currently for whole program devirtualization which is enabled for CFI, or separately via options - although I have a patch out for review to do WPD with ThinLTO alone (D55153).

Teresa

8

Thanks very much , that did resolve my problem.

9

Mystery solved - the short answer is to add “-Xclang -fno-lto-unit” to your compile steps (-c -flto=thin). I had forgotten that we add this for our internal distributed builds. Another option would be to add “-Wl,-plugin-opt,obj-path=astar.o” to the thin link step, and add the resulting (native) astar.o file to the final link.

FYI as of my commit a short while ago of r350948/r350949, this option is no longer needed (commits of the patches mentioned below to disable splitting by default unless something like CFI enabled).
Teresa