Updated RFC: ThinLTO Implementation Plan

Project Infrastructure LLVM Dev List Archives
1

As promised, here is an new version of the ThinLTO RFC, updated based
on some of the comments, questions and feedback from the first RFC.
Hopefully we have addressed many of these, and as noted below, will
fork some of the detailed discussion on particular aspects into
separate design doc threads. Please send any additional feedback and
questions on the overall design.
Thanks!
Teresa

Updated RFC to discuss plans for implementing ThinLTO upstream,
reflecting feedback and discussion from initial RFC
(http://lists.cs.uiuc.edu/pipermail/llvmdev/2015-May/085557.html). As
discussed in the earlier thread and below, more detailed design
documents for several pieces (native object format, linkage type
changes and static promotions, etc) are in progress and will be sent
separately. This RFC covers the overall design and the breakdown of
work at a higher level.

Background on ThinLTO can be found in slides from EuroLLVM 2015:
  Google Drive: Sign-in
As described in the talk, we have a prototype implementation, and
would like to start staging patches upstream. This RFC describes a
breakdown of the major pieces. We would like to commit upstream
gradually in several stages, with all functionality off by default.
The core ThinLTO importing support and tuning will require frequent
change and iteration during testing and tuning, and for that part we
would like to commit rapidly (off by default). See the proposed staged
implementation described in the Implementation Plan section.

ThinLTO Overview

2

My earlier statement about wrapping things in a native object file held in that it is controversial. It appears to be still central to your design.

It may help to look at the problem from a different viewpoint: LLVM is not a compiler. It is a framework that can be used to make compiler-like tools.

From that view, it no longer makes sense to discuss "the plugin," or gold, or $AR, because there isn't just one of any of those things. ld64 isn't the only outlier linker to consider. We have our own linker at Sony, for example. From this perspective, then it makes more sense to consider replacing the binary utilities with ones that support bitcode, because from a user-perspective, all of the linkers already transparently support bitcode directly today, as do ar, nm, etc. This has been necessary for the regular LTO process.

The only tool in the list of tools you mentioned that do not support bitcode directly is objcopy, and that's because nobody has yet written an LLVM-project implementation of it. Personally, I'd much rather you focus on making ThinLTO work by extending bitcode as needed, and we work as a community toward replacing objcopy with an LLVM-native one. It's a big missing piece of the LLVM project today and could be so much better if we could use it to replace Apple's lipo and possibly other extant object file modification tools. (Has anyone surveyed this area?)

That older toolchains have tried to slip non-object file data through the binary utilities isn't really proof that this is a good choice. It might simply reflect the realities of those engineering teams. I wasn't at Sun for this, but DTrace needed a linker feature that apparently the Sun linker team was unwilling or unable to provide, so dtrace(1) gained the ability to modify ELF files directly as needed. That doesn't prove that DTrace's USDT feature shouldn't have been implemented in the linker (as ld64 does directly for Apple), does it?

If in the end using native object-wrapped bitcode is the best solution, so be it. However, I think it is largely orthogonal to ThinLTO's needs for transporting symtab data alongside the existing bitcode format.

Alex

3

My earlier statement about wrapping things in a native object file held in that it is controversial. It appears to be still central to your design.

It may help to look at the problem from a different viewpoint: LLVM is not a compiler. It is a framework that can be used to make compiler-like tools.

From that view, it no longer makes sense to discuss "the plugin," or gold, or $AR, because there isn't just one of any of those things. ld64 isn't the only outlier linker to consider. We have our own linker at Sony, for example. From this perspective, then it makes more sense to consider replacing the binary utilities with ones that support bitcode, because from a user-perspective, all of the linkers already transparently support bitcode directly today, as do ar, nm, etc. This has been necessary for the regular LTO process.

Hi Alex,

It's true that the LLVM versions of these tools support bitcode
transparently, but not all build systems use LLVM versions of these
tools, particularly build systems that support a variety of compilers,
or legacy build systems. And not all build systems have the plugin or
currently pass it to the native tools that can take a plugin for
handling bitcode. In those cases the bitcode support is not
transparently available, and our aim is to reduce the friction as much
as possible. And not all use LTO currently (I know we don't due to the
scalability issues we're trying to address with this design), and in
those cases the migration to bitcode-aware tools and plugins was not
previously required.

For Sony's linker, are you using the gold plugin or libLTO interfaces?
If the latter, I suppose some ThinLTO handling would have to be added
to your linker (e.g. to invoke the LLVM hooks to write the stage-2
combined function map and either launch the backend processes in
parallel or write out a make or other build file). The current support
for reading native object wrapped bitcode is baked into IRObjectFile
so presumably the Sony linker can handle these native object wrapped
bitcode files if it uses libLTO. We would similarly embed the handling
of the function index/summary behind an API that can handle either so
it is similarly transparent to the linkers. Let me know if there would
be additional issues that make wrapped bitcode more difficult in your
case, or how we could make ThinLTO usage simpler for you in general.

The only tool in the list of tools you mentioned that do not support bitcode directly is objcopy, and that's because nobody has yet written an LLVM-project implementation of it. Personally, I'd much rather you focus on making ThinLTO work by extending bitcode as needed, and we work as a community toward replacing objcopy with an LLVM-native one. It's a big missing piece of the LLVM project today and could be so much better if we could use it to replace Apple's lipo and possibly other extant object file modification tools. (Has anyone surveyed this area?)

That older toolchains have tried to slip non-object file data through the binary utilities isn't really proof that this is a good choice. It might simply reflect the realities of those engineering teams. I wasn't at Sun for this, but DTrace needed a linker feature that apparently the Sun linker team was unwilling or unable to provide, so dtrace(1) gained the ability to modify ELF files directly as needed. That doesn't prove that DTrace's USDT feature shouldn't have been implemented in the linker (as ld64 does directly for Apple), does it?

I'd argue that the realities being addressed by using native object
format in those cases still exist.

If in the end using native object-wrapped bitcode is the best solution, so be it. However, I think it is largely orthogonal to ThinLTO's needs for transporting symtab data alongside the existing bitcode format.

That's certainly true, ThinLTO can be implemented using either format,
and bitcode only support can certainly be implemented. It is a matter
of prioritizing which format to implement first. I had added some
description to the updated RFC on how the function index/summary can
be represented, etc in bitcode. Prioritizing the native object format
doesn't make it easier to implement ThinLTO, but should make it easier
to deploy.

Thanks!
Teresa

4

> My earlier statement about wrapping things in a native object file held
in that it is controversial. It appears to be still central to your design.
>
> It may help to look at the problem from a different viewpoint: LLVM is
not a compiler. It is a framework that can be used to make compiler-like
tools.
>
> From that view, it no longer makes sense to discuss "the plugin," or
gold, or $AR, because there isn't just one of any of those things. ld64
isn't the only outlier linker to consider. We have our own linker at Sony,
for example. From this perspective, then it makes more sense to consider
replacing the binary utilities with ones that support bitcode, because from
a user-perspective, all of the linkers already transparently support
bitcode directly today, as do ar, nm, etc. This has been necessary for the
regular LTO process.

Hi Alex,

It's true that the LLVM versions of these tools support bitcode
transparently, but not all build systems use LLVM versions of these
tools, particularly build systems that support a variety of compilers,
or legacy build systems.

If a build system can do
CC=clang
why wouldn't it be able to do
AR=llvm-ar
?

-- Sean Silva

5

> My earlier statement about wrapping things in a native object file held
> in that it is controversial. It appears to be still central to your design.
>
> It may help to look at the problem from a different viewpoint: LLVM is
> not a compiler. It is a framework that can be used to make compiler-like
> tools.
>
> From that view, it no longer makes sense to discuss "the plugin," or
> gold, or $AR, because there isn't just one of any of those things. ld64
> isn't the only outlier linker to consider. We have our own linker at Sony,
> for example. From this perspective, then it makes more sense to consider
> replacing the binary utilities with ones that support bitcode, because from
> a user-perspective, all of the linkers already transparently support bitcode
> directly today, as do ar, nm, etc. This has been necessary for the regular
> LTO process.

Hi Alex,

It's true that the LLVM versions of these tools support bitcode
transparently, but not all build systems use LLVM versions of these
tools, particularly build systems that support a variety of compilers,
or legacy build systems.

If a build system can do
CC=clang
why wouldn't it be able to do
AR=llvm-ar
?

That assumes that the LLVM tools are all deployed in the build system,
and adds a requirement for using clang in this mode that wasn't there
before when using clang for -O2. We are trying to make the transition
from clang -O2 to clang -O2 + ThinLTO as seamless as possible.

Teresa

6

Hi Teresa,

Thanks for providing this updated RFC.

For Sony's linker, are you using the gold plugin or libLTO interfaces?
If the latter, I suppose some ThinLTO handling would have to be added
to your linker (e.g. to invoke the LLVM hooks to write the stage-2
combined function map and either launch the backend processes in
parallel or write out a make or other build file). The current support
for reading native object wrapped bitcode is baked into IRObjectFile
so presumably the Sony linker can handle these native object wrapped
bitcode files if it uses libLTO. We would similarly embed the handling
of the function index/summary behind an API that can handle either so
it is similarly transparent to the linkers. Let me know if there would
be additional issues that make wrapped bitcode more difficult in your
case, or how we could make ThinLTO usage simpler for you in general.

We use the libLTO interfaces.

We use the libLTO interfaces, more specifically we use the C API
located in llvm-c\lto.h.

Our linker won't support native object wrapped bitcode files as our
LTO is it currently stands. Right now, it will be recognized as an
object file and won't get anywhere near the libLTO libraries. We'd
need to teach our linker to recognize and differentiate native object
wrapped bitcode files and regular native object files. This isn't
straight forward as we cannot distinguish them just by looking at the
file header alone, we would need to parse the sections and look for a
.llvmbc section. We then need to add special handling of these native
object wrappers.

Handling the function index/summary behind an API sounds like a good idea.

7

Hi Teresa,

Thanks for providing this updated RFC.

For Sony's linker, are you using the gold plugin or libLTO interfaces?
If the latter, I suppose some ThinLTO handling would have to be added
to your linker (e.g. to invoke the LLVM hooks to write the stage-2
combined function map and either launch the backend processes in
parallel or write out a make or other build file). The current support
for reading native object wrapped bitcode is baked into IRObjectFile
so presumably the Sony linker can handle these native object wrapped
bitcode files if it uses libLTO. We would similarly embed the handling
of the function index/summary behind an API that can handle either so
it is similarly transparent to the linkers. Let me know if there would
be additional issues that make wrapped bitcode more difficult in your
case, or how we could make ThinLTO usage simpler for you in general.

We use the libLTO interfaces.

Hi Dave,

Thanks for the info.

We use the libLTO interfaces, more specifically we use the C API
located in llvm-c\lto.h.

Our linker won't support native object wrapped bitcode files as our
LTO is it currently stands. Right now, it will be recognized as an
object file and won't get anywhere near the libLTO libraries. We'd
need to teach our linker to recognize and differentiate native object
wrapped bitcode files and regular native object files. This isn't
straight forward as we cannot distinguish them just by looking at the
file header alone, we would need to parse the sections and look for a
.llvmbc section. We then need to add special handling of these native
object wrappers.

Ok, I see. Does it help that there are LTOModule (lto_module_* in the
C API) interfaces for checking if a file contains bitcode (regardless
of whether it is straight-up or native-wrapped)? I don't know how hard
in your linker it is to query these when deciding whether to treat the
object file as bitcode or not, or how hard it is to pass the resulting
object file along to the libLTO routines for handling (they
automatically handle the native-wrapped object files so the linker
shouldn't have to do anything special to read them).

Specifically, in the C API these are the lto_module_is_object_file*
variants, which will return true for either straight-up or
native-wrapped bitcode. All of the mechanics of handling bitcode vs
native object-wrapped bitcode are down in the IRObjectFile handling.
So the LTOModule:isBitcode*/lto_module_is_object_file* will correctly
identify native object-wrapped bitcode as bitcode. And the
LTOModule::createFrom*/lto_module_create* routines correctly parse the
native object-wrapped bitcode and return an LTOmodule.

As a result, the llvm-lto tool that also uses libLTO interfaces didn't
require any changes when the native-wrapped reading support went in
(r218078), and is able to handle native-wrapped bitcode out of the
box.

Handling the function index/summary behind an API sounds like a good idea.

I am going to work on fleshing out this part next so that the actual
format of the files is hidden from clients.

Thanks,
Teresa

8

My earlier statement about wrapping things in a native object file held
in that it is controversial. It appears to be still central to your design.

It may help to look at the problem from a different viewpoint: LLVM is
not a compiler. It is a framework that can be used to make compiler-like
tools.

From that view, it no longer makes sense to discuss "the plugin," or
gold, or $AR, because there isn't just one of any of those things. ld64
isn't the only outlier linker to consider. We have our own linker at Sony,
for example. From this perspective, then it makes more sense to consider
replacing the binary utilities with ones that support bitcode, because from
a user-perspective, all of the linkers already transparently support bitcode
directly today, as do ar, nm, etc. This has been necessary for the regular
LTO process.

Hi Alex,

It's true that the LLVM versions of these tools support bitcode
transparently, but not all build systems use LLVM versions of these
tools, particularly build systems that support a variety of compilers,
or legacy build systems.

If a build system can do
CC=clang
why wouldn't it be able to do
AR=llvm-ar
?

That assumes that the LLVM tools are all deployed in the build system,
and adds a requirement for using clang in this mode that wasn't there
before when using clang for -O2. We are trying to make the transition
from clang -O2 to clang -O2 + ThinLTO as seamless as possible.

If Clang is deployed, then the other LLVM tools can be as well. I'd wager that at Sony we deal with more build systems than most. That is, for each game team out there, an approximation to the number of build systems used is N game teams x M build systems per game team. (And they all believe that their game build is "simple.")

How Google intends to deploy ThinLTO internally is guaranteed to be different from how Sony would want to deploy it, from how Apple would, etc. From the perspective of LLVM as a framework, it is best to separate the design layers in such a way that it can be deployed without the native wrapper and leave that out as a requirement. LLVM has bitcode for the purpose of communicating with itself. It is the LLVM design that it passes bitcode into and through the binary utilities, prior patches for native wrappers (for Go if I'm not mistaken) notwithstanding.

What I don't understand is why it is important in Google's application to use the legacy [GNU] binary utilities. Last I heard, llvm-ar is much faster than the GNU ar, etc.

Alex

9

From: llvmdev-bounces@cs.uiuc.edu [mailto:llvmdev-bounces@cs.uiuc.edu] On
Behalf Of Teresa Johnson
Sent: Wednesday, June 03, 2015 7:02 AM
To: Dave Bozier
Cc: <llvmdev@cs.uiuc.edu> List
Subject: Re: [LLVMdev] Updated RFC: ThinLTO Implementation Plan

> Hi Teresa,
>
> Thanks for providing this updated RFC.
>
>> For Sony's linker, are you using the gold plugin or libLTO interfaces?
>> If the latter, I suppose some ThinLTO handling would have to be added
>> to your linker (e.g. to invoke the LLVM hooks to write the stage-2
>> combined function map and either launch the backend processes in
>> parallel or write out a make or other build file). The current support
>> for reading native object wrapped bitcode is baked into IRObjectFile
>> so presumably the Sony linker can handle these native object wrapped
>> bitcode files if it uses libLTO. We would similarly embed the handling
>> of the function index/summary behind an API that can handle either so
>> it is similarly transparent to the linkers. Let me know if there would
>> be additional issues that make wrapped bitcode more difficult in your
>> case, or how we could make ThinLTO usage simpler for you in general.
> We use the libLTO interfaces.

Hi Dave,

Thanks for the info.

>
> We use the libLTO interfaces, more specifically we use the C API
> located in llvm-c\lto.h.
>
> Our linker won't support native object wrapped bitcode files as our
> LTO is it currently stands. Right now, it will be recognized as an
> object file and won't get anywhere near the libLTO libraries. We'd
> need to teach our linker to recognize and differentiate native object
> wrapped bitcode files and regular native object files. This isn't
> straight forward as we cannot distinguish them just by looking at the
> file header alone, we would need to parse the sections and look for a
> .llvmbc section. We then need to add special handling of these native
> object wrappers.

Ok, I see. Does it help that there are LTOModule (lto_module_* in the
C API) interfaces for checking if a file contains bitcode (regardless
of whether it is straight-up or native-wrapped)? I don't know how hard
in your linker it is to query these when deciding whether to treat the
object file as bitcode or not, or how hard it is to pass the resulting
object file along to the libLTO routines for handling (they
automatically handle the native-wrapped object files so the linker
shouldn't have to do anything special to read them).

One twist is that we use the Darwin-style wrapper around our bitcode files
so that we have a place to hang a bitcode version number, which we also
want to check. Without reopening the debate about why we do that, we do
that, and I fully expect the libLTO API to silently ignore the wrapper that
we are depending on. I suppose we could add a new libLTO API that verifies
the bitcode wrapper but it would be yet another private change to maintain,
rather than just having the linker check it directly.
--paulr

10

Ok, I see. Does it help that there are LTOModule (lto_module_* in the
C API) interfaces for checking if a file contains bitcode (regardless
of whether it is straight-up or native-wrapped)?

Unfortunately no it won't help. We try to identify inputs on the
command line in the order 1) Object 2) static library 3) Bitcode 4)
Linker script. Our LTO implementation is split into a separate process
that is lazily loaded only in the case that we are trying to identify
an input and don't recognize it as an object or static library and
suspect it is a bitcode file.

I don't know how hard
in your linker it is to query these when deciding whether to treat the
object file as bitcode or not, or how hard it is to pass the resulting
object file along to the libLTO routines for handling (they
automatically handle the native-wrapped object files so the linker
shouldn't have to do anything special to read them).

It will be difficult with our current design and without compromise.
Either we change our Identify routines to process the section headers
(can be costly for COMDAT heavy code or code built with
-ffuntion-sections, --fdata-sections). Or we change our ELF scan
routines to specially identify and handle ELF with llvmbc differently.

Alternatively could we mark the native file header in a way that
identifies it as a bitcode wrapper is an option? That way we wouldn't
need to parse the section headers upfront to identify how the input
should be treated. This is possibly something we can look at as a
private change.

Specifically, in the C API these are the lto_module_is_object_file*
variants, which will return true for either straight-up or
native-wrapped bitcode. All of the mechanics of handling bitcode vs
native object-wrapped bitcode are down in the IRObjectFile handling.
So the LTOModule:isBitcode*/lto_module_is_object_file* will correctly
identify native object-wrapped bitcode as bitcode. And the
LTOModule::createFrom*/lto_module_create* routines correctly parse the
native object-wrapped bitcode and return an LTOmodule.

It's great to see these API functions have been considered and added
to libLTO and I'm sure they will help for other tools. Unfortunately
not ours.

11

Actually Paul that's a very good point and may work. If we could use
the bitcode wrapper that wraps the native object wrapper that wraps
the bitcode file, then that may actually work out of the box for us.
The file format doesn't specify if both wrappers can be used and in
what order. Also doubley wrapped kind of makes my head hurt thinking
about it...

12

> Ok, I see. Does it help that there are LTOModule (lto_module_* in the
> C API) interfaces for checking if a file contains bitcode (regardless
> of whether it is straight-up or native-wrapped)?

Unfortunately no it won't help. We try to identify inputs on the
command line in the order 1) Object 2) static library 3) Bitcode 4)
Linker script. Our LTO implementation is split into a separate process
that is lazily loaded only in the case that we are trying to identify
an input and don't recognize it as an object or static library and
suspect it is a bitcode file.

This sounds a little fragile and inflexible. How is the command line formed?

> I don't know how hard
> in your linker it is to query these when deciding whether to treat the
> object file as bitcode or not, or how hard it is to pass the resulting
> object file along to the libLTO routines for handling (they
> automatically handle the native-wrapped object files so the linker
> shouldn't have to do anything special to read them).

It will be difficult with our current design and without compromise.
Either we change our Identify routines to process the section headers
(can be costly for COMDAT heavy code or code built with
-ffuntion-sections, --fdata-sections). Or we change our ELF scan
routines to specially identify and handle ELF with llvmbc differently.

At least for ELF, there is a e_machine field in the Elf header that can
potentially be useful here.

thanks,

David

Alternatively could we mark the native file header in a way that

13

Last I heard, llvm-ar is much faster than the GNU ar, etc.

They are not fully feature compatible. For instance, support of thin
archive is missing (not that it is hard to implement it though).

David

14

This sounds a little fragile and inflexible. How is the command line formed?

Our command line is processing is 100% compatible with GNU ld.

Sorry I think my earlier comment is confusing. I didn't mean order of
inputs placed on the command line. I meant the order of file types we
try to identify files as.

At least for ELF, there is a e_machine field in the Elf header that can potentially be useful here.

Yes, this is what I was asking after.

15

>
>
>>
>> > My earlier statement about wrapping things in a native object file
held
>> > in that it is controversial. It appears to be still central to your
design.
>> >
>> > It may help to look at the problem from a different viewpoint: LLVM is
>> > not a compiler. It is a framework that can be used to make
compiler-like
>> > tools.
>> >
>> > From that view, it no longer makes sense to discuss "the plugin," or
>> > gold, or $AR, because there isn't just one of any of those things.
ld64
>> > isn't the only outlier linker to consider. We have our own linker at
Sony,
>> > for example. From this perspective, then it makes more sense to
consider
>> > replacing the binary utilities with ones that support bitcode,
because from
>> > a user-perspective, all of the linkers already transparently support
bitcode
>> > directly today, as do ar, nm, etc. This has been necessary for the
regular
>> > LTO process.
>>
>> Hi Alex,
>>
>> It's true that the LLVM versions of these tools support bitcode
>> transparently, but not all build systems use LLVM versions of these
>> tools, particularly build systems that support a variety of compilers,
>> or legacy build systems.
>
>
> If a build system can do
> CC=clang
> why wouldn't it be able to do
> AR=llvm-ar
> ?

That assumes that the LLVM tools are all deployed in the build system,
and adds a requirement for using clang in this mode that wasn't there
before when using clang for -O2. We are trying to make the transition
from clang -O2 to clang -O2 + ThinLTO as seamless as possible.

I'd just like the point out that downthread serious suggestions are being
fielded to use a nonstandard ELF header or nonstandard bits marking the
header. This "adds a requirement".

At this point at least 1 (the only?) concrete deployment use case (besides
yours) that has been brought up in the ThinLTO RFC threads is
inconvenienced by this design decision. This suggests that native object
wrapping doesn't offer as much seamlessness as it seems.

In general, your proposal contains lots of "we will start out with" type
constructs regarding the tooling and practical deployment, and it is not
clear that you have done any feasibility research into whether "we will
start out with" will turn into "practically speaking, this will only ever
be implemented in" due to failing to take a sufficiently diverse set of use
cases into account.

-- Sean Silva

16

Last I heard, llvm-ar is much faster than the GNU ar, etc.

They are not fully feature compatible. For instance, support of thin
archive is missing (not that it is hard to implement it though).

It's on the LLVM roadmap to make these a drop-in replacement, so any work
to improve them is not wasted and would be greatly appreciated. Don't let
their current state hold you back from considering them for ThinLTO
deployment; as you've said, the features are generally pretty
straightforward to implement (and ones that aren't are usually looking
deeply-enough into an object file that running them on
bitcode-in-object-file doesn't make sense).

-- Sean Silva

17

>
>
>>
>> > My earlier statement about wrapping things in a native object file
>> > held
>> > in that it is controversial. It appears to be still central to your
>> > design.
>> >
>> > It may help to look at the problem from a different viewpoint: LLVM
>> > is
>> > not a compiler. It is a framework that can be used to make
>> > compiler-like
>> > tools.
>> >
>> > From that view, it no longer makes sense to discuss "the plugin," or
>> > gold, or $AR, because there isn't just one of any of those things.
>> > ld64
>> > isn't the only outlier linker to consider. We have our own linker at
>> > Sony,
>> > for example. From this perspective, then it makes more sense to
>> > consider
>> > replacing the binary utilities with ones that support bitcode,
>> > because from
>> > a user-perspective, all of the linkers already transparently support
>> > bitcode
>> > directly today, as do ar, nm, etc. This has been necessary for the
>> > regular
>> > LTO process.
>>
>> Hi Alex,
>>
>> It's true that the LLVM versions of these tools support bitcode
>> transparently, but not all build systems use LLVM versions of these
>> tools, particularly build systems that support a variety of compilers,
>> or legacy build systems.
>
>
> If a build system can do
> CC=clang
> why wouldn't it be able to do
> AR=llvm-ar
> ?

That assumes that the LLVM tools are all deployed in the build system,
and adds a requirement for using clang in this mode that wasn't there
before when using clang for -O2. We are trying to make the transition
from clang -O2 to clang -O2 + ThinLTO as seamless as possible.

I'd just like the point out that downthread serious suggestions are being
fielded to use a nonstandard ELF header or nonstandard bits marking the
header. This "adds a requirement".

At this point at least 1 (the only?) concrete deployment use case (besides
yours) that has been brought up in the ThinLTO RFC threads is inconvenienced
by this design decision. This suggests that native object wrapping doesn't
offer as much seamlessness as it seems.

One big goal is to make it as painless as possible to transition from
plain -O2 to -O2+thinlto. Users of clang who don't already use LTO
have not had to use/deploy llvm versions of all of these tools
(llvm-nm, llvm-objcopy, llvm-ar, llvm-ranlib, etc), or the plugins for
the native versions of these tools, because they weren't dealing with
bitcode files. That is why we are prioritizing the native wrapped
approach for the initial implementation. For users of the gold linker
(which uses the LTOModule interfaces) this should make it much easier
to enable ThinLTO (from my browsing of ld64 source, it looks like
native-wrapped bitcode should be handled already there too due to the
handling being hidden behind the lto_module API).

Teresa

18

Quick question: Is the word required to support ThinLTO using llvm's native tools orthogonal to that required to supporting non-llvm tools? If not, would it make sense to start with a deployment of entirely LLVM based tools - since there seems to be general interest in that - and then come back to the non-llvm based tools separately?

Personally, I see both sides here. I can understand why you want to minimize build configuration changes - they tend to be painful - but I also am reluctant to design a major enhancement to LLVM under the assumption that LLVM's own tools aren't adequate for the purpose. That seems like it would be majorly problematic from the perspective of the project as a whole.

(I realize that LLVM's tools could simply extract the bitcode out of the wrapper file, but that seems unnecessarily complex for an *initial* LLVM only solution.)

Philip

19

My earlier statement about wrapping things in a native object file
held
in that it is controversial. It appears to be still central to your
design.

It may help to look at the problem from a different viewpoint: LLVM
is
not a compiler. It is a framework that can be used to make
compiler-like
tools.

From that view, it no longer makes sense to discuss "the plugin," or
gold, or $AR, because there isn't just one of any of those things.
ld64
isn't the only outlier linker to consider. We have our own linker at
Sony,
for example. From this perspective, then it makes more sense to
consider
replacing the binary utilities with ones that support bitcode,
because from
a user-perspective, all of the linkers already transparently support
bitcode
directly today, as do ar, nm, etc. This has been necessary for the
regular
LTO process.

Hi Alex,

It's true that the LLVM versions of these tools support bitcode
transparently, but not all build systems use LLVM versions of these
tools, particularly build systems that support a variety of compilers,
or legacy build systems.

If a build system can do
CC=clang
why wouldn't it be able to do
AR=llvm-ar
?

That assumes that the LLVM tools are all deployed in the build system,
and adds a requirement for using clang in this mode that wasn't there
before when using clang for -O2. We are trying to make the transition
from clang -O2 to clang -O2 + ThinLTO as seamless as possible.

I'd just like the point out that downthread serious suggestions are being
fielded to use a nonstandard ELF header or nonstandard bits marking the
header. This "adds a requirement".

At this point at least 1 (the only?) concrete deployment use case
(besides
yours) that has been brought up in the ThinLTO RFC threads is
inconvenienced
by this design decision. This suggests that native object wrapping
doesn't
offer as much seamlessness as it seems.

One big goal is to make it as painless as possible to transition from
plain -O2 to -O2+thinlto. Users of clang who don't already use LTO
have not had to use/deploy llvm versions of all of these tools
(llvm-nm, llvm-objcopy, llvm-ar, llvm-ranlib, etc), or the plugins for
the native versions of these tools, because they weren't dealing with
bitcode files. That is why we are prioritizing the native wrapped
approach for the initial implementation. For users of the gold linker
(which uses the LTOModule interfaces) this should make it much easier
to enable ThinLTO (from my browsing of ld64 source, it looks like
native-wrapped bitcode should be handled already there too due to the
handling being hidden behind the lto_module API).

Teresa

Quick question: Is the word required to support ThinLTO using llvm's native
tools orthogonal to that required to supporting non-llvm tools? If not,
would it make sense to start with a deployment of entirely LLVM based tools
- since there seems to be general interest in that - and then come back to
the non-llvm based tools separately?

Personally, I see both sides here. I can understand why you want to
minimize build configuration changes - they tend to be painful - but I also
am reluctant to design a major enhancement to LLVM under the assumption that
LLVM's own tools aren't adequate for the purpose. That seems like it would
be majorly problematic from the perspective of the project as a whole.

(I realize that LLVM's tools could simply extract the bitcode out of the
wrapper file, but that seems unnecessarily complex for an *initial* LLVM
only solution.)

Hi Phillip,

You are right, we can hide most of the details of the input format
behind an API, and make most of the ThinLTO implementation blissfully
unaware of what format was used. Given that, I am working on figuring
out what the right API is first.

One other thing about the bitcode-only approach is that we need to
define how the function index etc will be represented in bitcode,
which will presumably require some IR changes. So in order to test the
distributed build integration and deploy more quickly, I was thinking
of the following proposal:
1) Define API first
2) Define and implement native object format support (specifically ELF
first since I need that to test the distributed build integration)
3) While 2 is in progress, propose bitcode extensions to handle the
ThinLTO info in bitcode
4) Implement bitcode writing/reading support for ThinLTO sections
5) Implement support for other native-object formats: COFF/MachO

Interspersed with 2-5 is the rest of the ThinLTO support that is
independent of the format as it uses the API from 1.

Thanks,
Teresa

20

FYI, I no longer need this change as I decided it was cleaner to use
the gold plugin approach of interacting directly with libObject. I've
made that change locally and reverted my LTO restructuring.

Thanks,
Teresa