lld: --start-lib and --end-lib

Rafael,

You seems to be the person who added --start-lib and --end-lib to gold (*1), so I think you are the best person to ask about it. :slight_smile:

What is the intended use case of the options? Do you think we should support that in LLD?

If we want to support that, then I think we can think of two implementation strategies:

  • Link object files between --start-lib and --end-lib normally and garbage collect unused sections, or
  • create lazy symbols for object files between --start-lib and --end-lib and handle just like archive files.

What do you think?

*1 https://sourceware.org/ml/binutils/2010-03/msg00233.html

I hacked up a patch to support --start-lib and --end-lib in the second way.

http://reviews.llvm.org/D18814

In this patch, I introduced a new notion, LazyObject. That is analogous to Archive file type, but that works for a single object file instead of an archive file. This patch is +160 and -26 lines, so I can say that it is pretty simple.

This is not directly related to --start-lib and --end-lib, but LazyObject may be useful for --gc-sections as well. This is the suggestion I got in the EuroLLVM 2016 meeting (I’m sorry but I don’t remember who his name), but he suggested handling all object files as lazy object files for --gc-sections. It means by default no object files are linked. The entry point symbol would link one object file, and that triggers all used object files to be instantiated. I think this is very interesting idea because we don’t need to do mark-sweep garbage collection to do --gc-sections; instead, everything will be linked only when they are needed.

I believe the intent is that “–start-lib foo1.o foo2.o --end-lib” should act exactly the same as if you had given it an archive “tmp.a” which contained “foo1.o” and “foo2.o” in it.

That doesn’t really seem related to gc-sections: either way, you still need to have the normal logic to find the correct set of object files to include, at least in order to get the correct set of global constructors and such.

Rafael,

You seems to be the person who added --start-lib and --end-lib to gold (*1),
so I think you are the best person to ask about it. :slight_smile:

What is the intended use case of the options? Do you think we should support
that in LLD?

It was initially created for cluster builds. Running ar is really fast
if using thin archives, but one still has to copy the files to one
machine to create the archive.

Using --start-lib/--end-lib solves that problem. I find it a neat
feature, but to the best of my knowledge to this day it is only used
within google.

If we want to support that, then I think we can think of two implementation
strategies:

- Link object files between --start-lib and --end-lib normally and garbage
collect unused sections, or
- create lazy symbols for object files between --start-lib and --end-lib
and handle just like archive files.

What do you think?

You have to create lazy symbols. The semantics are not the same given
that some things (static constructors) are not gced.

Cheers,
Rafael

I hacked up a patch to support --start-lib and --end-lib in the second way.

http://reviews.llvm.org/D18814

In this patch, I introduced a new notion, LazyObject. That is analogous to
Archive file type, but that works for a single object file instead of an
archive file. This patch is +160 and -26 lines, so I can say that it is
pretty simple.

The description sounds reasonable. Taking a look.

This is not directly related to --start-lib and --end-lib, but LazyObject
may be useful for --gc-sections as well. This is the suggestion I got in the
EuroLLVM 2016 meeting (I'm sorry but I don't remember who his name), but he
suggested handling all object files as lazy object files for --gc-sections.
It means by default no object files are linked. The entry point symbol would
link one object file, and that triggers all used object files to be
instantiated. I think this is very interesting idea because we don't need to
do mark-sweep garbage collection to do --gc-sections; instead, everything
will be linked only when they are needed.

That is a very big change in semantics given static constructors. Even
llvm itself uses static constructors to register passes, so including
a file and then gcing is not the same an not gcing.

Cheers,
Rafael

I hacked up a patch to support --start-lib and --end-lib in the second way.

http://reviews.llvm.org/D18814

In this patch, I introduced a new notion, LazyObject. That is analogous to
Archive file type, but that works for a single object file instead of an
archive file. This patch is +160 and -26 lines, so I can say that it is
pretty simple.

Have you tried creating a thin archive in memory and loading it in the
normal way? That seems potentially simpler.

-- Sean Silva

I hacked up a patch to support --start-lib and --end-lib in the second
way.

http://reviews.llvm.org/D18814

In this patch, I introduced a new notion, LazyObject. That is analogous
to Archive file type, but that works for a single object file instead of an
archive file. This patch is +160 and -26 lines, so I can say that it is
pretty simple.

Have you tried creating a thin archive in memory and loading it in the
normal way? That seems potentially simpler.

Nevermind, that would require doing a bunch of redundant work building the
archive symbol table.

-- Sean Silva