RFC: A new llvm-dlltool driver and llvm-lib driver improvements

Hey llvm’ers,

I have been working on a dlltool replacement for llvm.
Here is my initial differential https://reviews.llvm.org/D29892
It is based on some functionality that already exists in lld.
I added functionality to support, PE COFF Weak Externals and of course a front end to actually use it.
I believe the work here can also be used for llvm-lib and lessen the load on lld.
I would like some comments about how this could be be structured to live in llvm with a shared code base across lib ar and dlltool.
I also have a section below called “Difference from lib” which is somewhat of a rationale for the tool.

Many Thanks,
Martell

Context

LIB.exe doesn’t not have support for this but dlltool does.

LIB.exe does not have support for this but dlltool does.

I should proof read more :slight_smile:

Hi Martell,

I wonder how llvm-dlltool would fit in the entire picture of mingw support.
I don't think dlltool is the last missing piece. What do you need to do
other than that to fully support mingw using LLVM toolchain?

Hey llvm'ers,

I have been working on a dlltool replacement for llvm.
Here is my initial differential https://reviews.llvm.org/D29892
It is based on some functionality that already exists in lld.
I added functionality to support, PE COFF Weak Externals and of course a
front end to actually use it.
I believe the work here can also be used for llvm-lib and lessen the load
on lld.
I would like some comments about how this could be be structured to live
in llvm with a shared code base across lib ar and dlltool.
I also have a section below called "Difference from lib" which is somewhat
of a rationale for the tool.

Many Thanks,
Martell

Context

Awhile back I talked to various llvm'ers about getting mingw-w64 support
for lld.
There were a few issues raised but the main issue was that mingw-w64
should try best to comply with the PECOFF spec, adding support for custom
sections and various binutils/mingw hacks would impact the performance of
the COFF linker and in general is not something that lld should support.

It's not a performance issue but a code maintenance issue. The initial
patches to support mingw was trying to add a new linker driver and a
different linkin semantics to the COFF linker which seemed too complicated
to me. IIRC, I suggested adding a shim which translates GNU command line
arguments to MSVC linker arguments. Didn't it work?

Hey Rui,

I wonder how llvm-dlltool would fit in the entire picture of mingw support. I don’t think dlltool is the last missing piece. What do you need to do other than that to fully support mingw using LLVM toolchain?

Other then changing lib/MC/WinCOFFStreamer.cpp to not use -aligncomm within the EmitCommonSymbol function and a single patch for mingw-w64 itself to pre-populate it’s .ctors and .dtors list, so llvm-dlltool is infact the only missing part really.
This gives us a fully working clang based mingw-w64 C compiler.

C++ and exception handling is a different story.

libc++ is somewhat working with the following test results

Expected Passes : 2188
Expected Failures : 44
Unsupported Tests : 588
Unexpected Failures: 2816

I was able to build it with exceptions disabled and I actually managed to bootstrap llvm and clang itself.
It didn’t run very well though as you can imagine based on the above tests.

It’s not a performance issue but a code maintenance issue. The initial patches to support mingw was trying to add a new linker driver and a different linkin semantics to the COFF linker which seemed too complicated to me. IIRC, I suggested adding a shim which translates GNU command line arguments to MSVC linker arguments. Didn’t it work?

There were just so many differences between link and ld I never followed down that path. I pushed forward with the short import library support based on your later suggestions. My patch to hack lld into accepting some very basic gnu front end arguments was enough to get all the above working which was enough to develop further.

Best,
Martell

Hey Rui,

I wonder how llvm-dlltool would fit in the entire picture of mingw
support. I don't think dlltool is the last missing piece. What do you need
to do other than that to fully support mingw using LLVM toolchain?

Other then changing `lib/MC/WinCOFFStreamer.cpp` to not use -aligncomm
within the EmitCommonSymbol function and a single patch for mingw-w64
itself to pre-populate it's .ctors and .dtors list, so llvm-dlltool is
infact the only missing part really.

Also you need to make a change to LLD/COFF to accept GNU command arguments,
right? (Looks like you already have that patch locally.)

Also you need to make a change to LLD/COFF to accept GNU command arguments, right? (Looks like you already have that patch locally.)

Yes

Also you need to make a change to LLD/COFF to accept GNU command

arguments, right? (Looks like you already have that patch locally.)

Yes

I wonder if it can be a wrapper for LLD/COFF. It can be an in-process
wrapper, meaning that you could add a main function for mingw which
translates all arguments into the MSVC style and then invoke the COFF
linker's main function.

I wonder if it can be a wrapper for LLD/COFF. It can be an in-process wrapper, meaning that you could add a main function for mingw which translates all arguments into the MSVC style and then invoke the COFF linker’s main function.

That should work, if I am understanding this correctly I can create an argument parser (probably partially based on the ELF one) then I can set the data in the COFF Config object and call COFF LinkerDriver::run etc directly from that?

This proposal is for generally moving code from lld into llvm that could be part of lib.exe / dlltool, what are your thoughts here?

I wonder if it can be a wrapper for LLD/COFF. It can be an in-process

wrapper, meaning that you could add a main function for mingw which
translates all arguments into the MSVC style and then invoke the COFF
linker's main function.

That should work, if I am understanding this correctly I can create an
argument parser (probably partially based on the ELF one) then I can set
the data in the COFF Config object and call COFF LinkerDriver::run etc
directly from that?

No, I meant an even thinner wrapper which textually translates arguments.
For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
with Config object nor LinkerDriver::run and have absolutely zero knowledge
on the internals of LLD.

(Why I'm asking this is because I want to sure that there's no difference
between the regular Windows linker and mingw. If all differences are
superficial, the command line translator should just work. If not, there's
some semantic difference.)

This proposal is for generally moving code from lld into llvm that could be

part of lib.exe / dlltool, what are your thoughts here?

That should be fine.

No, I meant an even thinner wrapper which textually translates arguments. For example, the wrapper would translates “/out:foo.exe foo.obj” to “-o foo.exe foo.obj” and then call lld::COFFF:link(). It doesn’t do anything with Config object nor LinkerDriver::run and have absolutely zero knowledge on the internals of LLD.

Ohh okay I misunderstood.

(Why I’m asking this is because I want to sure that there’s no difference between the regular Windows linker and mingw. If all differences are superficial, the command line translator should just work. If not, there’s some semantic difference.)

The libraries can be used with MSVC link.exe directly assuming the correct arguments are passed to the linker.
I tested this after creating a working llvm-dlltool.
The only change I had to make to support this was alter ming-w64 crt to change all references from image_base to _ImageBase to support that

That should be fine.

Great

No, I meant an even thinner wrapper which textually translates arguments.

For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
with Config object nor LinkerDriver::run and have absolutely zero knowledge
on the internals of LLD.

Ohh okay I misunderstood.

(Why I'm asking this is because I want to sure that there's no difference

between the regular Windows linker and mingw. If all differences are
superficial, the command line translator should just work. If not, there's
some semantic difference.)

The libraries can be used with MSVC link.exe directly assuming the correct
arguments are passed to the linker.
I tested this after creating a working llvm-dlltool.
The only change I had to make to support this was alter ming-w64 crt to
change all references from __image_base__ to _ImageBase to support that

You may be able to define __ImageBase as a weak external symbol to
__image_base__. Then, if __ImageBase is not defined, all references against
__ImageBase will be resolved using __image_base__.

No, I meant an even thinner wrapper which textually translates arguments.

For example, the wrapper would translates "/out:foo.exe foo.obj" to "-o
foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do anything
with Config object nor LinkerDriver::run and have absolutely zero knowledge
on the internals of LLD.

Ohh okay I misunderstood.

(Why I'm asking this is because I want to sure that there's no difference

between the regular Windows linker and mingw. If all differences are
superficial, the command line translator should just work. If not, there's
some semantic difference.)

The libraries can be used with MSVC link.exe directly assuming the
correct arguments are passed to the linker.
I tested this after creating a working llvm-dlltool.
The only change I had to make to support this was alter ming-w64 crt to
change all references from __image_base__ to _ImageBase to support that

You may be able to define __ImageBase as a weak external symbol to
__image_base__. Then, if __ImageBase is not defined, all references against
__ImageBase will be resolved using __image_base__.

Or you could have your wrapper driver pass
"/alternatename:__image_base__=__ImageBase".

Peter

No, I meant an even thinner wrapper which textually translates

arguments. For example, the wrapper would translates "/out:foo.exe foo.obj"
to "-o foo.exe foo.obj" and then call lld::COFFF:link(). It doesn't do
anything with Config object nor LinkerDriver::run and have absolutely zero
knowledge on the internals of LLD.

Ohh okay I misunderstood.

(Why I'm asking this is because I want to sure that there's no

difference between the regular Windows linker and mingw. If all differences
are superficial, the command line translator should just work. If not,
there's some semantic difference.)

The libraries can be used with MSVC link.exe directly assuming the
correct arguments are passed to the linker.
I tested this after creating a working llvm-dlltool.
The only change I had to make to support this was alter ming-w64 crt to
change all references from __image_base__ to _ImageBase to support that

You may be able to define __ImageBase as a weak external symbol to
__image_base__. Then, if __ImageBase is not defined, all references against
__ImageBase will be resolved using __image_base__.

Or you could have your wrapper driver pass "/alternatename:__image_base__
=__ImageBase".

Ah, that's much better indeed.

Ohh nice.
With that method I can support it without upsetting ld users by introducing an api breakage.

I took a look at the code. Looks like you need a library to create import library files in LLVM and use that from llvm-dlltool and LLD. Is that what you are planning?

Yes, that would be the long term plan.
llvm-link ideally would share the same library.
I’m not sure where it should live though.

Yes, that would be the long term plan.
llvm-link ideally would share the same library.
I'm not sure where it should live though.

What is the short-term plan? I don't think you can submit this llvm-dlltool
as-is because it has to much duplicate code with LLD.

Well that means it would just be the only plan then.
I assume the first step would be to move the code from lld into this new library and lld can use that? I can then re add the extra functionality for weak externals?
We can then have lib and dlltool use this then.

The code in LLD is somewhat tightly coupled with the rest of the linker. Probably it is better to create a new library rather than trying to split LLD into two. I believe this would result in a better API. Of course you can reuse code, but what we need is a clean interface.

The library shouldn’t be complicated. In fact, it would export only one function.

std::vector<uint8_t> createImportLibrary(std::vector)

I.e. it takes a list of DLLExported symbols and returns a .lib file. ExportSymbol type contains information about DLLExported symbols.

The code in LLD is somewhat tightly coupled with the rest of the linker. Probably it is better to create a new library rather than trying to split LLD into two. I believe this would result in a better API. Of course you can reuse code, but what we need is a clean interface.

The library shouldn’t be complicated. In fact, it would export only one function.

std::vector<uint8_t> createImportLibrary(std::vector)

I.e. it takes a list of DLLExported symbols and returns a .lib file. ExportSymbol type contains information about DLLExported symbols.