Hi again rui, 
I’ve got all the patches into llvm and clang for supporting mingw-w64 via compiler-rt and now we are able to build a full mingw-w64 toolchain without gcc
With great help from yaron and rnk.
I’ve CC’d them as they might have interest in seeing this target through with me to the end
So I have again turned my attention to LLD so that we can also remove ld as a depend for this target
Previously I raised the issue that while lld would infact link exe’s for me the issue was the sections were malformed and thus crashed on launch.
I’ve done some debugging reguarding this and discovered it is due to differences between what MSVC’s lib.exe and binutils dlltool produces
I have attached a very simple .def file and generated libs via dlltool along with an objdump -s log of the lib.exe version
I don’t know too much about the internals of lld to be able to tell the exact cause of the problem but there are 3 specific differences I noted.
-
With dlltool each object has the name of a compiled .o file this is the one it generates when building the code section
With lib each object has the dll name
-
With dlltool section 7 is used for the dll name and the function names with lib section 6 is used
-
Other differences are that there is no debug section with dlltool and the text section comes first rather than last with lib (that probably doesn’t matter much though)
Please see attached notes.txt with objdump’s of both libs
Can could you give me your thoughts on this and how best we support both layouts in COFF/PECOFF ?
What source files should I be looking at to patch this or is this something you can support with very little effort ?
I don’t know if you would be opposed to supporting this or not being honest dlltool should generate libs using the same section number as lib.exe but we can’t exactly change that now because existing toolchains and libs have already been built with this and changing that would break them.
If you are opposed I would like you to still tell me what I have to change to get it working after I do that I can narrow the scope for getting a patch up streamed into binutils to fix this
Kind Regards
Martell
libuser32.a (2.81 KB)
user32.def (53 Bytes)
I forgot to attach the notes.txt with the objdump.
notes.txt (4.36 KB)
After some more digging and creating a few testcases in lld I have narrowed it down to
The fact that dlltool generates
Contents of section .idata$7:
0000 55534552 33322e64 6c6c0000 USER32.dll..
Where as lld expects
Contents of section .idata$6:
0000 55534552 33322e64 6c6c0000 USER32.dll..
I recreated the hello64.test using dlltool for the lib and here is the asm dump of the final exe
hello64gnu.exe: file format COFF-x86-64
Disassembly of section .text:
.text:
3000: ff 25 26 f0 ff ff jmpq *-4058(%rip)
3006: 90 nop
3007: 90 nop
3008: ff 25 26 f0 ff ff jmpq *-4058(%rip)
300e: 90 nop
300f: 90 nop
3010: 48 83 ec 28 subq $40, %rsp
3014: 48 c7 c1 00 00 00 00 movq $0, %rcx
301b: 48 8d 15 e4 df ff ff leaq -8220(%rip), %rdx
3022: 4c 8d 05 d7 df ff ff leaq -8233(%rip), %r8
3029: 41 b9 00 00 00 00 movl $0, %r9d
302f: e8 d4 ff ff ff callq -44
3034: b9 00 00 00 00 movl $0, %ecx
3039: e8 c2 ff ff ff callq -62
So I am fairly certain this is the cause.
Many Thanks
Martell
Hey guys,
So I was able to modify dlltool to produce the exact same layout as lib.exe with the same section numbers etc.
I’ve first managed to first create the correct section so that lld gives me link errors and then resolve those errors to create an exe.
This is one thing that is still missing that sticks out like a sore thumb
In the actual imports of the functions the objects are very different
As you can see lib.exe generates a very simple function import object.
IMPORT_OBJECT[1-N]
→ contains an IMPORT_OBJECT_HEADER and DATA
data example → _MessageBoxA.USER32.dll
dlltool however doesn’t create an import object but infact has a complete IMAGE_FILE
IMPORT_OBJECT[1-N]
→ contains an IMAGE_FILE_HEADER and is a full object with SECTIONS
the .text section has the stub code.
the .idata$6 section has the name of the function
it starts with 0x01 0x4D then MessageBoxA without an underscore
The dll is not referenced and is assumed to be the one listed in the first object in .idata$6
This is where the actual problem lies because the dll functions never get resolved
Have you guys got any thoughts on weither we should support this or not within lld and if so how ?
There are some interesting things here
For example the gnu linker provides the section with the stub code and the offset for injection.
Should we be using the stub code provided in the text section or should we just ignore it and use the one in lld
They are the same anyway but its just interesting how the dlltool version gives us that option.
Kind Regards
Martell
For example the gnu linker provides the section with the stub code and the offset for injection.
Meant to say dlltool here not gnu linker
Hi Martell.
I too was trying to make the new COFF linker recognize the libraries generated by binutils, but it wasn’t resolving the DLLs, just like you described. Apparently it is not identifying the libraries as import libraries but archives.
For testing purposes, i’ve managed to make the old COFF linker work with a simple test case. Had to make two little changes, and link the exe manually, because the linker doesn’t understand the GNU flags, but it works. By the way, even the MS linker works with the binutils generated import libraries.
Hi Ivan,
For testing purposes, I’ve managed to make the old COFF linker work with a simple test case
Have you got patches for this that I can look at ?
Had to make two little changes, and link the exe manually, because the linker doesn’t understand the GNU flags
I’ve solved the issue of having a GNU driver for the linker so that’s not an issue anymore
I sent in patches to the mailing list on this previously
I keep them updated over in the msys2 project btw
By the way, even the MS linker works with the binutils generated import libraries.
Yes i assume binutils was based from a very old version of a MS linker which did this layout at the time.
lld is so much newer and the MS linker has been updated to a more modern layout which is why we have this issue
Kind Regards
Martell
Hi Martell,
Sorry for the belated response. I missed this email.
I took a quick look at the library file you have attached and noticed that the file consists of regular object files. LLD expects that all dllimported symbols are described using “short import library” (PE/COFF spec 7). Because I didn’t test LLD with dlltool-style archive files, it’s no surprise that it didn’t work (although it should theoretically work).
Do you think that you can make dlltool to generate short import libraries? If you can, it would be the easiest way to solve the issue.
Hi Rui,
Thanks for getting back to me on this
Do you think that you can make dlltool to generate short import libraries? If you can, it would be the easiest way to solve the issue.
On my current tests ld seems to not like libraries built with short import libraries so this would require an ABI break in binutils.
dlltool has support for loads of legacy targets and changing this would also possibly break some of those targets so
Rather than working on dlltool I opted for creating a new tool for mingw-64 to generate libs from def files using the “short import library”
Another reason for me doing this is to have a complete clang based mingw-w64 toolchain without any gnu gcc or binutils dependencies.
It would be ideal however if we could support what dlltool currently produces in lld.
mingw.org toolchains will not be using the new tool I am creating and every distribution of mingw would have to rebuild their crt’s just to work with lld.
This isn’t ideal when clang is currenly used like a drop in add on for mingw-64 toolchains.
So if it is an easy enough fix we should support this in lld
Especially when MSVC already does.
Is there much involved with supporting this on the lld end?
Many Thanks
Martell
It’s doable to support dlltool-style import libraries by LLD. .idata sections in dlltool-style libraries are designed so that they will naturally construct the import descriptor table by just sorting them by section name. Supporting them is not hard.
One thing we need to do is that currently we construct the import descriptor directly from short import libraries. We don’t create intermediate .idata$X sections from short import libraries. That means we cannot mix dlltool-style and short import libraries at this moment. We need to do something for that. Except that, supporting dlltool-style libraries should be straightforward.
It’s doable to support dlltool-style import libraries by LLD. .idata sections in dlltool-style libraries are designed so that they will naturally construct the import descriptor table by just sorting them by section name. Supporting them is not hard.
I don’t really know my way around the code base of COFF well enough todo this myself.
If you can point me in the right direction as to what functions/sources to be looking at I think I can come up with a patch.
If you think you can do it fairly easily I can also create a test case for you if you want ?
One thing we need to do is that currently we construct the import descriptor directly from short import libraries. We don’t create intermediate .idata$X sections from short import libraries. That means we cannot mix dlltool-style and short import libraries at this moment. We need to do something for that. Except that, supporting dlltool-style libraries should be straightforward.
This is a very unique scenario and I don’t know of any tool that generates both styles of libraries.
I can create one if we need it for a test case but I think this would be overkill
It's doable to support dlltool-style import libraries by LLD. .idata
sections in dlltool-style libraries are designed so that they will
naturally construct the import descriptor table by just sorting them by
section name. Supporting them is not hard.
I don't really know my way around the code base of COFF well enough todo
this myself.
If you can point me in the right direction as to what functions/sources to
be looking at I think I can come up with a patch.
If you think you can do it fairly easily I can also create a test case for
you if you want ?
If it's not urgent, I can take a look after finishing up x86 COFF port and
fixing up remaining x64 bugs.
One thing we need to do is that currently we construct the import
descriptor directly from short import libraries. We don't create
intermediate .idata$X sections from short import libraries. That means we
cannot mix dlltool-style and short import libraries at this moment. We need
to do something for that. Except that, supporting dlltool-style libraries
should be straightforward.
This is a very unique scenario and I don't know of any tool that generates
both styles of libraries.
I can create one if we need it for a test case but I think this would be
overkill
This is not actually very unique but rather common because you usually
always want to link against kernel32.lib which contains short import
libraries.
If it’s not urgent, I can take a look after finishing up x86 COFF port and fixing up remaining x64 bugs.
I will create a test case for it then
This is not actually very unique but rather common because you usually always want to link against kernel32.lib which contains short import libraries.
Originally I thought you meant having both import types in the same lib.
You actually meant 2 separate libs one with each type.
I would note however that mingw doesn’t use the kernel32.lib provided by MS
It has def file that was originally generated from the dll with the gendef tool which is then given to dlltool when building the mingw crt.
If it is a mingw lib being used with visual studio however then yes this needs to be supported
There are projects that have dll’s built with mingw-w64 and that are linked into visual studio like the vlc media player windows 8 app