build issue on mingw gcc, need some help

Dear people,

I've got some help from the people of mingw64 to get libc++abi, libc++ and
llvm to build with the mingw included gcc compiler. I was able to build
libc++abi and libc++ with g++. The static library libc++.a did work with a
little example program.

Then I was trying to get the shared version: libc++.dll to build with cmake.
This didn't work. I have to say I'm pretty annoyed by the complexity and
large makefile sizes using cmake. You can call me whatever you want about my
decision to stop using cmake for this. Also I believe it's important that I
for myself gain a better understanding of the building process.

Still experts here on the building process design this into cmake scripts
and python scripts. But unfortunately in this way it is too complicated to
fully understand this for myself so I can apply the knowledge into my own
makefile. I do found files like flags.make and link.txt to provide parts of
the g++ and ar calls to build the libraries. I hope to learn a bit more on
specific flags and defines.

I started to make a straightforward mingw64 makefile to build the shared
version of libc++abi. As I'm on Windows I believe _LIBCXXABI_FUNC_VIS should
end up being __declspec(dllexport) but when I do a preprocess only on a file
like cxa_personality.cpp I see _LIBCXXABI_FUNC_VIS end up being
__attribute__((dllexport)). Somehow the file libc++abi.dll is build but it
is just about 50 kB and llvm-readobj -coff-exports shows nothing.

My question is what kind of define or flag might have caused this to happen?
And how can I correct it?

Attached is the makefile I currently have. Doesn't look great but you know
I'm working on it.

Best regards,
Maarten

test.mak (6.44 KB)

Hi,

I started to make a straightforward mingw64 makefile to build the shared
version of libc++abi. As I'm on Windows I believe _LIBCXXABI_FUNC_VIS should
end up being __declspec(dllexport) but when I do a preprocess only on a file
like cxa_personality.cpp I see _LIBCXXABI_FUNC_VIS end up being
__attribute__((dllexport)).

GCC actually only supports the latter. The former is expanded into the latter by the preprocessor; if you dump the default defines by "gcc -E -dM - < /dev/null" (or similar with an empty normal input file)", you'll find "#define __declspec(x) __attribute__((x))".

Somehow the file libc++abi.dll is build but it is just about 50 kB and llvm-readobj -coff-exports shows nothing.

My question is what kind of define or flag might have caused this to happen?
And how can I correct it?

First off, I haven't been able to build libc++abi standalone as a DLL, as there are some amount of circular dependencies between libc++abi and libc++. (If linking doesn't end up including everything, you might not notice this.)

Due to this, when building libc++ as a DLL, I build libc++abi as a static library, but manually override defines to get the same effect as if I had built libc++abi dynamically, and making the libc++ includes behave as if building libc++ itself, as the object files will be included within libc++.dll. Likewise, for libc++, I add defines to make libc++abi headers behave as if building libc++abi itself:

https://github.com/mstorsjo/llvm-mingw/blob/b9eb66bc/build-libcxx.sh#L143
https://github.com/mstorsjo/llvm-mingw/blob/b9eb66bc/build-libcxx.sh#L181

In the end, I top it off by linking the DLL with -Wl,--export-all-symbols. Without that, for me, a number of symbols aren't exported even if they are expected to. I believe that's a Clang-specific issue though, but I haven't analyzed it further yet, as this hack/workaround seems to work well enough for now.

// Martin

Hi Martin and others,

Remember I was trying to get shared LLVM libraries: libc++abi and libc++ to
build with the mingw64 GCC compiler on Windows. The goal is to have separate
libc++dll and libc++abi.dll files.

I'm working on a Python script that generate a single mingw makefile that
can build both libraires. I'll show it when it's finished. I believe the
finish is near. Please see further below.

I started to make a straightforward mingw64 makefile to build the shared
version of libc++abi. As I'm on Windows I believe _LIBCXXABI_FUNC_VIS
should
end up being __declspec(dllexport) but when I do a preprocess only on a
file
like cxa_personality.cpp I see _LIBCXXABI_FUNC_VIS end up being
__attribute__((dllexport)).

GCC actually only supports the latter. The former is expanded into the
latter by the preprocessor; if you dump the default defines by
"gcc -E -dM - < /dev/null" (or similar with an empty normal input file)",
you'll find "#define __declspec(x) __attribute__((x))".

Yeah, I learned that both: __declspec(dllexport) and
__attribute__((dllexport)) will do.

Somehow the file libc++abi.dll is build but it is just about 50 kB and
llvm-readobj -coff-exports shows nothing.

My question is what kind of define or flag might have caused this to
happen?
And how can I correct it?

To answer my own question: The low libc++abi.dll size was due a missing
linker option:
-Wl,--whole-archive cxx_objects.a -Wl,--no-whole-archive

First off, I haven't been able to build libc++abi standalone as a DLL, as
there are some amount of circular dependencies between libc++abi and
libc++. (If linking doesn't end up including everything, you might not
notice this.)

As libc++ and libc++abi sources are quite large. I first setup a little
example with two dlls that use functions of each. In that makefile I first
build two import libraries for each dll with dlltool and later linking the
object code against the import library of the other dll. No problems.

Regarding the circular dependencies I believe a link commands such as:
-Wl,--start-group <multiple circular dependency libs> -Wl,--end-group
is the proper way to ask gcc to figure this out.

Something on SO about this:
https://stackoverflow.com/questions/9380363/resolving-circular-dependencies-by-linking-the-same-library-twice

Due to this, when building libc++ as a DLL, I build libc++abi as a static
library, but manually override defines to get the same effect as if I had
built libc++abi dynamically, and making the libc++ includes behave as if
building libc++ itself, as the object files will be included within
libc++.dll. Likewise, for libc++, I add defines to make libc++abi headers
behave as if building libc++abi itself:

https://github.com/mstorsjo/llvm-mingw/blob/b9eb66bc/build-libcxx.sh#L143
https://github.com/mstorsjo/llvm-mingw/blob/b9eb66bc/build-libcxx.sh#L181

In the end, I top it off by linking the DLL with -Wl,--export-all-symbols.
Without that, for me, a number of symbols aren't exported even if they are
expected to. I believe that's a Clang-specific issue though, but I haven't
analyzed it further yet, as this hack/workaround seems to work well enough
for now.

// Martin

I hope to get it a little better than -Wl,--export-all-symbols.

Ok, please read with me, I try to be as clear as I could:
What I'm facing right now. The shared library build of libc++abi needs to
process the import library: libc++.dll.a. But the linker gives undefined
references to things like __imp__ZNSt8bad_castC1Ev. When I run the nm tool
on the archive libary cxx_objects.a I see symbols like these are all
available. But I believe when I run the dlltool to make the import libary
from cxx_objects.a I loose the visibility of the symbols (exported stuff).
In the libcxx source folder in the subdirectory lib there are exp files that
contain symbols for exports which needs to be applied somehow to the build
of libc++. The file libc++abi2.exp contain the exports which I believe I
would need. The symbol names match. In the build of libc++ in the cmake
style they seem to do it on the final linking stage with options
like -Wl,-reexported_symbols_list,<.exp file>.
I my situation it seems to make more sense to apply these symbols to the
creation of the import library. But I cannot figure out with GNU binutil
tool I should use for this and what options to use. I know I can do
an --input-def option on the call to dlltool. But that libc++abi2.exp is not
a Windows def file. So, I do need a little help with this.

Best regards,
Maarten Verhage

Hi Martin and others,

Somehow the file libc++abi.dll is build but it is just about 50 kB and llvm-readobj -coff-exports shows nothing.

My question is what kind of define or flag might have caused this to happen?
And how can I correct it?

To answer my own question: The low libc++abi.dll size was due a missing linker option:
-Wl,--whole-archive cxx_objects.a -Wl,--no-whole-archive

I presume you mean cxxabi_objects.a, for building libc++abi.dll? You don't want to do that with cxx_objects.a, otherwise you'd end up including all of libc++ in libc++abi.

The whole concept of -Wl,--whole-archive and a static archive when linking a dynamic library is a rather cmake specific roundabout way of doing things - if you are building your own makefiles, you could just as well pass the object files directly to the linker, especially as there shouldn't be any issue with command line length with the relatively low number of object files in libc++/libc++abi.

First off, I haven't been able to build libc++abi standalone as a DLL, as there are some amount of circular dependencies between libc++abi and libc++. (If linking doesn't end up including everything, you might not notice this.)

As libc++ and libc++abi sources are quite large. I first setup a little example with two dlls that use functions of each. In that makefile I first build two import libraries for each dll with dlltool and later linking the object code against the import library of the other dll. No problems.

Regarding the circular dependencies I believe a link commands such as:
-Wl,--start-group <multiple circular dependency libs> -Wl,--end-group
is the proper way to ask gcc to figure this out.

Something on SO about this:
https://stackoverflow.com/questions/9380363/resolving-circular-dependencies-by-linking-the-same-library-twice

This is unrelated. Resolving circular dependencies between two static libraris can indeed be done this way (with ld.bfd; with lld it's not necessary).

Resolving circular dependencies between two dynamic libraries does instead need you do to bootstrapping tricks with import libraries, which it seems you are trying to do.

Due to this, when building libc++ as a DLL, I build libc++abi as a static library, but manually override defines to get the same effect as if I had built libc++abi dynamically, and making the libc++ includes behave as if building libc++ itself, as the object files will be included within libc++.dll. Likewise, for libc++, I add defines to make libc++abi headers behave as if building libc++abi itself:

https://github.com/mstorsjo/llvm-mingw/blob/b9eb66bc/build-libcxx.sh#L143
https://github.com/mstorsjo/llvm-mingw/blob/b9eb66bc/build-libcxx.sh#L181

In the end, I top it off by linking the DLL with -Wl,--export-all-symbols. Without that, for me, a number of symbols aren't exported even if they are expected to. I believe that's a Clang-specific issue though, but I haven't analyzed it further yet, as this hack/workaround seems to work well enough for now.

// Martin

I hope to get it a little better than -Wl,--export-all-symbols.

Don't comment on that until you actually have gotten it all done - that part is not related to how they're built and linked, but whether the compiler emits dllexport for all the necessary symbols.

I've looked a bit closer on that matter now, and it's related to how libcxx does explicit template instantiation with visibility attributes (dllexport). Neither g++ nor clang in mingw mode will create dllexports for symbols from an explicit template instantiation with a dllexport attribute, while msvc and clang in msvc mode does. I can open a bug report with full details later.

Ok, please read with me, I try to be as clear as I could:
What I'm facing right now. The shared library build of libc++abi needs to process the import library: libc++.dll.a. But the linker gives undefined references to things like __imp__ZNSt8bad_castC1Ev.

Do the undefined referenes come from libc++.dll.a or the libc++abi object files themselves? The former should be a plain import library which shouldn't give any undefined references to that. And the libc++abi object files should define the symbol _ZNSt8bad_castC1Ev, shouldn't produce any references to the __imp_ prefixed symbol (since it's declared dllexport while building libc++abi).

When I run the nm tool on the archive libary cxx_objects.a I see symbols like these are all available.

What? cxx_objects.a, I presume, is an archive of the object files from libcxx. That should have undefined references to _ZNSt8bad_castC1Ev (or with an __imp_ prefix), but shouldn't provide a definition of it.

But I believe when I run the dlltool to make the import libary from cxx_objects.a I loose the visibility of the symbols (exported stuff).

Sorry but you have to be much much more precise than this, it's impossible to know exactly what you're implying here. Without even knowing what dlltool command you're running, on what files, it's pretty much impossible to say anything about this.

In the libcxx source folder in the subdirectory lib there are exp files that contain symbols for exports which needs to be applied somehow to the build of libc++. The file libc++abi2.exp contain the exports which I believe I would need. The symbol names match. In the build of libc++ in the cmake style they seem to do it on the final linking stage with options like -Wl,-reexported_symbols_list,<.exp file>.

I my situation it seems to make more sense to apply these symbols to the creation of the import library. But I cannot figure out with GNU binutil tool I should use for this and what options to use. I know I can do an --input-def option on the call to dlltool. But that libc++abi2.exp is not a Windows def file. So, I do need a little help with this.

You should be able to create a def file from it just by adding two lines at the top, "LIBRARY libc++abi.dll" and "EXPORTS" at the top of it, and create an import library with "dlltool -m <machine> -d libc++abi.def -l libc++abi.dll.a".

// Martin

This issue is now at https://bugs.llvm.org/show_bug.cgi?id=40256, framed as a Clang issue. If Clang is fixed to behave like GCC with respect to this (which probably makes the most sense, instead of making it behave like MSVC even thogh operating in MinGW mode), or already now with GCC, it should be possible to get the necessary export directives with the following untested patch for libcxx:

diff --git a/include/__config b/include/__config
index b8c2c33..fc1a472 100644
--- a/include/__config
+++ b/include/__config
@@ -706,7 +706,11 @@ namespace std {
  #define _LIBCPP_EXCEPTION_ABI _LIBCPP_DLL_VIS
  #define _LIBCPP_HIDDEN
  #define _LIBCPP_METHOD_TEMPLATE_IMPLICIT_INSTANTIATION_VIS
+#if defined(__MINGW32__) && defined(_LIBCPP_BUILDING_LIBRARY)
+#define _LIBCPP_TEMPLATE_VIS _LIBCPP_DLL_VIS
+#else
  #define _LIBCPP_TEMPLATE_VIS
+#endif
  #define _LIBCPP_ENUM_VIS

  #endif // defined(_LIBCPP_OBJECT_FORMAT_COFF)

// Martin

I believe you’d want to modify _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS and _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS instead, since they’re applied to explicit instantiation declarations and definitions, respectively. _LIBCPP_TEMPLATE_VIS is applied to class templates in general, not explicit instantiations specifically.

Yes, that's probably more correct - and the diff ends up neater as well:

diff --git a/include/__config b/include/__config
index b8c2c33..7bdd6b2 100644
--- a/include/__config
+++ b/include/__config
@@ -690,8 +690,13 @@ namespace std {
  # define _LIBCPP_OVERRIDABLE_FUNC_VIS
  #elif defined(_LIBCPP_BUILDING_LIBRARY)
  # define _LIBCPP_DLL_VIS __declspec(dllexport)
-# define _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS
-# define _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS _LIBCPP_DLL_VIS
+# if defined(__MINGW32__)
+# define _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS _LIBCPP_DLL_VIS
+# define _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS
+# else
+# define _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS
+# define _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS _LIBCPP_DLL_VIS
+# endif
  # define _LIBCPP_OVERRIDABLE_FUNC_VIS _LIBCPP_DLL_VIS
  #else
  # define _LIBCPP_DLL_VIS __declspec(dllimport)

// Martin

Hi Martin and others,

Sorry for my inaccurate description of the issues I'm facing. Rather than
trying to address the topics of misunderstanding between us I thought it
would be best to share what I currently have of the Python script and what
my makefile writes to the command line.

I put it all in the attachments. result.txt is the output of the makefile.
Although it's a lot of text I believe it is as specific as can be and I hope
it will be easy to navigate through. I'm certainly willing to explain what
I'm trying to achieve if need to. And I do also take criticism on my script.

The python script first would need some edits (in the beginning) to find the
folders of libc++ and libc++abi sources on your harddisk.

As per your suggestion I put the following section into __config file of the
libcxx include folder:

# if defined(__MINGW32__)
# define _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS _LIBCPP_DLL_VIS
# define _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS
# else
# define _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS
# define _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS _LIBCPP_DLL_VIS
# endif

But still undefined reference to __imp__ZNSt8bad_castC1Ev and more.

Yes, this is not supposed to help with this issue.

I also edited libc++abi2.exp to make a valid def file and applied this to
libc++. I believe it actually needs to be applied to libc++ and not
libc++abi.

Still the same undefined references.

Please let me know where I got it wrong?

Ok, so the problem is this. The missing symbols, __imp__ZNSt8bad_castC1Ev, means that libc++abi tries to import the bad_cast class from another DLL. This class is declared in libc++ headers, and the libc++ visibility flags indicate that symbols declared there will be accessed as dllimport. But in this case, libc++abi itself provides the bad_cast class, even though it's declared by libc++ headers, not libc++abi headers.

ld.bfd doesn't seem to able to handle this situation, while ld.lld and link.exe can fix up such situations, with a minor warning, e.g. like this:

lld-link: warning: test.o: locally defined symbol imported: _Z4funcv (defined in test2.o) [LNK4217]

As a minimal example of this situation, I created two test cpp files. test.cpp:

void __declspec(dllimport) func(void);
void __declspec(dllexport) mainfunc(void) {
   func();
}

test2.cpp:

void __declspec(dllexport) func(void);
void func(void) {
}

When compiling and linking these files with GCC/ld.bfd, I get the following:

$ x86_64-w64-mingw32-g++ -c test.cpp
$ x86_64-w64-mingw32-g++ -c test2.cpp
$ x86_64-w64-mingw32-g++ test.o test2.o -o test.dll -shared
test.o:test.cpp:(.text+0xb): undefined reference to `__imp__Z4funcv'
collect2: error: ld returned 1 exit status

When linking with ld.lld instead, linking succeeds but with the warning I quoted above.

I don't have any good suggestion on how to proceed with this, other than building libc++ and libc++abi into one single DLL. While building libc++abi, it would need to include libc++ headers as if _LIBCPP_BUILDING_LIBRARY was set, but only for the declarations of those specific classes that are provided by libc++abi, nothing else.

Although, I'm not sure if there's much code in libc++abi that actually has undefined references to libc++ symbols, so in that case it might just work to define _LIBCPP_BUILDING_LIBRARY for all of the build of libc++abi. Or maybe just at the top of individual source files that provide the classes that are declared in libc++ headers.

// Martin

Note that stdlib_exception.cpp and stdlib_new_delete.cpp already define _LIBCPP_BUILDING_LIBRARY. Perhaps stdlib_typeinfo.cpp should follow suit?

    > Hi Martin and others,
    >
    > Sorry for my inaccurate description of the issues I'm facing. Rather than
    > trying to address the topics of misunderstanding between us I thought it
    > would be best to share what I currently have of the Python script and what
    > my makefile writes to the command line.
    >
    > I put it all in the attachments. result.txt is the output of the makefile.
    > Although it's a lot of text I believe it is as specific as can be and I hope
    > it will be easy to navigate through. I'm certainly willing to explain what
    > I'm trying to achieve if need to. And I do also take criticism on my script.
    >
    > The python script first would need some edits (in the beginning) to find the
    > folders of libc++ and libc++abi sources on your harddisk.
    >
    > As per your suggestion I put the following section into __config file of the
    > libcxx include folder:
    >
    > # if defined(__MINGW32__)
    > # define _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS _LIBCPP_DLL_VIS
    > # define _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS
    > # else
    > # define _LIBCPP_EXTERN_TEMPLATE_TYPE_VIS
    > # define _LIBCPP_CLASS_TEMPLATE_INSTANTIATION_VIS _LIBCPP_DLL_VIS
    > # endif
    >
    >
    > But still undefined reference to __imp__ZNSt8bad_castC1Ev and more.
    
    Yes, this is not supposed to help with this issue.
    
    > I also edited libc++abi2.exp to make a valid def file and applied this to
    > libc++. I believe it actually needs to be applied to libc++ and not
    > libc++abi.
    >
    > Still the same undefined references.
    >
    > Please let me know where I got it wrong?
    
    Ok, so the problem is this. The missing symbols, __imp__ZNSt8bad_castC1Ev,
    means that libc++abi tries to import the bad_cast class from another DLL.
    This class is declared in libc++ headers, and the libc++ visibility flags
    indicate that symbols declared there will be accessed as dllimport. But in
    this case, libc++abi itself provides the bad_cast class, even though it's
    declared by libc++ headers, not libc++abi headers.
    
    ld.bfd doesn't seem to able to handle this situation, while ld.lld and
    link.exe can fix up such situations, with a minor warning, e.g. like this:
    
    lld-link: warning: test.o: locally defined symbol imported: _Z4funcv
    (defined in test2.o) [LNK4217]
    
    As a minimal example of this situation, I created two test cpp files.
    test.cpp:
    
    void __declspec(dllimport) func(void);
    void __declspec(dllexport) mainfunc(void) {
       func();
    }
    
    test2.cpp:
    
    void __declspec(dllexport) func(void);
    void func(void) {
    }
    
    When compiling and linking these files with GCC/ld.bfd, I get the
    following:
    
    $ x86_64-w64-mingw32-g++ -c test.cpp
    $ x86_64-w64-mingw32-g++ -c test2.cpp
    $ x86_64-w64-mingw32-g++ test.o test2.o -o test.dll -shared
    test.o:test.cpp:(.text+0xb): undefined reference to `__imp__Z4funcv'
    collect2: error: ld returned 1 exit status
    
    When linking with ld.lld instead, linking succeeds but with the warning I
    quoted above.
    
    I don't have any good suggestion on how to proceed with this, other than
    building libc++ and libc++abi into one single DLL. While building
    libc++abi, it would need to include libc++ headers as if
    _LIBCPP_BUILDING_LIBRARY was set, but only for the declarations of those
    specific classes that are provided by libc++abi, nothing else.
    
    Although, I'm not sure if there's much code in libc++abi that actually has
    undefined references to libc++ symbols, so in that case it might just work
    to define _LIBCPP_BUILDING_LIBRARY for all of the build of libc++abi. Or
    maybe just at the top of individual source files that provide the classes
    that are declared in libc++ headers.
    
    // Martin

Oh, indeed. Then that's definitely needed for the other stdlib_*.cpp as well, in order to get dllexport defined for the symbols it generates.

However, that isn't enough. Almost all source files in libcxxabi seem to refer to std::terminate, which is implemented by libcxxabi itself. Thus, the declaration of std::terminate shouldn't be marked dllimport anywhere in any of the translation units.

Therefore I suspect it'd be best to just define _LIBCPP_BUILDING_LIBRARY for all of it.

If there legitimately are functions that would need to be imported from libcxx, they can be linked just fine via thunks, and data symbols should be handled by the mingw specific auto-import mechanism.

// Martin

Today I have been making progress in resolving undefined references for
things that are declared by libc++ but defined in libc++abi. Including
std::terminate. I strongly believe it is possible to get two shared
libraries DLLs out of it with careful visibility preprocessor settings. In
Linux you can also get two shared libraries right? I'm also motivated to see
a smooth operation of the functionality of static libc++abi into the libc++
DLL. Like the builds of Martin.

If there is dissatisfaction against the ability of libc++ to have two DLLs
under Windows I would like to know. I believe I read somewhere that libc++
should also be able to interoperate the gcc abi: libsupc++. when libc++abi
is not used.

With some visibility tweaks I'm now getting the libc++abi.dll out of it.
However I'm currently stuck on the following issue.

In libcxx there is stdexcept.cpp and in libcxxabi there is
stdlib_stdexcept.cpp. A class like logic_error has the majority of its
implementation in stdexcept.cpp but there are also some fragments in
stdlib_stdexcept.cpp. I'm not able to get two DLLs from this spread
implementation. It makes me wonder why there is stdlib_stdexcept.cpp in
libc++abi? Would it be possible that a decision is made where this should
end up?

Best regards,
Maarten Verhage

As of now, libcxx can be built with Clang in MinGW mode as a working DLL, without -Wl,--export-all-symbols.

With GCC, the following two issues would have to be solved, to be able to make the DLL work properly without -Wl,--export-all-symbols:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89087
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89088

// Martin