'too many sections' when building an object file for llvm/clang

Hi, I'm building llvm & clang on a Windows 7 64-bit machine using
MinGW-w64 (it's been a long day getting everything to work).
Everything goes well up to 82% through the build process, then when it
tries to build Registry.cpp (specifically the one for the clang
dynamic AST matchers), MinGW's as.exe errors out with "too many
sections (55123)". I've done everything I know how to do to try and
avoid this, and I've done searching on the web about this, but it
seems that this type of error happens very rarely and there's no well
known solution.

This is the verbose output of the exact command that is being
executed: Trying to build llvm and clang, running into this at 82% ยท GitHub

I was wondering if anyone here has had this happen and/or knows how to
fix it - I would even go as far as modifying some of the MinGW-w64
source files to raise the limit on as, but I don't know how to do that
without guidance. I know that llvm/clang are not very well supported
on Windows and that I'm shooting in the dark here, but I really want
to use clang as my primary compiler.

This is my first time using a mailing list, I am sorry if there are
formatting problems - please let me know how to fix them if there are
any.

- Nicholas "LB" Braden

Hi,

I have been discussing the same issue on llvm-commit mailing-list.
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20130603/177054.
html

At the moment, the debug build of LLVM/Clang is broken on Windows.
This is because when compiling ASTMatchers/Dynamic/Registry.cpp,
the compilers hit the allowed limit of 2^16 sections in the
COFF file.

For MSVC it is possible to work around this problem by compiling
with /bigobj command line option. However, I don't know of an
equivalent command line option for Min-GW gcc.

On Linux, Registry.o (debug-mode) is about 34M in size and has
36986 sections. I can expect it impacting the link-time and memory
usage.

The last commit has resulted in the build failure, but I don't think
it is responsible for the failure.

I don't know what should be the correct solution. Hence, I am raising
it here for discussion.

Regards,
Manish

+Sam

Hi,

Hi,
I am the person working on the dynamic layer (ASTMatchers/Dynamic/) on top
the matcher library.

I have been discussing the same issue on llvm-commit mailing-list.

http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20130603/177054.
html

At the moment, the debug build of LLVM/Clang is broken on Windows.
This is because when compiling ASTMatchers/Dynamic/Registry.cpp,
the compilers hit the allowed limit of 2^16 sections in the
COFF file.

Looking at the sections, it seems that we have way too many functions
defined in the compilation unit.
The matchers in ASTMatchers.h rely a lot on templates and macros and most
of the matcher code is directly on the ASTMatchers.h file.
Each matcher instantiation seem to generate a couple hundred.
The dynamic registry is instantiating all* the matchers in Registry.cpp. I
think only the test file (ASTMatchersTest.cpp) would instantiate that many
matchers together in one compilation unit before.

I see a few solutions for this:
1. For any non-template matcher, move its code to an ASTMatchers.cpp to be
created. This would take away a big chunk of the size of Registry.cpp. I
don't know if this will just move the problem to ASTMatchers.cpp instead.
2. Split Registry.cpp in smaller compilation units, each with a subset of
the matchers. It should be easier than (1).

* the goal is to have them all, but we are not there yet, so the problem
would get worse later.

For MSVC it is possible to work around this problem by compiling

with /bigobj command line option. However, I don't know of an
equivalent command line option for Min-GW gcc.

On Linux, Registry.o (debug-mode) is about 34M in size and has
36986 sections. I can expect it impacting the link-time and memory
usage.

This library is separate from ASTMatchers on purpose to prevent bloat to
anyone not explicitly requiring the dynamic implementation.
If this is being built/linked for clang/clang++ then it is a mistake.

The last commit has resulted in the build failure, but I don't think
it is responsible for the failure.

On the last commit (or one of the latest) I expanded the list of matchers
to more than 150. I expect this is what caused it to go over the limit.

_Sam

Hi Samuel,

Thank you for looking at this issue.

I see a few solutions for this:
1. For any non-template matcher, move its code to an ASTMatchers.cpp
to be created. This would take away a big chunk of the size of
Registry.cpp. I don't know if this will just move the problem to
ASTMatchers.cpp instead.
2. Split Registry.cpp in smaller compilation units, each with a
subset of the matchers. It should be easier than (1).

I think it would be better to implement option (2) while you look
for a better solution. We are happy to do the testing at our end,
as none of build-bots are testing this configuration.

Regards,
Manish

Hi Samuel,

Thank you for looking at this issue.

> I see a few solutions for this:
> 1. For any non-template matcher, move its code to an ASTMatchers.cpp
> to be created. This would take away a big chunk of the size of
> Registry.cpp. I don't know if this will just move the problem to
> ASTMatchers.cpp instead.
> 2. Split Registry.cpp in smaller compilation units, each with a
> subset of the matchers. It should be easier than (1).

I think it would be better to implement option (2) while you look
for a better solution. We are happy to do the testing at our end,
as none of build-bots are testing this configuration.

I found what appears to be a solution for the problem without having to
break the file.
Removed a bunch of template instantiations from the marshaller functions.
In particular, std::list<Matcher<T>> and std::vector<Matcher<T>*> for each
node T. Just that one change seemed to take the unit below the limit.
Sent the change on http://llvm-reviews.chandlerc.com/D948. If you want,
please patch the change and let me know if it fixes the issue. I don't have
that environment, so I can't test it there.

The change was submitted.
I found more opportunities to reduce the number of symbols changing some
things in ASTMatchers.h, but that will take a little more work. We can do
it if necessary.
_Sam

Hi Sam,

Thank you for the work. I can confirm that we are no longer seeing the build
failure with MSVC.

Cheers,
Manish