'too many sections' again when building ASTMatchers/Dynamic/Registry.cpp on Windows

Hi Samuel,

We've observed that in the past few days, debug-builds with Visual Studio
have started failing again because too many sections are generated for
ASTMatchers/Dynamic/Registry.cpp.

This has been observed and discussed before in this email thread:
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2013-June/030102.html.

It seems that since then, bit by bit, the number of sections for
Registry.cpp
has been creeping up. I've done a quick experiment on linux to measure how
the size and the number of sections in Registry.cpp.o has been increasing
over
time. The table does show that your fix in r183768 did reduce the number of
sections a lot; but since then, the number of sections has been increasing
steadily and quite rapidly.

                Size of Register.cpp.o
svn id bytes sections
189448 27122264 28108
189353 27204656 28259
189031 26473624 27284
187500 26451288 26952
186000 21940824 23119
183769 19289872 20544
183768 33647984 35738

You did say on the previous email thread that you had a few more ideas on
how
to reduce the number of sections in Register.cpp.o. I'm wondering if you
would
be able to share the ideas you've got?

The problem is that there are a lot of AST nodes, and each node is getting
a whole bunch of classes instantiated, many of them with virtual methods.
For example, there are around 200 symbols related to clang::WhileStmt. It
instantiates TrueMatcher<clang::WhileStmt>, DynCastMatcher<clang::Stmt, ,
lang::WhileStmt>, BindableMatcher<clang::WhileStmt>,
MatcherInterface<clang::WhileStmt>,
SingleNodeMatcherInterface<clang::WhileStmt>,
VariadicOperatorMatcherInterface<clang::WhileStmt>,
Matcher<clang::WhileStmt>::WrappedMatcher, Matcher<clang::WhileStmt>,
IdMatcher<clang::WhileStmt>.
All of these have virtual methods, which means that there are 2 symbols per
constructor, 3 for the destructor and 1 for the vtable. This is without
counting the actual methods in the class.
We then do more method/class instantiations for the dynamic binding.

The main fix I wanted to try is to reduce the number of classes that get
instantiated. On the previous fix I removed vector<T> and list<T> for each
AST node. I just replaced those uses with plain arrays.
I will do another round of fixes. Do you have a target size/sections ?

At the moment, it seems clang cannot be build with visual studio in debug

mode; and the large file size of Register.cpp.o and the large number of
sections seems to be slowing down the compile process on other platforms
too.

In the worst case we could use plan A, which was splitting Register.cpp
into many smaller files. We don't need to have all this code in the same
compilation unit. I wanted to avoid doing this because it doesn't help to
the total size of the objects and makes the code more complex.

_Sam

Hi Samuel,

Thanks for the quick response.

I agree that the best way forward would be to reduce the number of sections/symbols produced
in Register.cpp.o. I’m afraid I don’t know enough about this part of clang to know how feasible
that is.

I’m afraid I don’t really have a good target for the maximum number of sections to be produced
in a single object file. I think the target should be to at least halve the number of sections of what
is currently produced, and hope that future changes will not be creeping up the number of
sections again.

Thanks,

Kristof

Hi Samuel,****

** **

Thanks for the quick response.****

I agree that the best way forward would be to reduce the number of
sections/symbols produced
in Register.cpp.o. I’m afraid I don’t know enough about this part of clang
to know how feasible
that is.****

I’m afraid I don’t really have a good target for the maximum number of
sections to be produced
in a single object file. I think the target should be to at least halve
the number of sections of what
is currently produced, and hope that future changes will not be creeping
up the number of
sections again.

Just as fyi, I already submitted one change that took out ~10% of the
symbols.
I have 2 more for review with similar benefit (
http://llvm-reviews.chandlerc.com/D1564 and
http://llvm-reviews.chandlerc.com/D1560).
I think there are a couple easy ones after those, but I haven't explored
them yet.
_Sam

Thanks Sam,

I can confirm that debug builds with Visual Studio now succeed again.

Kristof