clang build (Debug+Asserts) fails with Cygwin and Mingw (ASTContext.o: File too big)

Hello,

After switching from a Linux development host to Windows7 with Cygwin, I notice the following problem when building clang (trunk, HEAD):
(also happens with Mingw)

...

llvm[4]: Compiling ParseInit.cpp for Debug+Asserts build
as: /cygdrive/d/Projects/Cpp/llvm/cfgCygwin/tools/clang/lib/AST/Debug+Asserts/ASTContext.o: too many sections (33066)
/tmp/ccmtDH7w.s: Assembler messages:
/tmp/ccmtDH7w.s: Fatal error: can't write /cygdrive/d/Projects/Cpp/llvm/cfgCygwin/tools/clang/lib/AST/Debug+Asserts/ASTContext.o: File too big
as: /cygdrive/d/Projects/Cpp/llvm/cfgCygwin/tools/clang/lib/AST/Debug+Asserts/ASTContext.o: too many sections (33066)
/tmp/ccmtDH7w.s: Fatal error: can't close /cygdrive/d/Projects/Cpp/llvm/cfgCygwin/tools/clang/lib/AST/Debug+Asserts/ASTContext.o: File too big
/cygdrive/d/Projects/Cpp/llvm/Makefile.rules:1509: recipe for target '/cygdrive/d/Projects/Cpp/llvm/cfgCygwin/tools/clang/lib/AST/Debug+Asserts/ASTContext.o' failed
make[4]: *** [/cygdrive/d/Projects/Cpp/llvm/cfgCygwin/tools/clang/lib/AST/Debug+Asserts/ASTContext.o] Error 1
make[4]: Leaving directory '/cygdrive/d/Projects/Cpp/llvm/cfgCygwin/tools/clang/lib/AST'
/cygdrive/d/Projects/Cpp/llvm/Makefile.rules:936: recipe for target 'AST/.makeall' failed
make[3]: *** [AST/.makeall] Error 2

...

The reason for this seems to be that the COFF object format in Windows only supports a maximum of 2^15 sections (32768) for obj files.

To work around this issue, Microsoft's compiler cl has a flag called /bigobj that allocates 32 bits in order to store the number of sections (instead of 16 bits, as this standard specifies: http://msdn.microsoft.com/en-us/library/gg463119.aspx )

Also check this for /bigobj:
http://msdn.microsoft.com/en-us/library/ms173499.aspx

Anybody else having this problem with Mingw/Cygwin? If yes, how did you overcome it?

My Cygwin version is this:
$ uname -a
CYGWIN_NT-6.1 livghe-i5 1.7.29(0.272/5/3) 2014-04-07 13:46 x86_64 Cygwin

The only option I currently see is to split the mammoth cpp file (ASTContext.cpp) in 2 parts, or to get rid of the debug info (not desired).

Any idea what all those sections are - comdat inline functions?

Does the debug info in COFF do something weird and emit a section per
type or function?

Any idea what all those sections are - comdat inline functions?

No idea, as I cannot run objdump -h on a object file that I couldn't build :slight_smile:
  Anyhow, I have an old, successful Debug+Asserts build of clang, made with Mingw also, for clang revision 206070. Apparently, the ASTContext.o file there contained 16453 sections, not 33066, as there are today.

Does the debug info in COFF do something weird and emit a section per
type or function?

I will do a release build tomorrow and check the sections. (in my timezone it's night) - I will let you know!

Hi again,

After googling a little I noticed this patch: https://sourceware.org/ml/binutils/2014-03/msg00110.html

introduced in Binutils in March 2014 (GNU assembler (GNU Binutils) 2.24.51.20140326), that extends GNU assembler for Cygwin and Mingw with this flag:

-mbig-obj generate big object files

So, clang can be built with this flag by adding this to llvm/Makefile.rules:

ifeq ($(HOST_OS), $(filter $(HOST_OS), Cygwin MingW))
     CXX.Flags += -Xassembler -mbig-obj
endif

The problem is that, although ASTContext.o is built successfully with -mbig-obj, the build fails later on in clang/unittests, like this:

as: /cygdrive/d/Projects/Cpp/llvm/cfgCygDebugAsserts/tools/clang/unittests/Tooling/Debug+Asserts/RecursiveASTVisitorTest.o: section .debug_frame$_ZN5clang19RecursiveASTVisitorINS_17LambdaExprVisitorEE32WalkUpFromCXXTemporaryObjectExprEPNS_22CXXTemporaryObjectExprE: string table overflow at offset 10000096
/tmp/ccQCVRXf.s: Assembler messages:
/tmp/ccQCVRXf.s: Fatal error: can't close /cygdrive/d/Projects/Cpp/llvm/cfgCygDebugAsserts/tools/clang/unittests/Tooling/Debug+Asserts/RecursiveASTVisitorTest.o: File too big
/cygdrive/d/Projects/Cpp/llvm/Makefile.rules:1509: recipe for target '/cygdrive/d/Projects/Cpp/llvm/cfgCygDebugAsserts/tools/clang/unittests/Tooling/Debug+Asserts/RecursiveASTVisitorTest.o' failed

For this latter problem ("string table overflow"), I didn't yet find a solution, apart from building without debug info, and with optimizations enabled. See this also: http://stackoverflow.com/questions/14125007/gcc-string-table-overflow-error-during-compilation

I will switch back to Linux.

Hi Liviu

The problem is that, although ASTContext.o is built successfully with -mbig-obj, the build fails later on in clang/unittests, like this:

For this latter problem ("string table overflow"), I didn't yet find a solution, apart from building without debug info, and with optimizations enabled. See this also: http://stackoverflow.com/questions/14125007/gcc-string-table-overflow-error-during-compilation

I will switch back to Linux.

You can use the -Og option from g++.. It optimizes the space of the generated code while keeping all structures for gdb...see below the tail of the build
$ CXXFLAGS=-Og make -j3
...
make[3]: Leaving directory '/cygdrive/z/dev/llvm/x64/debug/unittests/Transforms/Utils'
make[2]: Leaving directory '/cygdrive/z/dev/llvm/x64/debug/unittests/Transforms'
llvm[2]: Compiling ThreadLocalTest.cpp for Debug+Asserts build
llvm[2]: Compiling TimeValueTest.cpp for Debug+Asserts build
llvm[2]: Compiling UnicodeTest.cpp for Debug+Asserts build
llvm[2]: Compiling YAMLIOTest.cpp for Debug+Asserts build
llvm[2]: Compiling YAMLParserTest.cpp for Debug+Asserts build
llvm[2]: Compiling formatted_raw_ostream_test.cpp for Debug+Asserts build
llvm[2]: Compiling raw_ostream_test.cpp for Debug+Asserts build
llvm[2]: Linking Debug+Asserts unit test Support
llvm[2]: ======= Finished Linking Debug+Asserts Unit test Support
make[2]: Leaving directory '/cygdrive/z/dev/llvm/x64/debug/unittests/Support'
make[1]: Leaving directory '/cygdrive/z/dev/llvm/x64/debug/unittests'
make[1]: Entering directory '/cygdrive/z/dev/llvm/x64/debug/projects'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/cygdrive/z/dev/llvm/x64/debug/projects'
make[1]: Entering directory '/cygdrive/z/dev/llvm/x64/debug/bindings'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/cygdrive/z/dev/llvm/x64/debug/bindings'
llvm[0]: ***** Completed Debug+Asserts Build
llvm[0]: ***** Note: Debug build can be 10 times slower than an
llvm[0]: ***** optimized build. Use make ENABLE_OPTIMIZED=1 to
llvm[0]: ***** make an optimized build. Alternatively you can
llvm[0]: ***** configure with --enable-optimized.