Windows/Clang build instrumented/PGO

Tobias_Hieta · January 23, 2019, 7:53pm

Hello LLVM developers,

Following some hints on this mailing list earlier this year on how to make clang faster than stock llvm.org builds I have implemented a script that builds a PGO optimized version of clang by following the guide here: http://llvm.org/docs/HowToBuildWithPGO.html

This works great on macOS and Linux - we gained almost 15% in our project with this technique. I am now looking at doing this on Windows and I am running into problems.

I bootstrap the build by downloading the 7.0.1 binaries from llvm.org, then pass clang-cl as the compiler and lld-link as the linker. I have Visual Studio 2015 CE installed as well and run all the commands under a cmd that has the right vcvars set.

To build the instrumented build I pass -DLLVM_BUILD_INSTRUMENTED=ON, I started with passing -DLLVM_BUILD_INSTRUMENTED=IR but that doesn’t build because clang-cl doesn’t understand the flags passed then. Instrumeted=ON makes it build but it fails to link ( see build log at the end of the email ).

Has anyone managed to build a PGO version of clang on windows? Is there something in my configuration that is wrong for this setup? Or is this build configuration not supported at all?

Thanks for the help,
Tobias

[285/1938] Linking CXX executable bin\llvm-tblgen.exe
FAILED: bin/llvm-tblgen.exe
cmd.exe /C “cd . && C:.conan\6785u87h\1\bin\cmake.exe -E vs_link_exe --intdir=utils\TableGen\CMakeFiles\llvm-tblgen.dir --manifests – C:\code\clang-builder\src\bootstrap\bin\lld-link.exe /nologo utils\TableGen\CMakeFiles\llvm-tblgen.dir\AsmMatcherEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\AsmWriterEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\AsmWriterInst.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\Attributes.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CallingConvEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeEmitterGen.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenDAGPatterns.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenHwModes.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenInstruction.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenMapTable.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenRegisters.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenSchedule.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenTarget.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcherEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcherGen.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcherOpt.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcher.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DFAPacketizerEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DisassemblerEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\FastISelEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\FixedLenDecoderEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\GlobalISelEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\InfoByHwMode.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\InstrInfoEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\InstrDocsEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\IntrinsicEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\OptParserEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\PredicateExpander.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\PseudoLoweringEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\RISCVCompressInstEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\RegisterBankEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\RegisterInfoEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SDNodeProperties.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SearchableTableEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SubtargetEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SubtargetFeatureInfo.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\TableGen.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\Types.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86DisassemblerTables.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86EVEX2VEXTablesEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86FoldTablesEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86ModRMFilters.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86RecognizableInstr.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\WebAssemblyDisassemblerEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CTagsEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir__\resources\windows_version_resource.rc.res /out:bin\llvm-tblgen.exe /implib:lib\llvm-tblgen.lib /pdb:bin\llvm-tblgen.pdb /version:0.0 /machine:x64 -fuse-ld=lld /STACK:10000000 -fprofile-instr-generate=‘C:\code\clang-builder\build\instrumented\profiles%4m.profraw’ /INCREMENTAL:NO /subsystem:console lib\LLVMSupport.lib lib\LLVMTableGen.lib lib\LLVMSupport.lib psapi.lib shell32.lib ole32.lib uuid.lib advapi32.lib lib\LLVMDemangle.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cmd.exe /C “cd /D C:\code\clang-builder\build\instrumented\utils\TableGen && CMAKE_OBJCOPY-NOTFOUND --only-keep-debug C:/code/clang-builder/build/instrumented/bin/llvm-tblgen.exe C:/code/clang-builder/build/instrumented/bin/llvm-tblgen.exe.debug && CMAKE_STRIP-NOTFOUND -gx C:/code/clang-builder/build/instrumented/bin/llvm-tblgen.exe -R .gnu_debuglink && CMAKE_OBJCOPY-NOTFOUND --add-gnu-debuglink=C:/code/clang-builder/build/instrumented/bin/llvm-tblgen.exe.debug C:/code/clang-builder/build/instrumented/bin/llvm-tblgen.exe””
LINK: command “C:\code\clang-builder\src\bootstrap\bin\lld-link.exe /nologo utils\TableGen\CMakeFiles\llvm-tblgen.dir\AsmMatcherEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\AsmWriterEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\AsmWriterInst.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\Attributes.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CallingConvEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeEmitterGen.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenDAGPatterns.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenHwModes.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenInstruction.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenMapTable.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenRegisters.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenSchedule.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CodeGenTarget.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcherEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcherGen.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcherOpt.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DAGISelMatcher.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DFAPacketizerEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\DisassemblerEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\FastISelEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\FixedLenDecoderEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\GlobalISelEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\InfoByHwMode.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\InstrInfoEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\InstrDocsEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\IntrinsicEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\OptParserEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\PredicateExpander.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\PseudoLoweringEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\RISCVCompressInstEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\RegisterBankEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\RegisterInfoEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SDNodeProperties.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SearchableTableEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SubtargetEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\SubtargetFeatureInfo.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\TableGen.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\Types.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86DisassemblerTables.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86EVEX2VEXTablesEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86FoldTablesEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86ModRMFilters.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\X86RecognizableInstr.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\WebAssemblyDisassemblerEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir\CTagsEmitter.cpp.obj utils\TableGen\CMakeFiles\llvm-tblgen.dir__\resources\windows_version_resource.rc.res /out:bin\llvm-tblgen.exe /implib:lib\llvm-tblgen.lib /pdb:bin\llvm-tblgen.pdb /version:0.0 /machine:x64 -fuse-ld=lld /STACK:10000000 -fprofile-instr-generate=‘C:\code\clang-builder\build\instrumented\profiles%4m.profraw’ /INCREMENTAL:NO /subsystem:console lib\LLVMSupport.lib lib\LLVMTableGen.lib lib\LLVMSupport.lib psapi.lib shell32.lib ole32.lib uuid.lib advapi32.lib lib\LLVMDemangle.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:bin\llvm-tblgen.exe.manifest” failed (exit code 1) with the following output:
lld-link.exe: warning: ignoring unknown argument: -fuse-ld=lld
lld-link.exe: warning: ignoring unknown argument: -fprofile-instr-generate=‘C:\code\clang-builder\build\instrumented\profiles%4m.profraw’
lld-link.exe: error: undefined symbol: __llvm_profile_runtime

referenced by utils\TableGen\CMakeFiles\llvm-tblgen.dir\AsmMatcherEmitter.cpp.obj:(__llvm_profile_runtime_user)

lld-link.exe: error: undefined symbol: __llvm_profile_register_function

David_Major · January 24, 2019, 9:45pm

I don't think there's any reason that it would be fundamentally
impossible, but it could very well be that you are the first person to
attempt it and have discovered some bugs standing in the way.

Two issues that I'm seeing in the log:

* lld-link.exe is being given compiler flags (-fuse-ld, -fprofile...).
There could be a mixup between C flags and linker flags somewhere.

* The missing symbols happen because clang_rt.profile-$ARCH.lib was
not included on lld-link's command line.

I'm afraid I don't know LLVM's build system well enough to say how to
address these, but hopefully it's a start.

Leonardo_Santagada · February 23, 2019, 11:55am

I would be really interested in this as I've been thinking of doing
PGO builds of lld for windows (based on a profile of linking our
binaries). Tobias, did you advance?

Tobias_Hieta · February 23, 2019, 6:13pm

Leonardo,

Unfortunately not. I ended up just doing a LTO build for windows. I think I came to the conclusion that clang-cl doesn’t automatically add the relevant compiler-rt libs when linking, since it doesn’t seem to happen manually on any other platform. But I didn’t have time to investigate any closer since I am not to familiar and we manage to get by anyway.

Happy to help out if the real root of the issue can be pinpointed and maybe someone with more knowledge can point me in the right direction.