Build runtimes for the same archs as LLVM

cblc · June 21, 2023, 7:54pm

I built LLVM, Clang, and all runtimes (version 16.0.6) in MacOS Monterey, from scratch, and after some trial and error I believe I “almost” got it right. The “almost” is because LLVM and Clang are built as fat binaries (x86_64 and arm64), but runtimes where built as x86_64 only, which makes my build broken because I cannot use it for building universal binaries, obviously.

I found there was a commit last year for passing CMAKE_OSX_* options to the runtimes build: rG9cddfe3085c4 …but very sadly was reverted: rGececce1b5ec1

So, how am I supposed to build all the runtimes in the same archs as I’m building the rest of LLVM?

Edit: In case it matters, I’m building LLVM as fat binaries by using -DCMAKE_OSX_ARCHITECTURES="x86_64;arm64"

Thanks!

smeenai · June 21, 2023, 10:16pm

There seems to be a bunch of logic in the Darwin compiler-rt build to include both x86_64 and arm64 if your SDK supports it, e.g. llvm-project/compiler-rt/cmake/builtin-config-ix.cmake at b3569280586aec3e4d5ea61634f0a841e5827244 · llvm/llvm-project · GitHub. Are you building on an x86_64 or arm64 machine? Are both the builtins and sanitizers only built for x86_64 in your build, or are some libraries universal?

In general you can use the BUILTINS_CMAKE_ARGS and RUNTIMES_CMAKE_ARGS variables to pass additional argument to the default builtins and runtimes configures, but they might not be handling semicolon escaping properly. compiler-rt has some complex internal logic for its Darwin builds though, so I don’t think the standard CMake variables would do what you want anyway.

petrhosek · June 22, 2023, 6:21am

RUNTIMES_CMAKE_ARGS is the correct approach, see for example llvm-project/clang/cmake/caches/Fuchsia-stage2.cmake at cd482968dca4ab385a4298dfbf17e9d49ccae620 · llvm/llvm-project · GitHub

Note that it applies to all runtimes other than compiler-rt, since compiler-rt uses its own custom logic for building fat libraries which predates CMAKE_OSX_ARCHITECTURES, see llvm-project/compiler-rt/cmake/Modules/CompilerRTDarwinUtils.cmake at b3569280586aec3e4d5ea61634f0a841e5827244 · llvm/llvm-project · GitHub. Ideally we would remove that custom logic and replace it with CMAKE_OSX_ARCHITECTURES and it’s something I already considered, I just didn’t have time to look into yet.

cblc · June 22, 2023, 10:48am

Thanks a lot, @smeenai and @petrhosek !!!

@smeenai : I’m building on an Intel Mac. Regarding what I got for builtins and sanitizers, if you mean the libraries I got installed under lib/clang/16/lib/darwin, then yes everything there is universal build (and in some of them I even got archs I didn’t explicitly ask for, such as x86_64h, i386, and arm64e, and the iOS ones got the armv7* variants as well… I guess all of this belongs to the logic pointed out by @petrhosek )

@petrhosek : Do I need to build LLVM again? (Yesterday I installed my build and removed the build files). I tried to configure a new build with -DLLVM_ENABLE_PROJECTS="" -DLLVM_ENABLE_RUNTIMES=all with the hope that you can just build the runtimes, but it starts by building LLVM libs again… …which is related to a second question that I wanted to ask in another thread on how building flang if you don’t want to build it together with the rest of LLVM so that you can use different number of threads for the build because of the flang build memory needs… I have the fear that it will be like I’m experiencing now with the runtimes… it might start by building LLVM again.

cblc · June 22, 2023, 5:37pm

@petrhosek : I finally rebuilt the whole LLVM, so no worries, I have now a fully universal x86_64 + arm64 build!! Thanks a lot!! BTW, regarding the logic for automatically getting OSX settings when building compiler-rt, I found that it’s unfortunate that CMAKE_OSX_SYSROOT is ignored. If the user sets that variable, it’s for good reason, so the OSX sysroot chosen by the user should be respected throughout the whole build. OTOH, it’s nice that the build logic can find the SDK if CMAKE_OSX_SYSROOT is not defined, but not if it is.

cblc · June 25, 2023, 10:00am

Definitely, building LLVM is a huge challenge if you want make check-all to pass all tests, and the reason seems to be all this automatic logic. The build system insists in getting all the OSX settings automatically, which means the only way to succeed is to build for the defaults (ie: no universal build, no static libc++, and build only for the SDKROOT in the Xcode version you install, and with a default deployment target equal to the OSX version you are running). All the automatic guessing that the build does would be reasonable if LLVM was not for developers, but if you are a developer, you probably have your development settings chosen, and you want to use your settings, with no automatic logic at all. I think that if the LLVM system required you to define your OSX environment manually, most of the nightmares would be gone.

I build LLVM from time to time only (last time was 7.0.0), because it always takes me about 3-4 days to get a build that passes all or almost all make check-all tests, but with 16.0.6, getting a universal build with static libc++ and static libomp seems to be beyond my reach (I’ve been now trying to succeed for a week). The build always finishes, but with a lot of make check-all failures (sometimes hundreds, sometimes thousands)… I’m just a step from surrendering, and go back to the Xcode Apple build of llvm, and lose all the control I had over using a specific compiler version whose build I can control.

I almost got it, with a build which had just a dozen failures whose reason I knew and I thought I could fix, but then, out of a sudden, in the next build I got thousands of failures that I cannot debug, with this message:

********************
FAIL: llvm-libc++-shared.cfg.in :: std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp (76555 of 77731)
******************** TEST 'llvm-libc++-shared.cfg.in :: std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp' FAILED ********************
Script:
--
: 'COMPILED WITH';  /Users/programming/src/LLVM/16.0.6/build/./bin/clang++ /Users/programming/src/LLVM/16.0.6/src/llvm-project-16.0.6.src/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk --target=x86_64-apple-darwin21.6.0 -nostdinc++ -I /Users/programming/src/LLVM/16.0.6/build/include/c++/v1 -I /Users/programming/src/LLVM/16.0.6/build/include/c++/v1 -I /Users/programming/src/LLVM/16.0.6/src/llvm-project-16.0.6.src/libcxx/test/support -std=c++2b -Werror -Wall -Wctad-maybe-unsupported -Wextra -Wshadow -Wundef -Wno-unused-command-line-argument -Wno-attributes -Wno-pessimizing-move -Wno-c++11-extensions -Wno-noexcept-type -Wno-atomic-alignment -Wno-user-defined-literals -Wno-tautological-compare -Wsign-compare -Wunused-variable -Wunused-parameter -Wunreachable-code -Wno-unused-local-typedef -D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER -D_LIBCPP_ENABLE_EXPERIMENTAL -D_LIBCPP_DISABLE_AVAILABILITY -Werror=thread-safety -Wuser-defined-warnings  -lc++experimental -nostdlib++ -L /Users/programming/src/LLVM/16.0.6/build/./lib -Wl,-rpath,/Users/programming/src/LLVM/16.0.6/build/./lib -lc++ -pthread -o /Users/programming/src/LLVM/16.0.6/build/runtimes/runtimes-bins/test/std/utilities/format/format.range/format.range.fmtmap/Output/format.functions.format.pass.cpp.dir/t.tmp.exe
: 'EXECUTED AS';  "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9" /Users/programming/src/LLVM/16.0.6/src/llvm-project-16.0.6.src/libcxx/test/../utils/run.py --execdir /Users/programming/src/LLVM/16.0.6/build/runtimes/runtimes-bins/test/std/utilities/format/format.range/format.range.fmtmap/Output/format.functions.format.pass.cpp.dir --  /Users/programming/src/LLVM/16.0.6/build/runtimes/runtimes-bins/test/std/utilities/format/format.range/format.range.fmtmap/Output/format.functions.format.pass.cpp.dir/t.tmp.exe
--
Exit Code: 245

Command Output (stdout):
--
$ ":" "COMPILED WITH"
$ "/Users/programming/src/LLVM/16.0.6/build/./bin/clang++" "/Users/programming/src/LLVM/16.0.6/src/llvm-project-16.0.6.src/libcxx/test/std/utilities/format/format.range/format.range.fmtmap/format.functions.format.pass.cpp" "-isysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk" "--target=x86_64-apple-darwin21.6.0" "-nostdinc++" "-I" "/Users/programming/src/LLVM/16.0.6/build/include/c++/v1" "-I" "/Users/programming/src/LLVM/16.0.6/build/include/c++/v1" "-I" "/Users/programming/src/LLVM/16.0.6/src/llvm-project-16.0.6.src/libcxx/test/support" "-std=c++2b" "-Werror" "-Wall" "-Wctad-maybe-unsupported" "-Wextra" "-Wshadow" "-Wundef" "-Wno-unused-command-line-argument" "-Wno-attributes" "-Wno-pessimizing-move" "-Wno-c++11-extensions" "-Wno-noexcept-type" "-Wno-atomic-alignment" "-Wno-user-defined-literals" "-Wno-tautological-compare" "-Wsign-compare" "-Wunused-variable" "-Wunused-parameter" "-Wunreachable-code" "-Wno-unused-local-typedef" "-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER" "-D_LIBCPP_ENABLE_EXPERIMENTAL" "-D_LIBCPP_DISABLE_AVAILABILITY" "-Werror=thread-safety" "-Wuser-defined-warnings" "-lc++experimental" "-nostdlib++" "-L" "/Users/programming/src/LLVM/16.0.6/build/./lib" "-Wl,-rpath,/Users/programming/src/LLVM/16.0.6/build/./lib" "-lc++" "-pthread" "-o" "/Users/programming/src/LLVM/16.0.6/build/runtimes/runtimes-bins/test/std/utilities/format/format.range/format.range.fmtmap/Output/format.functions.format.pass.cpp.dir/t.tmp.exe"
$ ":" "EXECUTED AS"
$ "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9" "/Users/programming/src/LLVM/16.0.6/src/llvm-project-16.0.6.src/libcxx/test/../utils/run.py" "--execdir" "/Users/programming/src/LLVM/16.0.6/build/runtimes/runtimes-bins/test/std/utilities/format/format.range/format.range.fmtmap/Output/format.functions.format.pass.cpp.dir" "--" "/Users/programming/src/LLVM/16.0.6/build/runtimes/runtimes-bins/test/std/utilities/format/format.range/format.range.fmtmap/Output/format.functions.format.pass.cpp.dir/t.tmp.exe"
note: command had no output on stdout or stderr
error: command failed with exit status: 245

--

So, the only info it gives me is: command failed with exit status: 245. I don’t know why these thousands of errors suddenly appeared.

I think I need to go for a walk, think, and decide.