Im asking if you can give detailed information about building and testing compiler-rt. I dont want to build as runtime since I have to build the whole LLVM and clang and then its build can fail at the and spoil the whole LLVM and clang build. Is it possible to build and test as standalone and how ? Ok, Ive build LLVM and clang the usual way no sanitizers and then how I can configure the standalone build and standalone tests. Is it possible to build it without sanitizers and how. They use a lot of RAM. Shail I build the whole clang with these sanitizers.How about the fuzzers the same questions. Can it be linked to libstdc++ only or to libc++, libwuniwnd suite etc Please explain with detail explanation and configuration.
I think people would appreciate at least spreading the questions out to be more readable 
As far as I know, you can build compiler-rt standalone by putting it in LLVM_ENABLE_PROJECTS. That will use your host compiler to build it, instead of building another clang first (assuming compiler-rt is the only thing in the projects list that is).
Have you tried that or similar, and if so what were the results? What resources have you already looked at and what steps if any have you taken to try and work around the problems you have seen?
I do not say that to imply you haven’t “tried hard enough”, just to establish where you are at. Building the runtimes can be tricky and often particular to specific use cases. So it is best to be specific when discussing the subject.
Please also say what your goal is beyond building compiler-rt. Are you simply looking to develop a clang change and wonder if you should do so with sanitizers enabled? Are you trying to build a whole toolchain for your platform, etc.
Thank you DavidSpickett. I have tried as a project in the LLVM build tree but the build fails at fuzzers. As a standalone it builds but it requires the tests to be run within the LLVM build tree. LLVM website states that fuzzers are built with coverage and sanitizers. Which one is needed for the build we should build the whole clang with sanitizers and coverage or only the fuzzers? But fuzzers build require sanitizers first. It is not clear.simce
It also requires to turn off runtimes build. However, If I turn off runtimes, the compiler-rt, libc++ etc stop building. Compiler-rt builds its own libc++ and it cannot be tested how good it is so that the compiler-rt build will not fail either. A lot of questions come up and it is not explained.
https://compiler-rt.llvm.org/
Configuring LLVM to Build Fuzzers
https://www.llvm.org/docs/FuzzingLLVM.html#building-fuzzers
Developing libFuzzer
What cmake command did you use to configure this and with what error did it fail?
(I realise I might have tried to help with this in another question, but repeating it won’t hurt)
Ok I see what you mean now. If you just built compiler-rt, you won’t have the various utilities the tests call out to, makes sense.
Glancing at https://github.com/llvm/llvm-project/blob/4ff8d1e315bdd26daae8a6928aa46e347fe0d6eb/compiler-rt/CMakeLists.txt#L84 I think it is supposed to find the tools using llvm-config but of course this may have broken over time.
I will try out a build myself and see what I find.
Yeah, I am also confused about that I don’t blame you there.
Looking at https://github.com/llvm/llvm-project/blob/4ff8d1e315bdd26daae8a6928aa46e347fe0d6eb/llvm/cmake/modules/HandleLLVMOptions.cmake#L841 my impression is that you would need to build the sanitizers first. Then do a new build of llvm with this cmake option, and have your compiler pick up the sanitizers you just built.
The comment about DLLVM_BUILD_RUNTIME appears to mean that if you wanted to also build sanitizers in the build you were going to fuzz, you would need to use that option to prevent sanitizing the sanitizers (you’re right, this is a bit mind bending).
In your case I don’t think that’s needed? Not 100% sure.
I think what you want to do is:
- Build clang and compiler-rt, with the default build steps. This will include clang, sanitizers and libfuzzer.
- Build a second copy of llvm, using that clang you just built, and the
LLVM_USE_SANITIZERoption when you do that. This will link the sanitizer libraries built in step 1, into the resulting binaries. - Do whatever one needs to do to get fuzzing going, this part is out of my expertise.
I should also note that building with a sanitizer doesn’t look to be required, it simply makes it more likely that you’ll hit a bug.
I agree that the docs there are not that clear if you do not have a lot of context.
I will also give these steps a go myself and see what happens.
Hello. Well Im using latest Clang 15 as a build compiler and llvm projects clang, lld, compiler-rt, libcxx, libcxxabi. No sanitizers and nothing unusual about the build configure. The build error at compiler-rt is from linker ld:
libclang_rt.fuzzer.a(FuzzerDataFlowTrace.cpp.o): unrecognized relocation (0x2a) in section `.text._ZN6fuzzer13BlockCoverage14AppendCoverageERKNSt8__Fuzzer12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE’
/usr/bin/ld: final link failed: Bad value
I force the build with the lld linker and the same error again. So it is not a linker error.
I think it is related to the build of this library libFuzzer.
https://llvm.org/docs/LibFuzzer.html?highlight=libfuzzer
-DCOMPILER_RT_BUILD_LIBFUZZER=ON
However, I failed to compile it and test it. It is completely missing. Unofficial authors in the wild point out that one has to compile it manually by compiling every source in compiler-rt/lib/fuzzers and then link it as a library with the archiver ar. I doubt about it reliable since it cannot be tested.
I havent tried its build with the sanitizers and coverage. Logic looks strange to use testing tools like sanitizers to build other testing tools like fuzzers. And it may be impossible if RAM fails.
So from your opinion and my investigation so far it turns out that the correct way is building the clang several times.
The first one is building with sanitizers only. Thus, -DCOMPILER_RT_BUILD_LIBFUZZER=OFF.
Next build is building with sanitizers from the previous build and here -DCOMPILER_RT_BUILD_LIBFUZZER=ON. Which I think sadly will fail if RAM fails. By the way I experimented to build only libc++ with memory sanitizers. Sure most tests fail since Im using unsanitized clang but the RAM load is quite high and I dont think I will succeed for building the whole LLVM sanitized. So it may turn out finally that this feature is unsupported and disabled to me.
Hello, DavidSpickett. There are a couple of issues in the source that I’ve discovered.
The fuzzer build fails due to a linker issue. Clang is built without default its linker lld and this triggers most of the issues. The Gcc linker makes errors even in building LLVM, not only compiler-rt. Plus, one of the major issues is that fuzzers build a separate and a special release of the C++ library. And there are cmake files that dont use the lld linker. Cmake recognizes it but the build commands dont use it. Ive manually changed them and everything is built right. However, I don’t know if they can be patched permanently since users will use their own compiler and linker on Windows, Linux etc. So I think the best optoin is to build clang with its default linker and they are integrated very well to avoid such errors. It is not only the fuzzers build but also the tests of the thread sanitizers fail too. And the same solutions apply to them too. Still, I am unable to resolve the libfuzzer library issue and it doesn’t build at all without the sanitizers as we’ve discussed it above.
Well I tried the standalone build as documented. First I built the latest clang as normal then used that as the compiler.
$ cmake ../compiler-rt/ -DLLVM_CMAKE_DIR=/home/david.spickett/build-llvm-aarch64/ -DCMAKE_C_COMPILER=/home/david.spickett/build
-llvm-aarch64/bin/clang -DCMAKE_CXX_COMPILER=/home/david.spickett/build-llvm-aarch64/bin/clang++ -G Ninja
That builds and results in some fuzzer libraries being present:
$ ls lib/linux/ | grep fuzzer
libclang_rt.fuzzer-aarch64.a
libclang_rt.fuzzer_interceptors-aarch64.a
libclang_rt.fuzzer_no_main-aarch64.a
But you are right, you cannot ninja check anything from within the standalone build. I thought at first this was a configuration issue but looking at the cmake it looks intentional. I could be wrong.
I also tried LLVM_CONFIG_PATH as that was what was documented, but got a warning that it was deprecated and to use LLVM_CMAKE_DIR instead. The result was the same either way.
So that’s the state of that, take it for what you will, I will seek to clarify that and correct the docs (for which I opened compiler-rt standalone build documentation is misleading and somewhat outdated · Issue #59548 · llvm/llvm-project · GitHub).
I think at this point you are better off coming up with a new, more specific question and asking that in a new post.
Something that starts with a statement of what you are looking to achieve. E.g “I want to develop the fuzzer” “I want to fuzz llvm” “I want to fuzz ”. Something that isn’t more build configuration issues as they are very difficult to figure out without a clear goal in mind.
And please make use of paragraphs, it really, really helps parse what you are asking.
It will be easier for someone to answer that, who isn’t me, because I can figure out cmake but not fuzzers specifically. Sounds like you are far ahead of me on that front and that is probably the real understanding gap here.
Thank you. I’ve only built what is stated as it can be built at the LLVM website. And there is this libfuzzer library which is missing in your and my builds. So thats why I ask. The tests appear in a standalone build after I add
-DCOMPILER_RT_INCLUDE_TESTS=ON. And it is not stated that way at the LLVM website.
https://compiler-rt.llvm.org/
Tests for sanitizer runtimes are ported to llvm-lit and are run by make check-all command in LLVM/Clang/compiler-rt build tree.
So it is misleading and confusing.