Compared to manual IAT patching and TLS initialization and tools needing a static CRT, wouldn’t it be comparatively easier to have the tools themselves actually be DLLs and have stub executables?
Yes that’s completly possible instead of the proposed mechanism, if we add a LLVM build mode where each tool (also) becomes a loadable DLL. However the installation size have to be considered.
As an alternative that already works with this proposal, if LLVM releases are built with -DLLVM_TOOL_LLVM_DRIVER_BUILD=ONthen a multicall binary is created, which already includes all the tools (clang, lld). This mechanism for loading external EXE-as-DLL is only a fallback when the build is not using -DLLVM_TOOL_LLVM_DRIVER_BUILD. This was done primarly to support external toolchains which I don’t have control over (game consoles), but where I’d still like to use this in-process mode. If the toolchain providers adopt the -DLLVM_TOOL_LLVM_DRIVER_BUILD=ONmode, then we don’t need this loading EXE-as-DLL.
Also I don’t have experience with this, but wouldn’t scanning for LdrpHandleTlsData be prone to detection/blocking by AVs?
That is very possible, good point. Our security stack does not raise this as a problem (we have multiple security services running on our machines) however this probably depends on the product.
How would static variables be handled if it has the assumption of living only for the current TU’s/ tool call’s lifetime?
This is part of the Meta-RFC, the next step after this current RFC is to remove global state.
What is the expected behavior for a single file rebuild? Let’s say we have 1000 TUs in a library. On a full build I understand the expected behavior for the build tool is passing all 1000 to clang. What happens when some fraction needs a rebuild? In the normal case our build tool(ex. make,ninja) detects that and calls the compiler based on the compile commands it has. Is the build tool then expected to call clang with all the files that need a rebuild? How is it meant to pass the flags, what if some files have other compile flags? How does this coincide with the compilation database option? The compilation database won’t change in this case as it was generated by cmake eariler on and still contains all the files (as it should, changing this would break a lot of tooling).
The /compilation-database flag in Clang right now is only provided as an illustration, and for testing this in-process behavior. This proposed in-process RFC alone is not useful for improving build times. We need the next step in the Meta-RFC (Multithreaded Execution) which includes the removal of global state. If that it is done, ninja would call clang-cl /compilation-database changed_files.json, instead of invoking multiple clang-cl commands for each file. The database would be created in a temporary file, in the same way .rsp files are created right now by ninja.