We discussed this RFC in the Clang area team meeting, and we think it needs a more substantial design proposal if you want to move forward with upstreaming your PR.
I think there is actually a lot of shared project interest in having an IPC system:
- Lit tests suite process creation time is driving interest in a daemonized, buildozer-like solution for running LLVM tool invocations
- clangd uses grpc for IPC/RPC. gRPC is a portable, off-the-shelf solution. The PR uses Windows IPC APIs directly, and we aim to be a portable project. Have you considered gRPC, and weighed the tradeoffs? clangd is sort of a PCH-generation daemon, presumably similar in structure to what you’ve proposed.
- Distributed ThinLTO is sort of doing the opposite, it is taking jobs that normally would run as threads and farming them out as separate process invocations that can be run elsewhere. Is there some potential for sharing design work here?
My main concern with any IPC system is that we want to keep the core of the compiler functional (as in “functional programming”) and deterministic. This is one of the major flaws of the MSVC type server PDB system (i.e. the /Zi flag as opposed to /Z7), which uses IPC to assign type IDs online, making its compiler output inherently non-deterministic.
I could easily see us merging some kind of scheduler / executor in LLVM that schedules compilation work if it is very careful about state sharing, but if the proposal is to reuse cached AST nodes in memory, I don’t think the project should take on that maintenance burden, speaking as a former intern who proposed a very similar project 16 years ago.