Over the course of som years I’ve heard multiple compiler projects say LLVM is basically too slow and they’re working on replacing it with their own backend. Off the top of my head this includes: Zig, Jai, C3 and Odin.
In my own testing comparing it to the Free Pascal Compiler it appears to be 3x slower (without optimizations). I must say that’s pretty bad considering how FPC is just a hobby project.
My question is, if LLVM is this slow and projects are abandoning it, is anything being done to address this problem? Is the problem even acknowledged by LLVM or are they happy with the performance?
I’m really confused by this because LLVM seems to have the most experienced team and resources behind it so I don’t why this other smaller less experienced teams are able to succeed where LLVM fails.
The last time I heard of Zig in this context, they’ve been changing structure of their project so that LLVM becomes an external dependency. Which is different from replacing LLVM with another optimization and codegen solution. It is a step in that direction, though.
Coming from Clang side, it’s well known that most of the time is spent on optimization passes, and not on language front-end. You speak of compiler performance with optimization passes disabled, and I wonder where this metric lands on our priority list. Not too high, I guess. While compile times are a concern, runtime performance of compiled programs is still paramount.
Don’t want to speak for the project I don’t know much about, but I guess it’s an acknowledged problem that is lower on priorities list.
I’m not sure it’s correct to speak of LLVM as a team, because my experience so far has been that we are a collection of individuals. Some of us are paid for this work, but many are not. Our middle-end (optimizations) and back-end (codegen) enjoy contributions from companies that own major ISA we target (x86, ARM, Power to name some) by experts that those companies have in their staff. Smaller and newer projects don’t get this kind of attention too often, but LLVM gives them opportunity to benefit from those invaluable contributions. In this regard LLVM is a very successful project.
It’s an acknowledged problem, but isn’t a priority for most contributors. Nonetheless, there is slow but steady progress on making LLVM faster. You can see an overview of the progress here.
The summary is that over the last three years LLVM has become about 20% faster for optimized builds without debuginfo. However, unoptimized builds have become about 3% slower. (This is partly not LLVM’s fault, as this is mostly due to large increases in STL header sizes.)
Builds without optimization are probably the area where LLVM performs worst. You basically get all the overhead without any of the benefit.
This comes down to different goals and priorities. Compilers that are used in compile-time sensitive contexts (usually JIT compilers) actively trade off fast compilation times for worse generated code. Their entire architecture is optimized around producing reasonably good code in the smallest amount of time possible.
When we talk about improving LLVM compile-time performance, this is generally about improving compile-time without affecting the quality of the generated code at all, rather than about making a compile-time / performance tradeoff.
If you’re interested in reading about some recent compile-time work, the GSoC report from @0xdc03 may be of interest.
OK that makes sense. The Free Pascal compiler said LLVM (which they support too) is 10-15% faster (execution) than their own code generator but they pay for it in terms of compile times.
As for Zig he was very clear about the need for a custom backend because it makes compilation “orders of magnitude faster” which seems excessive and maybe has other factors relating to their own usage of LLVM.
In my own test I just made a program which was 5000 lines of functions that had some simple expressions in them and compared that to the same file in FPC which yields the 3x faster compilation. There conclusion is that LLVM can be used as the production generator while during development you should use theirs for the faster compile times.
LLVM should have fast pass also IMO because compile times very much matter since that’s where you spend your time in daily development. I don’t see why that isn’t a desirable goal, i mean, during development I often just want to see if it builds and gives the correct result and speed doesn’t matter at all. If your compiler is making your daily work slower for no gains that’s a problem I would say.
It’s very curious how Zig says LLVM is 90%+ of the compilation processes because I would think that once you have a syntax tree and do all the type checking that the LLVM IR would have LESS work to do. I wish I understood that.
I think LLVM is a great project but it’s disheartening that compile times are a looming concern down the road, if those other projects are any indication.
yeah I don’t get this. Daily development is where you spend your time and it often doesn’t matter if the code is not optimal just that is builds and returns the correct result.
For Zig they are making non-LLVM backends for compile times (from what I understand listening to podcasts) but LLVM will still be supported. I don’t think anyone doubts the code generation of LLVM but they want to work faster in their daily routines and worry about performance later. Pretty typical I should say.
In my daily development, I use Clang with -O3. When I wrote non-conforming C++ code, then the Clang frontend will immediately complain and the optimiser does not run. The response in that case is really fast. When I write conforming C++ code, then the optimiser runs and I get an optimised binary. In that case, I can run the application or the tests.
LLVM resp. Clang is not too bad in my daily development.
Do you know how Clang compares to other non-LLVM C++ compilers? I’ve actually used Clang too for C++ (only small files) and I didn’t notice it being slow either.
Maybe those other compilers don’t have a module system or something and they’re compiling more then they have to. Not sure where the disconnect could be happening otherwise.
Interesting but more complicated than I would expect. LLVM is single threaded but couldn’t you break IR files into the different modules and then build those with llc in parallel?
It could help in some cases to multi-thread, however most of the time when you use LLVM to build a compiler like clang, it will be driver by a build system that is already invoking many instances of the compiler in parallel using different processes. In this context using threads inside the compiler would be counter productive (you risk overloading the machines with thread stepping on each other, yielding actual slowdowns for the entire build).
I think a majority of LLVM compile time is due to optimizations. If you use -O0 you will probably get faster compile times.
If one is writing and then debugging code, the actual metric one is looking for is time between source code line change to debugger has reached that line with running.
In this scenario, it would probably be better if one could load .o files into RAM and run unit tests against them. If you have 100 .o files that make up the program, load all 100 .o files into ram and use in-memory linking, and allow for individual .o file to be recompiled and then update the in-memory .o files with only those that had changed.
LLVM Jit goes someway towards this.
I don’t think zig compiling is any faster than LLVM, its just that the zig compiler does many fewer optimizations and that is why it appears faster.
Did you watch that clip that Mr Lattner posted? He says they’re going to do per-function compiles, which makes sense to me. The problem Zig and others may be facing is modules which are too large.
The raises some questions:
Could you split up a file into many smaller sub-modules that have single functions or at least fewer functions?
Can you do the work in llc from the LLM library itself? I’m a beginner but from what I learned you need to call that as a separate process, which is going to be slow if you need to do that for many files.
Zig is using LLVM though? I saw they were in the process of replacing LLVM but I didn’t think it was finished yet. I don’t know myself, I’m not a zig user just listened to some interviews that caught my attention.
This only really helps in a case where you don’t have other parallelism ready (think of a JIT case for example). @Mogball (who is talking in the video) can elaborate though…
It’s not clear to me how this would make it faster than processing the original module?
(disabling inter-procedural optimizations and no inlining)
Of course this enables threading, but that’s about it.
No: it’s all in process. Clang for example reads the source file and then the next serialization is the ELF object file directly.
I would thinking if you had a large file, like 5000+ lines and you were held back by LLVM being single threaded. If you had one program of 100 files which were small then you can compile each file into object files on their own thread but what about the case where this is one big file?
Sure, but to be efficient you would need to share the pool of “workers” globally through the build system somehow (that is you need collaboration between the compiler and the build system).
Ultimately that still looks like something to handle in the frontend to me, which is what XLA does here by the way, the LLVM module is split into pieces and compiled with LLVM using threads.
This is 100% the opposite of my findings when profiling builds of the Linux kernel using clang. In that case, cycle counts are dominated by the clang front end; none of the top 20 most frequently sampled functions appear to be optimizations passes.
In general, at work I’ve raised the point that compile times aren’t competitive. People “care” in that they will say “this is important” but they don’t “care” enough to say “let’s hire people to work on this.”
