It’s an acknowledged problem, but isn’t a priority for most contributors. Nonetheless, there is slow but steady progress on making LLVM faster. You can see an overview of the progress here.
The summary is that over the last three years LLVM has become about 20% faster for optimized builds without debuginfo. However, unoptimized builds have become about 3% slower. (This is partly not LLVM’s fault, as this is mostly due to large increases in STL header sizes.)
Builds without optimization are probably the area where LLVM performs worst. You basically get all the overhead without any of the benefit.
This comes down to different goals and priorities. Compilers that are used in compile-time sensitive contexts (usually JIT compilers) actively trade off fast compilation times for worse generated code. Their entire architecture is optimized around producing reasonably good code in the smallest amount of time possible.
When we talk about improving LLVM compile-time performance, this is generally about improving compile-time without affecting the quality of the generated code at all, rather than about making a compile-time / performance tradeoff.
If you’re interested in reading about some recent compile-time work, the GSoC report from @0xdc03 may be of interest.