Memory Requirements For Compiling PassBuilder.cpp

Hi Everyone,

Recently I have realised that compiling the file PassBuilder.cpp takes a significant amount of memory. I have seen anywhere between 700 MB to over 1GB depending on which build compiler is used. This memory consumption recently caused a couple of build bots that run on a smaller machine to run out of memory and fail to build.

I have no intention of changing any of the functionality of this file but I was wondering if anyone could suggest a way to reduce its complexity (and hopefully memory consumption).
Any ideas of how I could try to go about this? Would it be possible to refactor parts into another file perhaps?

Thank you in advance,
Stefan

Probably the simple thing that might help would be to split up the file a bit - especially the functions that include PassRegistry.def as part of their implementation. I’m guessing the parse.*Pass functions are the worst parts and having each of those in a separate file is probably helpful.

Other than that - actually changing the constructs to use less memory overall, I’m not sure.

Probably the simple thing that might help would be to split up the file a bit - especially the functions that include PassRegistry.def as part of their implementation. I’m guessing the parse.*Pass functions are the worst parts and having each of those in a separate file is probably helpful.

This may not help as much as you’d like. I suspect they’re all going to instantiate a similar set of templates, so you may end up with N expensive files instead of 1.

=/

Other than that - actually changing the constructs to use less memory overall, I’m not sure.

Things that would likely help a lot here:

  1. Looking at the templates in PassManager.h that get instantiated per-pass and have nested types or other constructs and refactoring them so that they don’t.
  2. Looking at some of the especially large headers and refactoring them so that we don’t need to include them. AliasAnalysis.h and ScalarEvolution.h would be my first targets, but that’s just based on intuition, not profiling.

I suspect there is a decent amount of improvement available for both #1 and #2.

The other thing that might help would be splitting apart the parsing code from the fixed pipeline building code – those actually use somewhat different bits of pass management infrastructure and so we might get a win from separating their compilation.

Thank you both for the advice.
I’m going to try each of these ideas (including splitting the functions that include PassRegistry.def, looking at the templates in PassManager.h and looking at the large header files) and see what I get.
If any of those give me gains I’ll post a patch.

Thank you for the help!
Stefan