I remember doing so at one point when I was still using Windows, and found
my system much more responsive overall. But that was ~10 years ago and I
guess it’s not a common setup.
10 years ago does not seem very relevant, both Windows and VS are very different now. Also “… found my system much more responsive overall …” very much sounds like placebo/snake-oil.
If msbuild is 32-bit, why would that cause the (64-bit?) cl.exe process to
run out of memory?
I doubt msbuild is allocating much memory at all, but is the 64-bit version of cl.exe being called from msbuild? You need to up the build output so you can see what compiler is being called. Sounds like ninja is calling the 64-bit version, while msbuild is (could be, I’m speculating) using the 32-bit version to cross-build [and runs out of addressable space]. The VS defaults are all 32-bit.
The problem is that by default VS2017 builds Clang in parallel to how many cores you have. When I build LLVM/Clang, I’ve noticed the default setting will almost instantly consume 100% of my RAM and I have 16GB of RAM and 32cores.
To turn off parallel building, mark the appropriate CMAKE variables:
LLVM_PARALLEL_COMPILE_JOBS:STRING
Define the maximum number of concurrent compilation jobs.
LLVM_PARALLEL_LINK_JOBS:STRING
Define the maximum number of concurrent link jobs.
Also in Visual Studio, in Tools > Projects and Solutions > Build and Run, set
'maximum number of parallel project builds to 1.
16GB is about the minimal amount of RAM you want for a quad core developer system these days – I build mine with 32G in order to have room for a decent amount of disk caching. 16GB a completely crazy amount of RAM for 32 cores. I wouldn’t go with less than 128GB. This time last year I built a system with an i9 with 18 cores and 64GB and it was just barely enough – 32GB would have been tragic.
I note that AWS “general purpose” instances (e.g. M4/M5) have 4 GB per vCPU (hyperthread). “Compute Optimised” have 2 GB per vCPU. I find the M5 instances much better balanced for software development than the C5 instances, at about a 12% price premium.
The problem is that by default VS2017 builds Clang in parallel to how many cores you have
That would explain my problem in building clang then as I have an eight core Ryzen 7 1700X with only 8GB of RAM. I will try to build with parallel building off…
But did ninja ignore these build options? Because I was able to build clang using it without any problem.
I will look into it and post some results tomorrow.
Yeah that’s why I only build with 1 core. I know I need more RAM, but I refuse to buy more RAM until they lower prices closer to what they were: ~$80/16GB stick.
Interestingly, I’ve built Chromium many times with 32 core Linux systems with only 16GB of RAM, and there was a couple links that approached 16GB, but never
went over. Wouldn’t doubt that this is a VS issue.
Ninja is not MSBuild, it has its own algorithm for scheduling parallel jobs (you can control this with the -j option, but by default it picks something sane). You can see what happens if you pass -j 8.
I still don’t see how your situation would result in an out of memory error. How much physical memory your system has is totally irrelevant, you still cannot “run out of memory” from the process’s point of view unless you also run out of disk space for swap, which is extremely unlikely unless you have configured a very small swap.
I am sorry for taking so long to update my status, tough week
I was able to build clang with ninja -j8 on my computer while normally using it (firefox, unity 3D, visual studio …). So, I guess MSBuild uses way too much ran when building in a multithreaded environment? Should I try to debug MSBuild or file a bug report?
Anyway, I watched as the other thread on whether or not to abandon the MSBuild workflow…
What if cmake generated by default a solution that builds using only a single thread? This way people would not complain when trying to use this workflow and mantainers / more experienced users would change this setting when they find it appropriate.
Thank you very much for the support so far! Very nice community, I hope to contribute soon
MSVC has two knobs for parallel compilation: the compiler switch /MP which can be set per project or per source file, and “Number of parallel builds” which is a user setting and can’t be overridden for individual solutions/projects.
So I suppose the cmake system could generate projects without /MP, but the other setting is out of its control.
And I’m not sure it makes a good default to cripple the buildsystem like this. For every user it helps, it will kill the experience for somebody else.
The normal time for a clean build on this machine is 8.5 minutes. Even on
my little i7 NUC it's only about 30 minutes. Who would want to wait over
two hours?
