Ninja (make replacement)

Project Infrastructure LLVM Dev List Archives
1

Hi,

Ninja is being mentioned more and more frequently on this list, but I don’t think anybody has introduced it formally.

Ninja is, briefly put, a super-fast replacement for GNU Make, nmake, and all the other make tools out there.

For those of you who are a bit reluctant to waste time on Ninja or don’t know what it is, I can strongly urge you to download it from http://sourceforge.net/projects/cmakescript/files/ together with a Ninja-enabled version of CMake from the same URL. It takes you perhaps 5 minutes, and you are going to earn them back hundreds and thousands of times over the coming years.

My Clang build used to take 59:29 minutes using MinGW Makefiles. With Ninja, it takes 18:43 minutes! That’s a speed up of factor three.

Cheers,
Mikael

2

Hi Mikael,

My Clang build used to take 59:29 minutes using MinGW Makefiles. With
Ninja, it takes 18:43 minutes! That's a speed up of factor three.

sorry for asking the obvious question: Are you comparing against a
parallel "make -j<n>"? :slight_smile:

Best regards,
Christoph

3

Hi Christoph,

It is a good question even though it is obvious :slight_smile: No, for the last few days, I have not been able to build Clang with make -j2 or -j4 - the build always fails on a ranlib operation.

Also, on Windows, GNU Make seems to be somewhat broken with respect to parallel builds: It doesn’t change a thing whether you specify -j2 or -j4 on a quad-core machine (with hyper-threading disabled in BIOS): The load never passes 50 percent. I have searched the net and found various magic recipes for how to fix the problem, but none of them worked.

Perhaps somebody with a make-friendly platform (Linux, Solaris, etc.) will post their numbers with “make -j6” versus “ninja” (Ninja defaults to six threads).

All I can say is that for Windows users, Ninja is the best news in years. And I have seen Mac owners report a speed-up of 2.5 times. So I don’t think it is only on Windows it is a win-win situation.

Supposedly, Ninja is built for speed. I am myself VERY surprised that there were so large gains to be had just from switching out your make tool with ninja. And by the way, you invoke ninja just like make: “ninja”, “ninja check”, “ninja install”, and so on.

Best regards,
Mikael

2012/6/18 Christoph Erhardt <christoph@sicherha.de>

4

Christoph Erhardt <christoph@sicherha.de> writes:

Hi Mikael,

My Clang build used to take 59:29 minutes using MinGW Makefiles. With
Ninja, it takes 18:43 minutes! That's a speed up of factor three.

sorry for asking the obvious question: Are you comparing against a
parallel "make -j<n>"? :slight_smile:

I'm pretty sure he isn't. Building Clang+LLVM with MinGW, maybe ninja is
faster than gmake by a few dozen seconds on Windows. The real advantage
is over the build tools traditionally used with the Visual Studio
toolchain, because all of have limited support for parallelization.

The other aspect where ninja shines over all other build tools is at
being fast parsing the makefiles and determining what needs to be
rebuilt. This is something people appreciate while rebuilding big
projects after small changes.

BTW Mikael, have you checked Building LLVM with CMake — LLVM 18.0.0git documentation ?

5

I am building with MinGW and my experience is that “make -jN” either doesn’t work (because of problems with different processes updating the same library using ranlib or something like that). And then there’s the issue I mentioned in the previous doc.

I think Ninja is a gift from some heavenly dimension. Others may disagree. But I think any Windows developer should take a look at it, no matter if they are using GNU Make or Visual Studio.

By the way, I forgot to use my brain when I sent the first link: It is a Windows-only link. For non-Windows users and people who want to see the source before they use it, use this link: http://martine.github.com/ninja/

Yes, I am quite familiar with the CMake documentation, but why are you asking? I use CMake all the time for my MinGW32 and MinGW64 builds. Now I use “CMake -G Ninja” :slight_smile:

2012/6/18 Óscar Fuentes <ofv@wanadoo.es>

6

Mikael Lyngvig <mikael@lyngvig.org> writes:

[snip]

Yes, I am quite familiar with the CMake documentation, but why are you
asking?

That's not the cmake documentation, that's the LLVM cmake documentation:
a short document that tries to explain everything you need to know about
cmake to build LLVM. I had the impression that you were duplicating a
large chunk of the info contained on that page. OTOH it's my personal
opinion that you may be explaining too much: it is not necessary to
explain, nor even mention, the different available build tools on
Windows. As LLVM and Clang are developer tools, one can suppose that the
user knows how to run a build. IMHO you only need to explain the bits
that are specific to LLVM. As cmake might not be popular enough yet, a
bit of an introduction is needed, but no more.

I haven't thoroughly read your proposed documentation, sorry if my
opinions are off-base.

[snip]

7

Yes, sorry, the LLVM CMake documentation :slight_smile:

I don’t think I am duplicating a large chunk; I am continuously trying to shave off extraneous and redundant stuff off my Windows-specific document. I just managed to shave off about six lines or so because I don’t like the huge introduction and explanation of what MinGW is. Now that the document covers Ninja, there’s even more reason for Windows users to read it (and the document even got a bit simpler).

You might ask: Why not just make a generic document and patch it up to support Windows as well as Unices? Well, you guys keep making all sorts of, IMHO, undesirable assumptions regarding your potential userbase. That somebody in CS201 wants to try out LLVM or perhaps more likely Clang does not mean that they have a Ph. D. in converting Unix commands into Windows commands. I simply don’t understand your often repeated assumption that “any user of LLVM knows this and that (and probably has a beard according to many articles on how gurus look) and therefore we don’t need to cater to the less experienced”. That attitude strikes ME as arrogant. Because it makes life very complicated for those newbies that are just out to look and learn. And there are many pitfalls on the Windows path because Clang++ does not support the Microsoft C++ ABI and the MinGW64 C++ ABI: This was nowhere documented before, now it is documented together with an easy-to-use guide for how to build and use Clang on Windows.

One guy recently called LLVM an advanced, multi-backend, optimizing assembler language. There are even today newbies who throw themselves at coding in assembly, so why not code in LLVM IR? The potential for LLVM is, in my mind, much bigger than just 17 research dudes and five corporations.

Another angle is that most guys interested in Computer Science go through a period of their life where they want to know everything about how compilers (usually not linkers) work. So they want to study some source code and see how things are done. I’d like to wager that Clang is probably one of the best authored compilers out there. So why not make it easy for people to get going on their studies of Clang? I don’t mean to offend anyone, but I don’t personally think that the GNU compilers are overly much worth studying. I myself contemplated adding a new frontend to the GNU compiler suite, for my own language, but ended up dropping the idea because I found it to be too difficult and too complex for my taste (I am allergic to structured programming).

A third angle is that 80 percent of Windows users are breathing Windows and have never known anything but Windows. They don’t know what Linux is. Literally. They much less know what Unix is. The word Linux they may have heard here and there, but they don’t know what it means. And yet, some of these Linux/Unix ignorant people are great thinkers with great brains that can provide great assistance to the project if only they get a little helping hand in the beginning.

You cannot reasonably assume that any and all potential users of LLVM and especially Clang are willing to play around for two or three days with the various combinations of tools and settings to get something to work. Windows users are perhaps a bit spoiled, but on Windows, people normally get a binary release that works straight out of the box without additional downloads. Compare that to two or three days of experimentation and you see why I am so intent on making LLVM and especially Clang EASY to get to work (I have not given up making a binary distribution of Clang).

I know I am probably being perceived as a arrogant, lunatic n00b, but all I want is to make Clang READILY available to as many Windows users as possible. I have at least two reasons for this:

  1. I want the Windows support in LLVM (and to some extent Clang) to be 100 percent mature as I need it myself for a project I am working on. Since I don’t code on LLVM/Clang myself, I have to find others to do it.
  2. I think the Windows platform has too few options with respect to compiling your C and C++ programs. MSVS is expensive. MSVS Express is a joke (IMHO). OpenWatcom is, I suspect, slowly dying (this used to be my favorite compiler throughout the past 19 years!). And GCC has a somewhat unsettling track record of broken features.

I hope this makes my actions and intentions more understandable :slight_smile: Also, please remember that MinGW32 has been downloaded more than a million times. Why aren’t millions downloading Clang for Windows?

But all that stuff being said, please don’t hesitate to criticize my documents. I know they are not perfect. They will never be perfect. But the more feedback I get, the closer to perfect they’ll hopefully become.

Try checking out this latest version of the “Building using MinGW on Windows” document. It is reStructured Text and therefore requires Python+Sphinx to be installed. I’ve attached it. It is not a formal release, it is intended only for those worrying that I am duplicating too much or writing too much text.

Cheers,
Mikael
– Disclaimer: I am not arrogant in real life, so if you perceive me as being arrogant, there’s a communications problem somewhere.

2012/6/18 Óscar Fuentes <ofv@wanadoo.es>

Building with MinGW on Windows.rst (18.2 KB)

8

IMO, Mikael’s efforts are on-target and very useful, and the level of detail is close to “right”. Windows builds for almost any tool are, despite many of our most fervent desires, still very much different from linux builds. There are umpteen different variants on exactly which shims to select to make them work in the windows universe, and guidance through that maze is quite useful even to people familiar with all the options. In particular, the “it currently doesn’t work with such-and-such” potentially saves false starts down those paths.

(Thanks Mikael!)

9

Thank you so much for your hard work!

LLVM/Clang is in need of motivated Windows developers willing to put in the time to make the LLVM/Clang experience better on Windows :slight_smile:

Quick note on the reST:

instead of using a construct like:

Notice: If you only want to build 32-bit programs, you do not need to
install MinGW64.

Prefer to use the reStructuredText “admonitions” <http://docutils.sourceforge.net/docs/ref/rst/directives.html#note>, so the above would look like:

… note:: If you only want to build 32-bit programs, you do not need to
install MinGW64.

reST also has a number of other semantically rich admonitions (“hint”, “warning”, etc.), which are listed on the page I linked above.

–Sean Silva

10

Jim Rowan <jmr@codeaurora.org> writes:

[snip]

Windows builds for almost any tool are,
despite many of our most fervent desires, still very much different
from linux builds.

FYI: building LLVM+Clang on Windows has *exactly* the same requirements
and commands than on Unix (letting aside problems with specific MinGW
versions.)

[snip]

11

Done - and thank you again for enligthening me with respect to Sphinx :slight_smile:

2012/6/18 Sean Silva <silvas@purdue.edu>

12

See http://llvm.org/bugs/show_bug.cgi?id=13147

Sean

13

I think there is a problem in Make as well. No matter what library or when during the build, I never see it go above 50 percent CPU load even if I specify -j4 on a quad-core system. With Ninja, on the other hand, the load consistently stays at 100 percent.

Check out this ancient article: http://dannythorpe.com/2008/03/06/parallel-make-in-win32/ . And the magic, setting SHELL to cmd.exe, does not seem to work for me.

2012/6/19 Sean Hunt <scshunt@csclub.uwaterloo.ca>