It appears the behavior of "make check" has changed recently (configure style build). I believe it used to just run the tests. Now, it seems to recompile the project just like a "make" would and then run the tests. Is it just me or are others seeing this too? Is this a intended behavior change?
Best,
Arnold
Eric changed it in svn 179116. He didn't provide any explanation in the commit message. Eric, why did you do that?
Because almost no other piece of software out there doesn't have
'check' depend on rebuilding the target it's testing and it's bitten
people whereas other than the lameness of our build system needing to
recurse all the directories to figure out that all has already been
rebuilt there's no penalty to requiring the dependency. 
-eric
Given this explanation my vote goes for reverting it.
-Krzysztof
Give a compelling argument and I might
-eric
The cost is the extra time spent verifying that the binaries are fresh. In my experience it has never happened that I ran "make check" on a wrong set of binaries. On the other hand, it does happen that I run "make check" repeatedly, when working on a testcase (or a set of testcases), or when I fixed something that previously caused some testcases to be XFAILed.
-Krzysztof
I don't know if this is compelling enough, since it hinges on my own working habits. It would be interesting to see the more opinions about it.
-Krzysztof
The extra cost is pretty trivial and you can run individual test cases
by using lit explicitly. We've had annoying bot failures because
someone ran "make check" but it didn't rebuild the targets that check
depends upon. On my laptop it's a total of 4 seconds to get to
check... you probably would save more time by running lit explicitly
on a testcase you're changing
-eric
If development speed / turnaround time is important for you, 'make' is the
wrong solution in the first plane. Use a cmake+ninja build, which is way
faster, especially for null builds (where all the build system does is make
sure there's nothing new to build, or very few things to build).
Here's a ninja build run on a fully built debug LLVM+Clang checkout:
$ time ninja
ninja: no work to do.
real 0m0.088s
user 0m0.070s
sys 0m0.010s
You can't beat that with 'make' no matter what you do 
Eli
Because almost no other piece of software out there doesn't have
'check' depend on rebuilding the target it's testing and it's bitten
people whereas other than the lameness of our build system needing to
recurse all the directories to figure out that all has already been
rebuilt there's no penalty to requiring the dependency.
As a single datapoint, I've been relying on the fact that make check doesn't rebuild. Sometimes I work on underpowered machines and will continue to reformat my code after the .o file has been built. Relinking can take more than five minutes. I do not like this change, but my argument is also rather silly, per xkcd: Workflow .
Nick
cmake+ninja is a non-option for many developers. Fixing that is a separate problem. We can’t just ignore problems in our configure+make build system in the meantime.
Understood, and this is why I didn't express an opinion on the original
issue. I just thought it may help a fellow developer to point this out if
he really cares about the speed of null builds. Personally I've saved a lot
of time since switching to ninja a few months ago.
Eli
For me it really boils down to my personal working habits. Something that has worked for me just fine, now is about to become more time-consuming, that is pretty much it. I wouldn't want that preference of mine to be the sole cause for any further changes, but I wanted to share my thoughts about it.
-Krzysztof
FWIW, the cmake build has the "check" target depend on the binaries under test.
By the way, i'm curious what this means. It can be interpreted in a
number of ways, but it would be good to know what problems you are
specifically saying making it a non-option for many developers
cmake+ninja is a non-option for many developers.
By the way, i’m curious what this means. It can be interpreted in a
number of ways, but it would be good to know what problems you are
specifically saying making it a non-option for many developers
Hi Danny,
Sorry for being unclear. My fault for posting when I’m in a hurry.
I don’t think there’s any unsolvable problems. Well, not technical ones anyway. It’s a question of supporting a broad enough cross section of environments and workflows that devs not only can move to cmake+ninja, but actively want to do so.
For me personally, it’s more a question of supported workflows being inconvenient than it being a strict non-starter. That is, it’s a question of which system has fewer inconveniences for me in my daily work. Right now, that’s configure+make. I’m very much looking forward to the day when the balance of that equation changes.
Note, I’m explicitly excluding the whole “remove autoconf+make” thing from this. That’s another topic with a whole extra set of issues. Likewise, I’m not going to make any sorts of claims about exhaustiveness here, but rather just relate mostly what I’ve encountered.
AFAIK ninja still only has very preliminary support for Windows, so anyone working there can’t use it. Perhaps that’s changed or I’m misinformed? Last I tried it, OSX support was better, but still very much “try it and see if it works for you.” I personally found ninja’s OSX support to be fine, but I didn’t really beat on it to put it through its paces, either. Relatedly, some work environments have varying ability to install and use “non-standard” tools. I’ve been in workplaces before that I’d have had to fight tooth and nail to get something like cmake or ninja available to me on my workstation. Even now when I don’t have that policy sort of problem, it’s still a pain when every time I install a new system (which I do disturbingly often), I need to install additional tools before I can start using llvm. Not a big problem on its own of course, but an inconvenience.
cmake+ninja doesn’t support building and running the llvm test suite. That’s pure configure+make still. I rely on configure automatically setting up the test suite to run on whatever compiler I just built.
For example, I build Debug+Asserts, Release+Asserts and Release configurations all in the same build directory and rely on the configure+make machinery to create separate build results subdirectories named for those. That is, it’s pretty common for me to do something that boils down to (with lots of steps in between of course), “/configure && make && make check && make ENABLE_OPTIMIZED=1 && make check ENABLE_OPTIMIZED=1 && make ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1 && make check ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1”. Depending on what I’m doing, I bounce back and forth between them. With cmake, I have to have completely separate configured build directories. I already have 10+ llvm build directories lying around that I regularly use. I’d prefer to keep that number from growing too horribly. When I last tried using cmake+ninja, this is the one that I kept running into and getting annoyed with.
Do the cmake scripts support cross-compiling? Last I tried, I had trouble with that. Many moons ago, I had to beat the plain Makefiles into some sort of submission for that. I not infrequently build llvm to run on ARM, for example.
Somewhat relatedly, I like to keep my development builds as close as reasonably possible to production builds. There are few things that make my stomach churn more than bugs that only reproduce on the production builders. That’s somewhat off-topic, though, as that’s getting into “what would it take to remove configure+make support” territory.
In short, I’m a big fan of ninja and really want to use it. CMake, I’m less thrilled about, but I’ll put up with it if it gets me ninja. Right now though, it’s a close but not quite a fully baked replacement for my uses.
Anyways, none of this should be construed to be a slight to the work people have put into cmake+ninja support for llvm/clang. Quite the opposite. I’m excited enough about what’s been done that I’m frustrated it doesn’t fit my use-cases well enough for me to switch.
My point with that comment, which Eli correctly and effectively addressed in his reply, is that we can’t assume we can define away problems with the configure+make system by just telling people it’s deprecated and they should use cmake+ninja instead. We’re not there yet.
Thanks,
Jim
Ninja works fine for me on Windows. I've been using it to compile LLVM and co. for a few months. It's also really easy to compile Ninja itself on Windows.
There's just one issue: When a *.td file is changed, I have to invoke ninja twice. But I haven't investigated yet wether that's an issue with Windows, ninja, cmake or LLVM's build scripts.
-Nico
Nice! Sounds like things have indeed moved in the right direction since I last looked at that. Most excellent.
-Jim
From: "Jim Grosbach" <grosbach@apple.com>
To: "Daniel Berlin" <dberlin@dberlin.org>
Cc: llvmdev@cs.uiuc.edu
Sent: Wednesday, April 17, 2013 2:46:49 PM
Subject: Re: [LLVMdev] make check rebuilds the project?cmake+ninja is a non-option for many developers.
By the way, i'm curious what this means. It can be interpreted in a
number of ways, but it would be good to know what problems you are
specifically saying making it a non-option for many developersHi Danny,
Sorry for being unclear. My fault for posting when I’m in a hurry.
I don’t think there’s any unsolvable problems. Well, not technical
ones anyway. It’s a question of supporting a broad enough cross
section of environments and workflows that devs not only can move to
cmake+ninja, but actively want to do so.For me personally, it’s more a question of supported workflows being
inconvenient than it being a strict non-starter. That is, it’s a
question of which system has fewer inconveniences for me in my daily
work. Right now, that’s configure+make. I’m very much looking
forward to the day when the balance of that equation changes.Note, I’m explicitly excluding the whole “remove autoconf+make” thing
from this. That’s another topic with a whole extra set of issues.
Likewise, I’m not going to make any sorts of claims about
exhaustiveness here, but rather just relate mostly what I’ve
encountered.AFAIK ninja still only has very preliminary support for Windows, so
anyone working there can’t use it. Perhaps that’s changed or I’m
misinformed? Last I tried it, OSX support was better, but still very
much “try it and see if it works for you.” I personally found
ninja’s OSX support to be fine, but I didn’t really beat on it to
put it through its paces, either. Relatedly, some work environments
have varying ability to install and use “non-standard” tools. I’ve
been in workplaces before that I’d have had to fight tooth and nail
to get something like cmake or ninja available to me on my
workstation. Even now when I don’t have that policy sort of problem,
it’s still a pain when every time I install a new system (which I do
disturbingly often), I need to install additional tools before I can
start using llvm. Not a big problem on its own of course, but an
inconvenience.cmake+ninja doesn’t support building and running the llvm test suite.
That’s pure configure+make still. I rely on configure automatically
setting up the test suite to run on whatever compiler I just built.For example, I build Debug+Asserts, Release+Asserts and Release
configurations all in the same build directory and rely on the
configure+make machinery to create separate build results
subdirectories named for those. That is, it’s pretty common for me
to do something that boils down to (with lots of steps in between of
course), “<src>/configure && make && make check && make
ENABLE_OPTIMIZED=1 && make check ENABLE_OPTIMIZED=1 && make
ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1 && make check
ENABLE_OPTIMIZED=1 DISABLE_ASSERTIONS=1”. Depending on what I’m
doing, I bounce back and forth between them. With cmake, I have to
have completely separate configured build directories. I already
have 10+ llvm build directories lying around that I regularly use.
I’d prefer to keep that number from growing too horribly. When I
last tried using cmake+ninja, this is the one that I kept running
into and getting annoyed with.Do the cmake scripts support cross-compiling? Last I tried, I had
trouble with that. Many moons ago, I had to beat the plain Makefiles
into some sort of submission for that. I not infrequently build llvm
to run on ARM, for example.Somewhat relatedly, I like to keep my development builds as close as
reasonably possible to production builds. There are few things that
make my stomach churn more than bugs that only reproduce on the
production builders. That’s somewhat off-topic, though, as that’s
getting into “what would it take to remove configure+make support”
territory.In short, I’m a big fan of ninja and really want to use it. CMake,
I’m less thrilled about, but I’ll put up with it if it gets me
ninja. Right now though, it’s a close but not quite a fully baked
replacement for my uses.Anyways, none of this should be construed to be a slight to the work
people have put into cmake+ninja support for llvm/clang. Quite the
opposite. I’m excited enough about what’s been done that I’m
frustrated it doesn’t fit my use-cases well enough for me to switch.My point with that comment, which Eli correctly and effectively
addressed in his reply, is that we can’t assume we can define away
problems with the configure+make system by just telling people it’s
deprecated and they should use cmake+ninja instead. We’re not there
yet.
I'd like to second all of this.
Also, I'll add that I also sometimes build on machines where linking a Debug+Asserts clang takes several minutes, and I like to be able to cut off that process to run make check. However, I discovered that if you 'cd test' before running make check, then there are no dependency problems and it will just run the tests with your current build as-is. So long as that works, I'm fine with keeping the top-level make check dependent on the rest of the build.
-Hal
Yeah, check is just: make -C test so that'll work as well.
The point was that if you've made source changes you should really
probably retest what you've changed and the dependency highlights that
in the normal use case. There are many ways around it, but the general
goodness should still be there.
-eric