Looks like I missed a party.
I'll try to give my thoughts on some of the things that were said here:
I don't think make -C does what you think it does. "make -C foo" is
basically equivalent to "cd foo && make", which is what we are doing
now already. Of course, you can make this work, but you would have to
pass an extra OUTDIR=... argument to make and then modify the
Makefiles to reference $(OUTDIR) for its outputs:
$(CC) -o $(OUTDIR)/a.out main.cc ...
The standard way of doing an out-of-tree build with make is to have
the Makefile in the build-directory and to set the magic VPATH
variable in the Makefile (or as a part of make invocation). VPATH
alters make's search path, so when searching for a dependency foo and
the foo is not present in the current (build) directory, it will go
searching for it in the VPATH (source) directory. You still need to be
careful about paths in the command line (generally this means using
make variables like $@ and $< instead of bare file names), but our
makefiles are generally pretty good at this already. We even have a
couple of makefiles using VPATH already (see TestConcurrentEvents) --
Todd added this to speed up the build by spreading out tests over
different folders while sharing sources (the serial execution
I still fully support being able to build the tests out of tree, I
just think it may be a bit more involved than you realise.
I agree that using cmake for building tests would some things simpler.
Building fully working executables is fairly tricky, especially when
you're cross-compiling. Take test/testcases/Android.rules for example.
This is basically a reimplementation of the android cmake toolchain
file distributed with the android ndk. If we had cmake for building
tests we could delete all of that and replace it with
However, I only had to write Android.rules just once, so it's not that
big of a deal for me.
explicit RUN lines:
Yes, it's true that all you need is custom CXXFLAGS (and LDFLAGS), but
those CXX flags could be quite complex. I'm not really opposed to
that, but I don't see any immediate benefit either (the only impact
for me would be that I'd have to translate Android.rules to python).
running clean after every test
Currently we must run "make clean" after every test because make does
not track the changes in it's arguments. So, if you first run "make
MAKE_DWO=NO" and then "make MAKE_DWO=YES", make will happily declare
that it has nothing to do without building the DWO binary. (This will
go away if we run each test variant in a separate directory).
I don't have any data, but I would expect that running make upfront
would make a significant performance improvement on windows (spinning
up tons of processes and creating/deleting a bunch of files tends to
be much slower there), but to have no noticable difference on other