lldb inline tests seem to generate their own Makefile and then clean it up after they’re done. lldb\test\lang\objc\objc-runtime-ivars seems to have a Makefile checked into the repo. So when I run the test suite, it deletes this repo and creates an annoyance every time I go to commit a changelist I’m working on, because I have to remember to undo the fact that this file is about to get deleted from the repo.
What’s the correct thing to do here? Should we: a) Remove this Makefile from the repo and rely on the inline test to generate it, b) Only do something in CleanMakefile() if BuildMakefile() was previously called? (this isn’t the case for me locally, since all of these tests are disabled on Windows, but the cleanup isn’t behind a similar check), c) some combinatino of the above?
Would appreciate some assistance, as this is very annoying to keep having the test suite modify my in-progress CLs.
Zach,
I think what would be idea is to have some way to tell the python file for a given test to build the binary for that test.
In keeping with the minimalist philosophy for inline tests, it would be nice if this didn’t require any modification (or required only very minimal modification) to the testcase python file.
Sean
lldb inline tests seem to generate their own Makefile and then clean it up
after they're done. lldb\test\lang\objc\objc-runtime-ivars seems to have a
Makefile checked into the repo. So when I run the test suite, it deletes
this repo and creates an annoyance every time I go to commit a changelist
I'm working on, because I have to remember to undo the fact that this file
is about to get deleted from the repo.
What's the correct thing to do here? Should we: a) Remove this Makefile
from the repo and rely on the inline test to generate it, b) Only do
something in CleanMakefile() if BuildMakefile() was previously called?
(this isn't the case for me locally, since all of these tests are disabled
on Windows, but the cleanup isn't behind a similar check), c) some
combinatino of the above?
It would be extremely nice if building & testing didn't modify or create any files in the directory checked out from svn/git. Having that makes it a bit easier to provide build reproducibility guarantees on shipped toolchains.
Cheers,
Jon
Agreed, but that’s a difficult problem to solve given the current architecture of the test suite. In my ideal world, we would just build all the test executables at the same time we build lldb, and that would even speed up the test suite, but i think we’re a ways away from that.
I would like to see "Makefile" files for each test and avoid the "python will magically do this for me" stuff. I had an inline test that was failing and wanted to test what it was doing to figure out what was wrong and I added a Makefile to the directory.
I vote to make each directory contain a makefile so we can CD into and just type "make". Most inline tests might be simple, but it would be nice to easily be able to build them.
The right fix for this is to fix the inline test code to check for a Makefile first and if one doesn't exist, create one, else leave the existing one there.
Greg
Another possibility is dont’ check in anything called Makefile, but instead call the file something else. Like perhaps Makefile.test. You can still easily cd into the directory and test this with make -f. This also requires no code change, and there’s some value in not making any code more complex than it needs to be.
I like your idea: have Python generate a Makefile if one is not present and not delete the Makefile after the run. We just have to remember not to check inline test cases’ Makefiles into the repository.
This way you can have your own Makefiles to build things – and in fact Python will happily leave those Makefiles for you to reproduce test results – but adding a new “inline” testcase is still as simple as possible.
Sean
I put up a patch to lldb-commits. It’s pretty trivial, but please take a look anyway.
Zachary,
your patch looks fine, except for one tiny point: could you leave the CleanMakefile method in there and just have it do nothing? It was quite annoying to find out what the right place to invoke that was, and I’m thinking I might want to add an environment variable later that triggers deletion of Makefiles.
Sean