Is anyone using the gtest Xcode project?

In lldb/gtest there is a gtest.xcodeproj folder with what I guess is an Xcode project. If I understand the way Xcode works, the way to use this is by opening this in another instance of Xcode separate from your normal LLDB project, and then building it. Is this right?

I have a patch that moves some files around, and if nobody is using this Xcode project, I would like to delete it. Then, after I get the tests up and running in the CMake build, we can add it to the “real” Xcode project as a separate target similar to how you currently run the LLDB Test suite.

Any objections to deleting the Xcode project?

Xcode has "projects" and then "workspaces" and "schemes". Workspaces aggregate projects. Schemes exist in both workspaces and projects and are the way to say "do something with some of the stuff referred to by this project/workspace." So the way to do this formally is to have the gtest scheme build & run the tests from the gtest project.

The lldb.xcworkspace file does reference the gtest xcode project, and it has a scheme for the gtest.

Not sure what the scheme does yet, I'll look in a few minutes if nobody beats me to it, I'm in the middle of things right now.

Jim

So I’m guessing the scheme runs do-gtest.py. I’d like to delete that file as well as all the Makefiles in the directory if possible. It seems like these files should be built using the normal Xcode build system the same way the rest of LLDB is built.

The way the CMake does it is that each test folder generates a new executable. So right now it will build HostTests.exe, ProcessLinuxTests.exe, and UtilityTests.exe. And then CMake will invoke lit (the LLVM test runner) to run each of the executables one by one and print the output.

I’m not sure if that’s easy or feasible to do in the Xcode build. I kind of don’t want to leave this do-gtest.py and Makefiles in the build though, because the more of this stuff we have the more maintenance it is, and things tend to rot.

I'm not sure if this is what you meant, but I don't see a lot of value in making an Xcode project that has targets for each of the gtest binaries, and then tries to run the tests. Seems to me it would be better if the gtest project just invokes whatever mechanism the cmake build would do to run the tests. That's just another set of things to keep in sync.

It is sufficient to have a target that just does whatever steps cmake/lit do to build the gtests & run them, if that is possible. I guess if you can't do this without running cmake in the lldb top-level directory that would be a problem. But it still seems better to me to wire that up, than to have to add tests to both Xcode & cmake.

Jim

Oh I’m all for reusing as much of the existing mechanism as possible. Was just stating how the CMake worked as a discussion point. Another possibility would be to just have the Xcode project build one executable that pulls in sources recursively from the entire subtree. Is this as easy in Xcode as just adding all sources from a subfolder to a single target?

One day far off in the future it would be nice if all of LLDB’s tests were ported to lit (even if that meant extending lit to make it do what we needed it to do), so I can definitely see some value in hooking lit up to the Xcode build so it does everything the CMake build does. I’ll have to look into exactly what steps the CMake and/or autoconf build are taking, but I suspect it’s going to involve running CMake from a script, so not very desirable. I’m still learning a lot of this stuff though, so there may be a better way. Either way, I’ll have to look into it a little bit.

In the meantime, if running unit tests from Xcode is not part of anyone’s usual workflow, can I remove it for now?

We use it.

Oh I'm all for reusing as much of the existing mechanism as possible. Was just stating how the CMake worked as a discussion point. Another possibility would be to just have the Xcode project build one executable that pulls in sources recursively from the entire subtree. Is this as easy in Xcode as just adding all sources from a subfolder to a single target?

One day far off in the future it would be nice if all of LLDB's tests were ported to lit (even if that meant extending lit to make it do what we needed it to do),

Why would this be nice? It looks like lit is a good test runner for tests that have some input, do something with the input, produce an output and check that output is matches some pattern. That is not at all what the lldb tests look like. They often have to do complex dances - for instance depending on how the line tables come out there are many "correct" ways to step through code. If you are going to test this you've got to do "step, if I got to a close bracket, step again, if I got past it don't. Etc...

I see no benefit in extending a simple runner like lit to do the complex dances the lldb testsuite sometimes has to do. I'm all for sharing, but it is also okay to have two implementations of some functionality if the two uses are sufficiently different, and this certainly seems like one of those cases.

so I can definitely see some value in hooking lit up to the Xcode build so it does everything the CMake build does. I'll have to look into exactly what steps the CMake and/or autoconf build are taking, but I suspect it's going to involve running CMake from a script, so not very desirable. I'm still learning a lot of this stuff though, so there may be a better way. Either way, I'll have to look into it a little bit.

Jim

Wasn’t really trying to get into an extended discussion about this, but FWIW I definitely realize that lldb’s tests are more complicated than what lit currently supports. But that’s why I said “even if it meant extending lit”. It was mostly just a general comment about how it’s nice if everyone is focused on making one thing better instead of everyone having different things.

As for specifics, my understanding is that lit parallelizes better (so running tests is faster), understands how to build programs (so doesn’t require makefiles), and has a richer language for specifying how and under what circumstances different tests should be run. It’s also familiar to other LLVM developers (so encourages cross-collaboration), and allows one to write self-contained tests with the program to test and the check in a single file (less maintenance).

In any case, I’m really not an expert on lit, so +bogner and +chandlerc in case they want to chime in. I do think it’s at least worth thinking about whether lit could be extended to meet LLDB’s needs – if nothing else as a thought exercise, and maybe learning more about how it works would give us some ideas to make our own test suite better.

Wasn't really trying to get into an extended discussion about this, but FWIW I definitely realize that lldb's tests are more complicated than what lit currently supports. But that's why I said "even if it meant extending lit". It was mostly just a general comment about how it's nice if everyone is focused on making one thing better instead of everyone having different things.

Depending on how different the different things are. Compiler tests tend to have input, output and some machine that converts the input to the output. That is one very particular model of testing. Debugger tests need to do: get to stage 1, if that succeeded, get to stage 2, if that succeeded, etc. Plus there's generally substantial setup code to get somewhere interesting, so while you are there you generally try to test a bunch of similar things. Plus, the tests often have points where there are several success cases, but each one requires a different "next action", stepping being the prime example of this. These are very different models and I don't see that trying to smush the two together would be a fruitful exercise.

Jim

Well, as a quick example of where I think there’s a considerable amount of overlap between the high level model of how the test operates is the case of the lldbinline tests.

The lldbinline tests are an okay way to write a very simple class of tests. But they will not suffice for many of the tests we need to write. I am actually not a big fan of these tests because when they fail it is a royal pain to reproduce the steps that led to the failure. I don't think making a wholly different runner to run this is going to make that situation any better.

Jim

Well, like I said. I’m just thinking :slight_smile: No need to worry

Back to the original question, is it as easy as it seems to just create one target in Xcode that manually includes each file recursively in the subtree? So it just builds one executable?

Wasn't really trying to get into an extended discussion about this, but
FWIW I definitely realize that lldb's tests are more complicated than
what lit currently supports. But that's why I said "even if it meant
extending lit". It was mostly just a general comment about how it's
nice if everyone is focused on making one thing better instead of
everyone having different things.

As for specifics, my understanding is that lit parallelizes better (so
running tests is faster), understands how to build programs (so doesn't
require makefiles), and has a richer language for specifying how and
under what circumstances different tests should be run. It's also
familiar to other LLVM developers (so encourages cross-collaboration),
and allows one to write self-contained tests with the program to test
and the check in a single file (less maintenance).

I don't know much about lldb's testing needs, but I thought I'd chime in and mention that there's a lot of magic in the lit TestFormat (which testsuites can define themselves), and that it's waaaaay more flexible than it looks.

LLVM uses the ShTest test format a lot, and that one is well-suited for the process Jim describes: building shell script pipelines with checkers at the end, where the test file itself serves as both the input to the beginning of the pipeline, and the input to the checker at the end.

Libcxx and Libcxxabi have their own LibcxxTestFormat that behaves a little differently: they make the assumption that every test is a c/c++ file which needs to be compiled, executed, and that an exit status of 0 is a pass.

From a quick look at the lldb testsuite, I imagine if you wanted to LIT-ify it, it would make sense to have yet another TestFormat. This test format would know that it needs to look for *.py files, invoke the Makefile next to them (or put metadata in comments at the top of the *.py that says how to build it), then run the *.py file. I think the changes to the tests themselves would be minimal, as would writing the lit.cfg to glue it all together.

Cheers,

Jon

Well, like I said. I'm just thinking :slight_smile: No need to worry

Back to the original question, is it as easy as it seems to just create one target in Xcode that manually includes each file recursively in the subtree? So it just builds one executable?

I don't think so. You can ADD a folder of sources to a project, but that just makes all the files available to the project. You then have to manually tell Xcode which files build into which targets. That's pretty easy to do, but I don't know of a way to get it to "include all .c files in the current target."

Jim

(I removed Chandler & Justin 'cause I doubt they care about Xcode...)

Ahh yea, i think it’s fine the way you describe (explicitly adding each one to the right target), because that’s how the Xcode project works for regular LLDB no? So we could have a Tests folder in Xcode, which contains Host, Plugins, and Utility folders, and those folders contain more files (or subfolders) and all of this gets compiled into a single executable named lldb-unit-tests.

That’s interesting! So essentially we could use all of LLDB’s existing test infra except the part that builds the executables. That would at least get rid of the need to write Makefiles.

I know we have some logic to compile the same executables multiple times with different target architectures and even different compilers, but i think lit supports this too because ASAN does something similar.

Right, you could either do this in the lldb.xcodeproj or in a separate project which you include in the lldb.xcworkspace. There's no real reason to cram everything into the lldb.xcodeproj, as long as it is added to the workspace it is easy to access it, set up dependencies, etc.

Jim

What do you use it for? How bad will it disrupt you if it’s temporarily broken? What if, during the temporarily broken time, it still works through the cmake build on osx?

Trying to figure out if i need to synchronize fixes to the Xcode build.

jingham@apple.com writes:

Wasn't really trying to get into an extended discussion about this,
but FWIW I definitely realize that lldb's tests are more complicated
than what lit currently supports. But that's why I said "even if it
meant extending lit". It was mostly just a general comment about
how it's nice if everyone is focused on making one thing better
instead of everyone having different things.

Depending on how different the different things are. Compiler tests
tend to have input, output and some machine that converts the input to
the output. That is one very particular model of testing. Debugger
tests need to do: get to stage 1, if that succeeded, get to stage 2,
if that succeeded, etc. Plus there's generally substantial setup code
to get somewhere interesting, so while you are there you generally try
to test a bunch of similar things. Plus, the tests often have points
where there are several success cases, but each one requires a
different "next action", stepping being the prime example of this.
These are very different models and I don't see that trying to smush
the two together would be a fruitful exercise.

lit's pretty flexible. It's certainly well suited to the "input file,
shell script, output expectations" model, but I've seen it used in a
number of other contexts as well. jroelofs described another use in his
reply, and I've also personally used lit to run arbitrary python
programs and FileCheck their output. That said, I don't know a ton about
lldb's test infrastructure or needs.

I'd expect lldb tests that can't be focused into a unit test to be
somewhat "expect"-like (pardon the pun), where there's something to be
debugged and a kind of conversation of input and expected output. Is
this a reasonable assumption?

I don't know of any current lit tests that work like this, but it would
be pretty simple for lit to act as a driver for a program that tested
things in an interactive way. I'd imagine the interactivity itself being
a separate utility, much like we delegate looking at output to FileCheck
or clang -verify in the llvm and clang test suites.

Anyways, I don't necessarily know if it's a good fit, but if we can make
lit suit your needs I think it'd be a nice gain in terms of bots being
easy to set up to show errors consistently and for random folks who are
familiar with LLVM but not necessarily LLDB to be able to quickly figure
out and fix issues they might cause.

we can add it to the “real” Xcode project as a separate target

What does this mean?