clangd testcase style

Hi,

I have noticed that many of the unit tests in clangd are written in a style where a single testcase from the gtest point of view actually runs a large array of testcases. Some examples are SemanticHighlighting.GetsCorrectTokens and LocateSymbol.All.

I realize this saves a bit of typing when writing the test cases, but I find it impedes debuggability.

My current workflow for debugging an individual test case in a test like this, is to comment out all the other test cases in the array, and then run the test under the debugger, which is pretty laborious I was wondering, does anyone else have a better workflow?

Thanks,
Nate

Hi Nathan,

I usually ensure the test outputs its inputs in case of failure and then do a text search for that.

It’s not super convenient, but definitely does the job.

The issue is not with locating the failing testcase, but with running only that one test case under the debugger.

Suppose I want to investigate a failure in a particular testcase, and I put a breakpoint in some relevant code. I don't want to be running all the other testcases as well, and have to skip over that breakpoint for the other test cases, before reaching mine. Hence the commenting-out strategy.

Hi Nathan,

Sorry about not replying at the time, thanks for the reminder!
The table-based tests are mostly my fault IIRC, and I think what you’re describing is the biggest downside of them.

I have noticed that many of the unit tests in clangd are written in a style where a single testcase from the gtest point of view actually runs a large array of testcases. Some examples are SemanticHighlighting.GetsCorrectTokens and LocateSymbol.All.

I realize this saves a bit of typing when writing the test cases,

I don’t think that quite covers it :slight_smile: Compared to how we used to write unit tests, the main benefits are:

  • maintenance: improving the patterns of a collection of related tests, or even identifying them, was hard. Changing (non-test) APIs used in hundreds of tests was so painful.
  • quality: many of the critical table tests have somewhat elaborate assertions that allow the test to be expressed concisely and giving clear failure messages. These (rightly) don’t get written unless there’s a way to reuse them.
  • coverage: the barrier to introducing a new test (that fits the schema) is very low, so people write more tests (maybe sometimes too many!)
  • readability: the table entries specify tests in a declarative way, which helps avoid the intention being tangled up with mechanisms. The form is fairly compact, and hierarchical in the source code (test → entry) which helps with navigation.
  • writability: we do in fact save some time writing tests :slight_smile: I think this is probably the least important factor.

(Now I’m getting bad memories of when we only had lit tests and they had to exactly reflect our printf’d json output and Content-Length needed to be exactly right and…)

but I find it impedes debuggability.

My current workflow for debugging an individual test case in a test like this, is to comment out all the other test cases in the array, and then run the test under the debugger, which is pretty laborious I was wondering, does anyone else have a better workflow?

+1 this is annoying, that’s the best workflow I have too, it’d be great to have something better.

The alternatives I’ve thought about:

  • TEST_CASE_F with a fixture class. Considerably more boilerplatey. No forcing function to keep the structure the same for tests, so they tend to diverge somewhat.
  • INSTANTIATE_TEST_CASE_P: my experience with this is it’s a surprisingly badly designed piece of gtest that’s painful to write, run, or understand tests in :frowning:
  • Something custom for table driven tests. I’m imagining some hybrid of TEST_CASE_F and INSTANTIATE_TEST_CASE_P like TEST_INSTANCE(GoToDefTest, Heuristic, { … table entry … }). But needs some investigation.

Are there others?

I realize this saves a bit of typing when writing the test cases,

I don't think that quite covers it :slight_smile: Compared to how we used to
write unit tests, the main benefits are: [...]

Thanks for going over these benefits, they all make sense to me and are good to keep in mind when considering alternatives.

The alternatives I've thought about:
- TEST_CASE_F with a fixture class. Considerably more boilerplatey. No forcing function to keep the structure the same for tests, so they tend to diverge somewhat.
- INSTANTIATE_TEST_CASE_P: my experience with this is it's a surprisingly badly designed piece of gtest that's painful to write, run, or understand tests in :frowning:
- Something custom for table driven tests. I'm imagining some hybrid of TEST_CASE_F and INSTANTIATE_TEST_CASE_P like TEST_INSTANCE(GoToDefTest, Heuristic, { ... table entry ... }). But needs some investigation.

I initially envisioned a fixture based approach, but you're right that we could further reduce boilerplate with a custom macro. I'll try to prototype something next time I need to write a test like this.

Thanks,
Nate