Easy way to generate lit tests for clang?

Clang Frontend
1

Hi,

I’m attempting to refactor a large test file and was hoping to see if there was an easier way to automatically update this test to check exact IR output. Perhaps something along the lines of utils/update_llc_test_checks.py in llvm, where the checks are automatically generated for each RUN to match IR exactly. Does such a tool exist?

Thanks,
Leonard

2

Leonard Chan via cfe-dev <cfe-dev@lists.llvm.org> writes:

I'm attempting to refactor a large test file and was hoping to see if
there was an easier way to automatically update this test to check
exact IR output. Perhaps something along the lines of
`utils/update_llc_test_checks.py` in llvm, where the checks are
automatically generated for each RUN to match IR exactly. Does such a
tool exist?

Personally, I really would prefer we not go further down that road.
Such tests are very brittle and result in a lot of noise in diffs up for
review. When I see tests like that in the diff I gloss over them
because they are so big. I have no idea if the person who updated the
test just ran a script to do so or actually understood the differences
and deemed them correct.

Tests should be small and focused, checking only the very specific thing
they are testing. We should be striving to change existing tests as
little as possible.

Over time, large tests should be broken up into several smaller tests if
possible.

                     -David

3

From: cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org] On Behalf Of David
Greene via cfe-dev
Sent: Friday, June 14, 2019 4:47 PM
To: Leonard Chan via cfe-dev
Cc: Leonard Chan
Subject: Re: [cfe-dev] Easy way to generate lit tests for clang?

Leonard Chan via cfe-dev <cfe-dev@lists.llvm.org> writes:

> I'm attempting to refactor a large test file and was hoping to see if
> there was an easier way to automatically update this test to check
> exact IR output. Perhaps something along the lines of
> `utils/update_llc_test_checks.py` in llvm, where the checks are
> automatically generated for each RUN to match IR exactly. Does such a
> tool exist?

Personally, I really would prefer we not go further down that road.
Such tests are very brittle and result in a lot of noise in diffs up for
review. When I see tests like that in the diff I gloss over them
because they are so big. I have no idea if the person who updated the
test just ran a script to do so or actually understood the differences
and deemed them correct.

Hear hear. That test tactic does nothing more than use the software
under test as its own oracle and lock down its behavior as a reference.
Basically you are asserting the post-patch behavior is correct, which
would rankle even aside from any preference for test-first development.
I understand the motivation, but the net result is that the valuable
parts of the test are obscured by the irrelevant parts.
--paulr