>>>> > Dear all,
>>>> > Recently, I and a number of my colleagues have run into cases where
we would like the ability to write tests that involve components from
multiple LLVM projects, for example using both clang and LLD. Similarly, I
have seen a few instances recently where tests would ideally make use of
LLD but only to help generate input objects for testing an LLVM tool, such
as llvm-symbolizer (see for example ⚙ D88988 [llvm-symbolizer] Add inline stack traces for Windows. <
⚙ D88988 [llvm-symbolizer] Add inline stack traces for Windows.>). Currently, there is no location where
lit tests that use both clang and LLD can be put, whilst the
llvm-symbolizer cases I’ve hit are testing llvm-symbolizer (and not LLD),
so don’t really fit in the LLD test suite. I therefore have prototyped a
lit test suite that would be part of the monorepo, and which can support
tests that use elements from multiple projects - see
⚙ D95339 [RFC][test] Adapt debug-info lit framework for more general purposes - part 1. Tests
could be added to this suite as needed. The suite is modelled as an
additional top-level directory, and is enabled by enabling the
“cross-project-tests” project in CMake. I have initially added support for
both LLD and clang. At configuration time, the tests that require LLD or
clang will only be enabled when the respective projects are enabled, so
that developers continue to benefit from the subset of tests that are
applicable for the projects they are building. Note that I am not
especially familiar with CMake or lit, so this code may not be perfect, but
it should be sufficient to demonstrate what it can do.
>>>> > One could argue that these sorts of tests should belong in the
external (to the monorepo) test-suite, but this is a) quite distant from
the existing testing, and therefore easily forgotten, delaying potential
feedback for any breakages/resulting in potentially duplicate testing etc,
and b) is not as easy to set up and run (owing to the fact that it isn’t
part of the monorepo, isn’t connected to check-all etc), therefore making
it harder for developers to maintain the tests. Back in October 2019, there
was an extensive discussion on end-to-end testing and how to write them
[cfe-dev] RFC: End-to-end testing <
[cfe-dev] RFC: End-to-end testing>). The
suggestion was that these tests would be lit-based and run as part of
check-all, and would not be inside the clang tree, although there was some
opposition. This concluded with a round table. Unfortunately, I am unaware
of what the conclusion of that round table conversation was, so it’s
possible that what I am proposing is redundant/being worked on by someone
else. Additionally, I don’t consider all classes of tests that the proposed
lit suite would be useful for to be “end-to-end” testing. For example,
llvm-symbolizer is usually used on linked output containing debug
information. Usually tests that consume objects that have debug data in
them rely on assembly that has been written by hand or generated by clang
prior to commit, with a limited set able to make use of yaml2obj to
generate the debug data instead. However, the output of these approaches is
typically not a fully linked output (yaml2obj output can be made to look
like one, but getting all the addresses to match up in a maintainable
manner makes this approach not particularly viable). Being able to use LLD
to link the object file produced would make the test significantly more
readable, much as using llvm-mc and assembly to generate test inputs is
more preferable to using prebuilt binaries. Such a test is ultimately not
really any more an end-to-end test than an llvm-symbolizer test that just
uses the object produced by the assembler directly.
>>>> > What do people think?
>>>> Some concerns (the usual: Things should be tested in isolation, things
>>>> should be tested independently - but end to end tests have some value
>>>> too), but generally seems good.
>>>> Indeed this is a usual concern: such tests shouldn't be seen as
replacing isolated lit tests ("unit tests").
>>>> I completely agree. Indeed, the llvm-symbolizer test referenced in
the review I'd class as an isolated test - LLD was just being used to
generate the input for the test. The key here is that the thing being
tested was the llvm-symbolizer behaviour, and not the linker behaviour. As
mentioned, this isn't really different to how llvm-mc or llc might be used
to convert some input source (asm/IR etc) into something the tool under
test wants to be run on. Potentially, changes in those tools might break
things, but as long as the input is specific enough, this shouldn't happen
>>>> But I have another question about the cost of maintenance here: are
we gonna revert patches to either project when one of the integration tests
>>>> Possibly, yeah. If they demonstrate a bug.
>>>> That would be my intention - these tests should be classed as
first-class citizens as much as any other lit test. They're just unit tests
that use other components (LLD, clang etc) to generate inputs.
>>>> What about integration tests that require to be updated manually when
changing another component?
>>>> If they need to be updated, because their failure isn't
representative of a bug, yes.
>>>> Hopefully these sorts of failures would occur only infrequently. As
noted, my aim here isn't to provide a place in opensource LLVM to do
integration testing, but rather unit tests that just can't sit in the
corresponding lit area, so input variability should be minimal.
>>>> That being said, the proposed suite could be used for integration
testing, if the community agreed such testing belonged in the monorepo -
indeed, I have plans for some downstream integration tests that would make
use of this if it lands - but that isn't the goal here.
>>>> I find the cost of maintenance of end-to-end tests is often hard to
carry over, especially since they are only supplementing and not replacing
>>>> One of the nice thing about end to end tests (as with all tests, if
designed carefully - eg: don't take some arbitrary code, compile it with
optimizations, and expect a very specific backtrace - optimizations might
lead to different line table/stack frame details (if some code was merged,
or moved, it might lose or gain a specific source file/line)) is that they
can be pretty resilient to implementation details, so less likely to need
updating due to changes in implementation details. If someone changes the
output format of llvm-symbolizer these would require updating and I think
it'd be reasonable to expect that to be updated/not left failing.
>>>> Right, just as we would change other tests for tools where the output
>>>> - Dave
>>>> Though perhaps debuginfo-tests (this presumably already supports the
>>>> multiple-subproject mechanical isssue you're discussing?) could be
>>>> generalized/renamed to be all our cross-project lit testing
>>>> (Essentially an in-monorepo, lit-only, high-reliability/not-flakey/etc
>>>> version of the test-suite).
>>>> The existing debug-info test area looks like it could work if it was
more generalised. It looks like we'd want the ability to have tests that
would still work without clang/lldb being built, and similarly which could
use LLD, but I don't think those are insurmountable issues - it would just
require taking some of the ideas from my existing prototype (or equivalent
from elsewhere) and merging them in.
>>>> Sounds good to me - might want some buy-in from other debuginfo-tests
>>> I don't mind generalizing debuginfo-tests to support more use-cases,
but I have some very practical concerns: Because of the hardware
constraints on green-dragon (currently a single Intel Mac Mini running all
LLDB tests) I would like to avoid getting into a situation where we need to
build more than clang+lldb in order to run the debuginfo-tests. I also
don't want to have to run more than the debuginfo tests on that bot. So as
long as a separate "check-..." target is used and the bots keep working,
I'm fine with this.
>>> Build design question: How would you feel if it was the same check-
target, but it dynamically chose what to run based on which projects were
checked out/being built? I'm not sure if that's a good design or not, but
having a combinatorial set of explicit check-* names based on which
projects are enabled would seem unfortunaet.
>> Generally I'm not a fan of automatic behavior: You could have a
misconfiguration that leaves out a target and the bot would still run the
same target and it would aways be green, but it wouldn't run any tests. But
I'm not going to stand in the way if such a set up makes most sense here.
>>> (though, inversely - if some set of tests (like the existing debug
info ones) are in a subdirectory, and like the current
check-llvm-debuginfo-* etc where you can specify a target that's a
subdirectory, would mean you coudl run just the debuginfo ones - that
wouldn't be so bad, but maybe we'd eventually have tests under there that
might require lld for instance - and it'd still be nice for those tests to
degrade gracefully to "unsupported" if you weren't building lld (similarly,
if you aren't building clang, the existing debuginfo tests could degrade to
>> That actually sounds nice. The unsupported category would show up in
the test result xml file and thus any change would show in the Jenkins
>> -- adrian
> I just tested a build. After `ninja lldb` (~4000 actions), `ninja lld`
requires just ~152 actions. `ninja llvm-symbolizer` requires 7 actions.
> Perhaps the additional actions are fine? (Linking lld may need some
resources, as it links in IR/CodeGen libraries for LLVM LTO.)
> There are currently some magic `IN_LIST` in CMakeLists.txt: compiler-rt,
> compiler-rt & mlir are for one or two tiny targets.
I guess this is fine — that particular bot is already building the lld