[RFC] Lanai backend

Project Infrastructure LLVM Dev List Archives
#1

But I also see another option, which someone else mentioned up-thread:
simply make only the regression tests be supported. Without a regression
test case that exhibits a bug, no reverts or other complaints. It would be
entirely up to the maintainer to find and reduce such test cases from any
failure of execution tests out-of-tree.

IMHO, it's about transparency and commitment.

For experimental back-ends (like BPF), what you describe is perfectly
fine. If we wanted BPF to be official, I'd personally only accept it
if there was at least one buildbot with a minimal domain-specific set
of tests. In the BPF case, I'd expect a Linux booting and running some
arbitrary code expecting a certain result. In a hardware-simile, like
your back-end, I'd expect some generic code to be compiled and run
successfully, strongly biased to getting the test-suite running on it.

BPF backend has been the first class citizen for a year or so.
All major linux distros ship llvm with it turned on.
(unlike some other backends)
BPF buildbot processes more commits than arm64 :slight_smile:
buildername | completed

failed | time

clang-aarch64-lnt |
44 | 21 | 02:25:25
clang-atom-d525-fedora |
136 | 130 | 09:02:23
clang-atom-d525-fedora-rel |
146 | 117 | 01:57:06
clang-bpf-build |
311 | 32 | 00:03:29
clang-cmake-aarch64-42vma |
150 | 31 | 00:47:52
clang-cmake-aarch64-full |
50 | 15 | 03:38:37
clang-cmake-aarch64-quick |
162 | 36 | 00:43:11

#2

The BPF back-end is, indeed, built by default, but the nomenclature of
"first class citizen" is hazy. I don't think we ever got around
defining it properly and the list that used to mean that [1] is
severely outdated.

The page that may have a bit more meaning is [2], when it states the
criteria for quality as:

1. Code must adhere to the LLVM Coding Standards.
2. Code must compile cleanly (no errors, no warnings) on at least one platform.
3. Bug fixes and new features should include a testcase so we know if
the fix/feature ever regresses in the future.
4. Code must pass the llvm/test test suite.
5. The code must not cause regressions on a reasonable subset of
llvm-test, where “reasonable” depends on the contributor’s judgement
and the scope of the change (more invasive changes require more
testing). A reasonable subset might be something like
“llvm-test/MultiSource/Benchmarks”.

Item 2 above cannot apply to the BPF of Lanai back-ends. We normally
use a stronger criteria saying that the code "must build on at least
one *major* platform", that normally meaning x86 or ARM32/64, maybe
Mips or PPC? Since most people develop on an x86 anyway, that ends up
being the common line.

Having said that, item 2 above doesn't apply to most LLVM back-ends,
so that's not a good criteria for "first class citizen".

In my view, a "first class citizen" is a target (not just the
back-end) that is supported throughout its life-cycle. Meaning you can
prove that user code (including benchmarks) can correctly run on your
target at expected speeds. If you pass the FileCheck tests but no one
can use the results, than I *cannot* assert that your target is
"supported". I can create an infinite number of tests that do nothing
and say my fake target has a million tests passing and they finish in
under 10 seconds, but that is also pointless.

The BPF back-end has a real use case and indeed a lot of good tests,
but I don't know if your buildbots test the resulting code in a kind
of simulator or if you have added real code to the regression tests,
so that we can spot one some of them stop working. If that's so, then
it fits all (my personal) criteria of a "first class citizen". If GPU
targets' buildbots do test the resulting code in their GPUs, then the
same apply to them.

Right now, the Lanai back-end has no means of execution by the
community, so could never become a "first class citizen". In the event
that Google puts up a public buildbot with a Lanai
simulator/emulator/hardware and passes the appropriate tests, it'll be
a community-expensive target to have (because reproducibility is only
allowed to one company), but could become a "supported" back-end IFF
Google is quick enough to solve all problems. If Google releases an
emulator public, then it'll be a lot easier to make the target
"supported" and promote it to a "first class citizen".

cheers,
--renato

[1] http://llvm.org/docs/GettingStarted.html#requirements
[2] http://llvm.org/docs/DeveloperPolicy.html#quality

#3

> BPF backend has been the first class citizen for a year or so.
> All major linux distros ship llvm with it turned on.
> (unlike some other backends)
> BPF buildbot processes more commits than arm64 :slight_smile:

The BPF back-end is, indeed, built by default, but the nomenclature of
"first class citizen" is hazy. I don't think we ever got around
defining it properly and the list that used to mean that [1] is
severely outdated.

The page that may have a bit more meaning is [2], when it states the
criteria for quality as:

1. Code must adhere to the LLVM Coding Standards.
2. Code must compile cleanly (no errors, no warnings) on at least one
platform.
3. Bug fixes and new features should include a testcase so we know if
the fix/feature ever regresses in the future.
4. Code must pass the llvm/test test suite.
5. The code must not cause regressions on a reasonable subset of
llvm-test, where “reasonable” depends on the contributor’s judgement
and the scope of the change (more invasive changes require more
testing). A reasonable subset might be something like
“llvm-test/MultiSource/Benchmarks”.

Item 2 above cannot apply to the BPF of Lanai back-ends. We normally
use a stronger criteria saying that the code "must build on at least
one *major* platform", that normally meaning x86 or ARM32/64, maybe
Mips or PPC? Since most people develop on an x86 anyway, that ends up
being the common line.

Having said that, item 2 above doesn't apply to most LLVM back-ends,
so that's not a good criteria for "first class citizen".

I think that's perhaps not the right interpretation of (2) - it looks to me
like rule 2 is talking about the code in question (the backend), not the
code it can generate. (eg: when the Clang AST matchers were contributed,
rule 2 applies equally there - that the code for the matchers must compile
cleanly without warnings or errors on at least one platform (I take it the
"on at least one platform" is for the sake of platform specific
code/features - like code to support running the compiler on Windows (as
distinct from the Windows support for the output of Clang)))

I don't think this rule has anything to do with backends and what they
can/cannot generate.

In my view, a "first class citizen" is a target (not just the
back-end) that is supported throughout its life-cycle. Meaning you can
prove that user code (including benchmarks) can correctly run on your
target at expected speeds. If you pass the FileCheck tests but no one
can use the results, than I *cannot* assert that your target is
"supported". I can create an infinite number of tests that do nothing
and say my fake target has a million tests passing and they finish in
under 10 seconds, but that is also pointless.

Sure - and we wouldn't accept those, nor would anyone propose them.

The BPF back-end has a real use case and indeed a lot of good tests,
but I don't know if your buildbots test the resulting code in a kind
of simulator or if you have added real code to the regression tests,
so that we can spot one some of them stop working. If that's so, then
it fits all (my personal) criteria of a "first class citizen". If GPU
targets' buildbots do test the resulting code in their GPUs, then the
same apply to them.

Right now, the Lanai back-end has no means of execution by the
community, so could never become a "first class citizen". In the event
that Google puts up a public buildbot with a Lanai
simulator/emulator/hardware and passes the appropriate tests, it'll be
a community-expensive target to have (because reproducibility is only
allowed to one company), but could become a "supported" back-end IFF
Google is quick enough to solve all problems.

This seems odd to me - and much like the discussions we've previously had
about buildbots. If only group X cares about hardware Y then I think it's
an entirely acceptable community behavior for group X to run tests
internally and be responsible for triaging failures on Y and communicating
that back to the community in whatever timeline is important to them -
since the hardware isn't of interest to anyone else. Having a bot that
sends mail is where the urgency is created that requires the bot to be of
high quality, its results to be independently actionable, etc - but I don't
see it as a necessity & I think other targets we already have could benefit
from that approach even though their hardware is public/accessible (for a
price, that no one generally pays). Most of us only have one architecture
available to us (maybe two - X86/desktop hardware + test hardware of the
target we're working on), so we just care that when something says
"something is broken" we can act on that. I don't personally mind if that
is automated or not - but the longer the latency the more onus is on the
invested group to do the legwork. They can expect to be broken and for the
codebase to have moved on by the time they root cause (& have to do more
work to figure out where/how to fix it - becomes more of a discussion with
the community members with more onus on the target owners, etc) - this
happens today with high latency bots, etc. Doesn't seem like anything new.