After searching the whole project, I only find about ~10000 cases from
“llvm/test” for each commit, and a separate testsuit wrote with high level
language(i.e. C/C++) to verify the quality and performance. As a general
Backend, you know, it must be strong enough to cope with all the IR
generated by Frontend. I cannot believe what I see. Did I miss something ?
You’re missing all the Clang, compiler-rt, libc++ tests on their own
repositories. That’ll add a few more tens of thousands of tests into
the bundle. But that’s just the regression / base validation, the
release has a second stage which is less visible than, I believe, for
GCC. We also run the test-suite and benchmarks on trunk validation,
which is something I believe GCC doesn’t do, so releases have a lot
less bugs than GCC to begin with. GCC focus on fixing all bugs at
release time, while LLVM focus on fixing them as they happen, which is
much easier and more stable for all developers.
Yes, there are many test cases that we could leverage as LLVM is widely used. However, I am not sure if these cases are part of the release process,
as they are not part of this project. Is it that, when LLVM is going to release, we must pass all the regression tests and test-suit to make sure that, the basic functionality and performance is good. Then, release it. Clang/rt and other project would use the new LLVM and report bugs to us if have issues?
The regression test and test-suit is our release criteria, and people would do more tests for us from other projects. Is it right?
Further, I notice that, under llvm project repo, there is also a clang-tests
that using gcc test
Is that test suit used by llvm to guarantee the quality before the release ?
The GCC test-suite, AFAIK, has very poor quality on what’s considered
a pass or a failure, and it’s common to release GCC with thousands of
failures on those tests. Some people may run it, but I honestly don’t
trust it myself, nor have the time to sift through every single test
to make sure my errors are compiler errors or test errors. You can’t
assume that just because GCC runs more tests, that what they’re
testing is more thorough. There are also lots of tests that have
erratic behaviour, which only adds noise to the process.
The release process also involves passing standard compiler benchmarks
from the part of the base testers, and higher level applications (like
Chromium) from the community. Different targets may get different
community interest, but most targets have an additional phase inside
companies like, ARM, Mips, Intel, Apple, Google, Sony, Qualcomm, etc.
They all have internal work-loads that represent a larger piece of
real world code that the test-suite can offer. Whenever those work
loads fail, we get bug reports. It’s also good practice to add a
snippet to the test-suite or the regression tests in these cases.
As a separate quality control, there are a few efforts tracking
trunk/releases to build the Linux kernel, Debian, FreeBSD, Mandriva,
OpenEmbedded and other large scale projects. Whenever something breaks
on those projects, bugs are reported and fixed on the next stable
I think it’s a pretty solid validation process for both trunk and releases.
Agree that, LLVM tests have much better quality than GCC’s. So all the tests are most provided by developers and bugzilla. Do we have the test team to do that ?