Relationship of LLVM tests with GCC tests

I wonder about the relationship between LLVM and GCC test suites. I imagine it makes sense for both sides to use the other compiler’s tests as well. Is this already happening in some way? I was considering running my LLVM / clang developments also against the GCC test suite (and hacking something to orchestrate DejaGnu and Lit), but if the two test suites are already overlapping this would lead to a lot of redundant testing.

Is there a place where this relationship between both is described?

I think the problem here is the different infrastructure. Clang uses -verify for semantics and FileCheck for CodeGen and transformations. But GCC uses their own test settings. Although it would be good clearly if we could share one test suite. But considering the current situation, I think it is not realistic to do.

I agree that this is difficult to integrate. If we were to ignore these engineering difficulties, is it possible to get an intuition for what kind of coverage overlap the the existing GCC and LLVM tests for clang have? Or in other words, if I “somehow” manage to run my clang also against the GCC tests, what kind of percentage of additional bugs could I reasonably expect?

It is hard too. The key point here is that we couldn’t summarize the testing aim of each test in clang and gcc. It is not described in formal structure. Some of them are described in bug pages or in review pages and both of them is in natural languages. And some of them is not described at all.

So I feel like it is not nearly possible to me.

I am not sure about what GCC test-suite you have in mind, but just FYI llvm-test-suite already contains part of GGC’s C ‘torture’ tests: llvm-test-suite/SingleSource/Regression/C/gcc-c-torture at main · llvm/llvm-test-suite · GitHub.

1 Like