Summary
I suggest we extend our current test suite with tests that tracks changes to the C++ Standard and update it as Standard text evolves. To make maintenance as incremental as possible, the Git repo with the LaTeX sources of the Standard can be used to version the test suite. With a fixed source of truth for changes to the Standard, a number of opportunities arise. First, it becomes possible to leverage the experience we had with defect reports: breaking the test suite into smaller pieces (e.g. paragraphs), and generating coverage reports (like cxx_dr_status). Second, it gives us more flexibility when referencing things, if it’s deemed useful to use something beyond stable section names.
Even if we don’t have resources for this today, I think it’s useful to have this idea written down for future reference.
Status quo
I’d like to highlight some of the properties of the current test suite under clang/test/CXX that are of significance to this RFC.
First, paragraph numbers are widely used, but sometimes it’s not clear what wording they test. Consider basic/basic.lookup/basic.lookup.unqual/p3.cpp. The latest draft says “Unqualified name lookup from a program point performs an unqualified search in its immediate scope”, which doesn’t seem to match the test. Trying to find the matching paragraph manually won’t get you anywhere because the entire unqualified lookup section was replaced in P1787. While it’s possible to do such digging (and git blame is of great help) for a single test, we should be able to avoid needing to do that.
Second, there’s nothing that resembles a coverage report or a way to generate one automatically (which is one of larger-scale things mentioned above). Compare that to cxx_dr_status, which gives visibility into the total scope of work, and what the implementation status of DRs is (unknown, not available, partial, available). It’s also generated automatically based on CWG issues list and inline markup left at every test. If we try to put something like this together for the current conformance suite, we face at least two issues. 1) it’s hard to define total scope, when many tests were written decade ago against C++11 and 14, but there are, for example, tests for C++20 modules (basic/basic.namespace/basic.namespace.general/p2.cppm). 2) while we can generate a report based on directory names and presence of tests for a particular paragraph, we’d have to assume they are complete, without a way to systematically check this assumption.
The cxx_dr_status page is what made it possible for the Clang C/C++ Language Working Group to suggest new contributors “pick something there” as a starting point. I was one of those newcomers, and here I am, stuck with P1787 
Third, there are inconsistencies in directory layout. Above I used a test of C++20 modules as an example. It’s placed under CXX/basic, because stable name of this sections begins with basic. Whereas the rest of the tests for [basic.namespace] are placed under CXX/dcl.dcl, which is done after the Table of Contents of recent revisions of the Standard. Another example is CXX/dcl, CXX/dcl.dcl, and CXX/dcl.decl existing simultaneously on the same directory depth, which doesn’t make sense however I look at it. The fact that regular Clang contributors (both authors and reviewers) inadvertently make things worse suggests that our processes around conformance test suite should be improved.
Versioning
The granularity of the Standard text updates could be anything from an official publication (once every three years) and draft publication (several times per year), to individual papers and Git commits to official repo. I suggest to version based on commits to the GitHub repository, as not all significant changes to wording are tracked through papers, e.g. CWG and LWG defect report resolutions, or subclauses being rearranged (note: changes that rearrange the standard may or may not be significant for testing, though).
While tracking the official repo seems the most obvious choice, for practical purposes I suggest we use eel.is fork instead because pre-rendered HTML is much more accessible than the LaTeX sources, preserving commit hashes from official repo.
Scope
I believe that testing [lex] through [cpp], [support], and [meta] covers the most of what should be tested within Clang. Notable omissions are:
<stacktrace>std::addressofstd::launder
Note that libc++ may want to consider a similar approach for testing, but this is out of scope for this RFC.
Coverage reporting
I suggest applying our experience with DRs, meaning that we define the total scope using e.g. Table of Contents from a particular version of the Standard, and use a combination of directory layout and inline markup inside tests to fill that total scope with the actual status of the tests. All of that should be done automatically, like make_cxx_dr_status. Here is an example of how such a report could look, using colors from cxx_dr_status:
A test file example that correspond to the table above (I picked up a markup syntax that is independent of directory layout, like the one currently used for DRs, but it’s not the focus here):
// [namespace.udecl]/1: yes
/// test case
// [namespace.udecl]/2: yes
/// test case
// [namespace.udecl]/15: na
// [namespace.udecl]/16: yes
/// test case
// [namespace.udecl]/ex1: no
/// example 1 from the Standard
// [namespace.udecl]/ex2: partial
/// example 2 from the Standard
/// unsupported lines are commented out with a FIXME
/// example 3 is not mentioned anywhere, but present in the wording, so it's considered untested
// [dcl.asm]/1: yes
/// test case
Example commit
[clang] Update conformance test suite to 2023-04-02 draft
Commits applied:
[module.interface] Fix outdated example
Commits silently skipped over:
"[stmt] Fix cross-references for condition" - too big for the sake of example, even to downgrade existing tests to partial
Commits skipped over as editorial:
[over.literal] Cross-reference deprecated grammar
diff --git a/clang/test/CXX/Notes.rst b/clang/test/CXX/Notes.rst
-Wording version 6c039939693c2e5c22a75ed5a8468e717867a7fb
+Wording version dcac5eaf993a190a1bb1335217779bd9ef13a38e
Commits silently skipped over
=============================
+`[stmt] Fix cross-references for condition<https://github.com/Eelis/draft/commit/39c1510d443b647c46de3e84d49a21d442154795>`_
diff --git a/clang/test/CXX/module.interface.cpp b/clang/test/CXX/module.interface.cpp
// [module.interface]/ex6: yes
export module M;
+int g;
export namespace N {
int x; // OK
- static_assert(1 == 1); // error: does not declare a name
+ using ::g; // error: ::g has module linkage
}
Note that the example above depicts how an update would look when put together and does not insist on exact syntax of inline markup (// [module.interface]/ex6: yes) or directory layout (CXX/module.interface.cpp).
Corner cases
Technical Specifications and omnibus papers (like P1787) pose a challenge even with the most granular versioning. I suggest to skip over those leaving notes, ideally marking affected tests as incomplete, but even that could be challenging when there are dozens of pages of wording changes. Being in sync with eel.is and addressing smaller changes seems better to me than being stuck at a big change.
Directory layout
I don’t suggest any particular directory and file layout, as I deem it secondary at this point. Available options depend on other decisions like granularity of updates and inline markup syntax.
Transition
I’d like to lay out a possible scenario for the transition from status quo:
- we decide on versioning, coverage report format, inline markup syntax, and directory layout;
- we prepare tooling for generating coverage report like make_cxx_dr_status, using the scope proposed in corresponding section;
- we decide to take tip of eel.is fork as “reference wording”;
- starting out with the coverage report all red, we assess existing tests against the reference wording, adding inline markup (and possibly moving them to another file, depending on decided directory layout);
- we keep our new test suite in sync with eel.is fork while assessing existing tests.
Resources required
Given we have full and up-to-date test suite, keeping it this way could easily be a full-time job, I think. I understand that community resources are scarce, and I’m not advocating that executing this plan is a better way to utilize those resources over keeping status-quo and dedicating them to e.g. improved C++20 support.