Paragraph numbers in C++ tests

Hi,

An annoying part of writing tests for C++ is that the paragraph numbers are not
stable between versions of the standard. Is there an accepted way to deal with
this issue ? I can imagine several ways to deal with this:
  - Rename the tests so that they match the latest standard/draft.
  - Add a suffix/prefix with the version for the paragraph number.
  - ... ?

Bruno

Even if they're updated to reflect current paragraph numbers, they'd
still fall into disrepair in future standards & be hard to know which
version they applied to. So I'd support adding C++ version numbers to
them. At least on principle - but the reality/situations they're used
in might make that awkward/problematic, I don't know.

Even if they're updated to reflect current paragraph numbers, they'd
still fall into disrepair in future standards & be hard to know which
version they applied to. So I'd support adding C++ version numbers to
them. At least on principle - but the reality/situations they're used
in might make that awkward/problematic, I don't know.

I was thinking about the naming of tests for sections of the C++ standard.
For example the tests for [namespace.udecl] are:
  p1.cpp
  p3.cpp
  p4.cpp
  p5-cxx0x.cpp
  p6-cxx11.cpp
  p7.cpp
  p8.cpp
  p8-cxx0x.cpp
  p10.cpp
  p11.cpp
  p12.cpp
  p13.cpp
  p15.cpp
  p18.cpp

I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of cxxmm"
is reasonable, with the understanding that naming a test p3-cxx17.cpp does not
mean that the corresponding rule is specific to C++17, just that it was written
against C++17.

Bruno

From: cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org] On Behalf Of Bruno
Ricci via cfe-dev
Sent: Tuesday, April 30, 2019 10:34 AM
To: cfe-dev@lists.llvm.org
Subject: Re: [cfe-dev] Paragraph numbers in C++ tests

> Even if they're updated to reflect current paragraph numbers, they'd
> still fall into disrepair in future standards & be hard to know which
> version they applied to. So I'd support adding C++ version numbers to
> them. At least on principle - but the reality/situations they're used
> in might make that awkward/problematic, I don't know.

I was thinking about the naming of tests for sections of the C++ standard.
For example the tests for [namespace.udecl] are:
  p1.cpp
  p3.cpp
  p4.cpp
  p5-cxx0x.cpp
  p6-cxx11.cpp
  p7.cpp
  p8.cpp
  p8-cxx0x.cpp
  p10.cpp
  p11.cpp
  p12.cpp
  p13.cpp
  p15.cpp
  p18.cpp

I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of
cxxmm"
is reasonable, with the understanding that naming a test p3-cxx17.cpp does
not
mean that the corresponding rule is specific to C++17, just that it was
written
against C++17.

Bruno

The real issue is, "p3" in C++11 might be renumbered "p4" in C++14
and "p5" in C++17 and ... so what does "p3.cpp" actually test?
You can't tell just from the name of the file because the name's
meaning is context-sensitive.

Intuitively I'd think "p3-cxx11.cpp" means "tests p3 as it existed in
the C++11 standard" and whether that paragraph is still p3 in later
editions is an exercise for the reader.

I recommend, give up on naming tests after paragraph numbers, because
they aren't actually meaningful.
--paulr

> From: cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org] On Behalf Of Bruno
> Ricci via cfe-dev
> Sent: Tuesday, April 30, 2019 10:34 AM
> To: cfe-dev@lists.llvm.org
> Subject: Re: [cfe-dev] Paragraph numbers in C++ tests
>
>
>
> > Even if they're updated to reflect current paragraph numbers, they'd
> > still fall into disrepair in future standards & be hard to know which
> > version they applied to. So I'd support adding C++ version numbers to
> > them. At least on principle - but the reality/situations they're used
> > in might make that awkward/problematic, I don't know.
>
> I was thinking about the naming of tests for sections of the C++ standard.
> For example the tests for [namespace.udecl] are:
> p1.cpp
> p3.cpp
> p4.cpp
> p5-cxx0x.cpp
> p6-cxx11.cpp
> p7.cpp
> p8.cpp
> p8-cxx0x.cpp
> p10.cpp
> p11.cpp
> p12.cpp
> p13.cpp
> p15.cpp
> p18.cpp
>
> I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of
> cxxmm"
> is reasonable, with the understanding that naming a test p3-cxx17.cpp does
> not
> mean that the corresponding rule is specific to C++17, just that it was
> written
> against C++17.
>
> Bruno

The real issue is, "p3" in C++11 might be renumbered "p4" in C++14
and "p5" in C++17 and ... so what does "p3.cpp" actually test?
You can't tell just from the name of the file because the name's
meaning is context-sensitive.

Intuitively I'd think "p3-cxx11.cpp" means "tests p3 as it existed in
the C++11 standard" and whether that paragraph is still p3 in later
editions is an exercise for the reader.

I think that^ is what Bruno was saying.

I recommend, give up on naming tests after paragraph numbers, because
they aren't actually meaningful.

Eh, I think there's merit to having tests by feature in the standard
and being able to inspect their coverage (yeah, that gets wonky when
paragraphs are renumbered - so you might have to update the numbering
- but the benefit of a correspondence between some part/version of the
standard even if you have to do some work remapping to a consistent
version.

From: David Blaikie [mailto:dblaikie@gmail.com]
Sent: Tuesday, April 30, 2019 12:54 PM
To: Robinson, Paul
Cc: Bruno Ricci; Clang Dev
Subject: Re: [cfe-dev] Paragraph numbers in C++ tests

>
>
>
> > From: cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org] On Behalf Of
Bruno
> > Ricci via cfe-dev
> > Sent: Tuesday, April 30, 2019 10:34 AM
> > To: cfe-dev@lists.llvm.org
> > Subject: Re: [cfe-dev] Paragraph numbers in C++ tests
> >
> >
> >
> > > Even if they're updated to reflect current paragraph numbers, they'd
> > > still fall into disrepair in future standards & be hard to know
which
> > > version they applied to. So I'd support adding C++ version numbers
to
> > > them. At least on principle - but the reality/situations they're
used
> > > in might make that awkward/problematic, I don't know.
> >
> > I was thinking about the naming of tests for sections of the C++
standard.
> > For example the tests for [namespace.udecl] are:
> > p1.cpp
> > p3.cpp
> > p4.cpp
> > p5-cxx0x.cpp
> > p6-cxx11.cpp
> > p7.cpp
> > p8.cpp
> > p8-cxx0x.cpp
> > p10.cpp
> > p11.cpp
> > p12.cpp
> > p13.cpp
> > p15.cpp
> > p18.cpp
> >
> > I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of
> > cxxmm"
> > is reasonable, with the understanding that naming a test p3-cxx17.cpp
does
> > not
> > mean that the corresponding rule is specific to C++17, just that it
was
> > written
> > against C++17.
> >
> > Bruno
>
> The real issue is, "p3" in C++11 might be renumbered "p4" in C++14
> and "p5" in C++17 and ... so what does "p3.cpp" actually test?
> You can't tell just from the name of the file because the name's
> meaning is context-sensitive.
>
> Intuitively I'd think "p3-cxx11.cpp" means "tests p3 as it existed in
> the C++11 standard" and whether that paragraph is still p3 in later
> editions is an exercise for the reader.

I think that^ is what Bruno was saying.

> I recommend, give up on naming tests after paragraph numbers, because
> they aren't actually meaningful.

Eh, I think there's merit to having tests by feature in the standard
and being able to inspect their coverage (yeah, that gets wonky when
paragraphs are renumbered - so you might have to update the numbering
- but the benefit of a correspondence between some part/version of the
standard even if you have to do some work remapping to a consistent
version.

Having contended with a test suite that used this naming scheme, I
beg to disagree; using paragraph numbers was not particularly helpful.
I basically ignored them whenever I had to sort out what a given test
was up to. But it's not my test suite, so I will bow out here.
--paulr

From: cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org] On Behalf Of Bruno
Ricci via cfe-dev
Sent: Tuesday, April 30, 2019 10:34 AM
To: cfe-dev@lists.llvm.org
Subject: Re: [cfe-dev] Paragraph numbers in C++ tests

Even if they're updated to reflect current paragraph numbers, they'd
still fall into disrepair in future standards & be hard to know which
version they applied to. So I'd support adding C++ version numbers to
them. At least on principle - but the reality/situations they're used
in might make that awkward/problematic, I don't know.

I was thinking about the naming of tests for sections of the C++ standard.
For example the tests for [namespace.udecl] are:
  p1.cpp
  p3.cpp
  p4.cpp
  p5-cxx0x.cpp
  p6-cxx11.cpp
  p7.cpp
  p8.cpp
  p8-cxx0x.cpp
  p10.cpp
  p11.cpp
  p12.cpp
  p13.cpp
  p15.cpp
  p18.cpp

I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of
cxxmm"
is reasonable, with the understanding that naming a test p3-cxx17.cpp does
not
mean that the corresponding rule is specific to C++17, just that it was
written
against C++17.

Bruno

The real issue is, "p3" in C++11 might be renumbered "p4" in C++14
and "p5" in C++17 and ... so what does "p3.cpp" actually test?
You can't tell just from the name of the file because the name's
meaning is context-sensitive.

Intuitively I'd think "p3-cxx11.cpp" means "tests p3 as it existed in
the C++11 standard" and whether that paragraph is still p3 in later
editions is an exercise for the reader.

I think that^ is what Bruno was saying.

Yes

I recommend, give up on naming tests after paragraph numbers, because
they aren't actually meaningful.

Eh, I think there's merit to having tests by feature in the standard
and being able to inspect their coverage (yeah, that gets wonky when
paragraphs are renumbered - so you might have to update the numbering
- but the benefit of a correspondence between some part/version of the
standard even if you have to do some work remapping to a consistent
version.

I think I agree with the coverage argument. In any case the person writting
a new test has to consider the previous versions of the standard to see if
the rule was changed.

Bruno

>>
>>
>>
>>> From: cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org] On Behalf Of Bruno
>>> Ricci via cfe-dev
>>> Sent: Tuesday, April 30, 2019 10:34 AM
>>> To: cfe-dev@lists.llvm.org
>>> Subject: Re: [cfe-dev] Paragraph numbers in C++ tests
>>>
>>>
>>>
>>>> Even if they're updated to reflect current paragraph numbers, they'd
>>>> still fall into disrepair in future standards & be hard to know which
>>>> version they applied to. So I'd support adding C++ version numbers to
>>>> them. At least on principle - but the reality/situations they're used
>>>> in might make that awkward/problematic, I don't know.
>>>
>>> I was thinking about the naming of tests for sections of the C++ standard.
>>> For example the tests for [namespace.udecl] are:
>>> p1.cpp
>>> p3.cpp
>>> p4.cpp
>>> p5-cxx0x.cpp
>>> p6-cxx11.cpp
>>> p7.cpp
>>> p8.cpp
>>> p8-cxx0x.cpp
>>> p10.cpp
>>> p11.cpp
>>> p12.cpp
>>> p13.cpp
>>> p15.cpp
>>> p18.cpp
>>>
>>> I think that naming the tests pn-cxxmm.cpp to mean, "paragraph pn of
>>> cxxmm"
>>> is reasonable, with the understanding that naming a test p3-cxx17.cpp does
>>> not
>>> mean that the corresponding rule is specific to C++17, just that it was
>>> written
>>> against C++17.
>>>
>>> Bruno
>>
>> The real issue is, "p3" in C++11 might be renumbered "p4" in C++14
>> and "p5" in C++17 and ... so what does "p3.cpp" actually test?
>> You can't tell just from the name of the file because the name's
>> meaning is context-sensitive.
>>
>> Intuitively I'd think "p3-cxx11.cpp" means "tests p3 as it existed in
>> the C++11 standard" and whether that paragraph is still p3 in later
>> editions is an exercise for the reader.
>
> I think that^ is what Bruno was saying.

Yes

>
>> I recommend, give up on naming tests after paragraph numbers, because
>> they aren't actually meaningful.
>
> Eh, I think there's merit to having tests by feature in the standard
> and being able to inspect their coverage (yeah, that gets wonky when
> paragraphs are renumbered - so you might have to update the numbering
> - but the benefit of a correspondence between some part/version of the
> standard even if you have to do some work remapping to a consistent
> version.
>

I think I agree with the coverage argument. In any case the person writting
a new test has to consider the previous versions of the standard to see if
the rule was changed.

Perhaps a better approach is to use a stable name + a counter for the
test (stable_name_1.cpp, stable_name_2.cpp) and the file can have a
list of comments after the RUN lines:

// C++11 [foo.bar]p2
// C++14 [foo.bar]p3
// whatever

This also helps with coverage if we use a consistent convention across
files, I believe. It also means the file names don't have to be as
meaningful, and we can keep interesting historical information rather
than losing it when changing the name of the file.

~Aaron

The problems here are not restricted to changes in paragraph numbers.
We lay out test/CXX by section structure, and that too has changed a
lot between versions.

Here are some options:

# Exploded-with-symlinks

Approach: rearrange test/CXX to have a subdirectory for each standard
version, each structured as test/CXX is today. When the same test can
be reused across standard versions, symlink it from the old location
to the new one. Teach lit to pass the appropriate -std= flag to
%clang_cc1 invocations in test/CXX/<std version>.
Pro: test layout matches the standard and we can more easily see that
we have coverage of all sections of all supported standards.
Pro: when a new standard is added, can easily check which sections
have been tested and easily add a copy of all current tests that will
be run with the new -std= mode.
Con: when editing a test, it may not be obvious that it's also used by
other language modes.
Con: huge cost, both up-front and when each new standard is added.

# Exploded-with-copies

Approach: as exploded-with-symlinks, but make copies rather than symlinks.
Pro: same as for exploded-with-symlinks.
Pro: easy to remove the C++98 test suite when the day comes to remove
C++98 support.
Con: duplication of tests; more work when adding or updating tests, or
changing diagnostic text; more "real" files in test suite.
Con: huge cost, both up-front and when each new standard is added.

# Flat-by-section

Approach: under test/CXX, have only one level of directories named
after standard section names. Name all new paragraph-numbered tests
with the standard version corresponding to the paragraph.
(Alternatively: concatenate section label and paragraph number and
remove all hierarchy.)
Pro: easier to find the test for a particular paragraph, can avoid duplication
Con: hard to tell which paragraphs of which standard versions are
covered, and by what tests, based on filename

# Flat-unumbered-by-section

Approach: as flat-by-section, but name tests after the functionality
they're testing rather than paragraph numbers (alternatively: just
number the tests arbitrarily rather than trying to pick meaningful
names).
Pro: no paragraph number confusion
Con: very hard to tell which paragraphs of which standard versions are
covered, and by what tests, based on filenames

# Status quo

Approach: sections arranged in directory hierarchy roughly by the
earliest standard version to contain them, file names numbered with
paragraph numbers of roughly the earliest standard version to contain
those paragraphs. Use standard version suffixes only to avoid file
name collisions.
Pro: zero up-front cost
Con: organization is outdated by ongoing standard changes

# Match the latest standard

Approach: sections arranged by directory hierarchy attempting to match
the current working draft, file names numbered with paragraph numbers
attempting to match the latest working draft. Use version suffixes for
tests that don't correspond to anything in the latest standard.
Pro: matching tests against the current specification is simple; can
more easily determine coverage for latest specification
Con: ongoing maintenance cost keeping up with standardization
Con: unclear how to organize and find tests for sections of the
standard that have been removed

Largely orthogonally to the above, we have Aaron's suggestion to
enable coverage analysis: include comments in each test file in a
parseable format indicating each standard version, section label, and
paragraph covered by that test.

Goals:
* obvious organizational structure: no time spent thinking about
where to put a new test
* easy to find existing tests
* feasible to build coverage checking tool to determine how much of
which standards we test for
* low up-front and ongoing maintenance cost

Thoughts? Perhaps flat-by-section + Aaron's suggestion is the best
tradeoff here? (That is: remove all hierarchy from the test tree other
than the leaf directories, add a -c++XY suffix to all new test names,
and include machine-readable comments in test files indicating which
paragraphs of which standards are covered.)

The problems here are not restricted to changes in paragraph numbers.
We lay out test/CXX by section structure, and that too has changed a
lot between versions.

Here are some options:

# Exploded-with-symlinks

Approach: rearrange test/CXX to have a subdirectory for each standard
version, each structured as test/CXX is today. When the same test can
be reused across standard versions, symlink it from the old location
to the new one. Teach lit to pass the appropriate -std= flag to
%clang_cc1 invocations in test/CXX/<std version>.
Pro: test layout matches the standard and we can more easily see that
we have coverage of all sections of all supported standards.
Pro: when a new standard is added, can easily check which sections
have been tested and easily add a copy of all current tests that will
be run with the new -std= mode.
Con: when editing a test, it may not be obvious that it's also used by
other language modes.
Con: huge cost, both up-front and when each new standard is added.

# Exploded-with-copies

Approach: as exploded-with-symlinks, but make copies rather than symlinks.
Pro: same as for exploded-with-symlinks.
Pro: easy to remove the C++98 test suite when the day comes to remove
C++98 support.
Con: duplication of tests; more work when adding or updating tests, or
changing diagnostic text; more "real" files in test suite.
Con: huge cost, both up-front and when each new standard is added.

# Flat-by-section

Approach: under test/CXX, have only one level of directories named
after standard section names. Name all new paragraph-numbered tests
with the standard version corresponding to the paragraph.
(Alternatively: concatenate section label and paragraph number and
remove all hierarchy.)
Pro: easier to find the test for a particular paragraph, can avoid duplication
Con: hard to tell which paragraphs of which standard versions are
covered, and by what tests, based on filename

# Flat-unumbered-by-section

Approach: as flat-by-section, but name tests after the functionality
they're testing rather than paragraph numbers (alternatively: just
number the tests arbitrarily rather than trying to pick meaningful
names).
Pro: no paragraph number confusion
Con: very hard to tell which paragraphs of which standard versions are
covered, and by what tests, based on filenames

# Status quo

Approach: sections arranged in directory hierarchy roughly by the
earliest standard version to contain them, file names numbered with
paragraph numbers of roughly the earliest standard version to contain
those paragraphs. Use standard version suffixes only to avoid file
name collisions.
Pro: zero up-front cost
Con: organization is outdated by ongoing standard changes

# Match the latest standard

Approach: sections arranged by directory hierarchy attempting to match
the current working draft, file names numbered with paragraph numbers
attempting to match the latest working draft. Use version suffixes for
tests that don't correspond to anything in the latest standard.
Pro: matching tests against the current specification is simple; can
more easily determine coverage for latest specification
Con: ongoing maintenance cost keeping up with standardization
Con: unclear how to organize and find tests for sections of the
standard that have been removed

Largely orthogonally to the above, we have Aaron's suggestion to
enable coverage analysis: include comments in each test file in a
parseable format indicating each standard version, section label, and
paragraph covered by that test.

Goals:
* obvious organizational structure: no time spent thinking about
where to put a new test
* easy to find existing tests
* feasible to build coverage checking tool to determine how much of
which standards we test for
* low up-front and ongoing maintenance cost

Thoughts? Perhaps flat-by-section + Aaron's suggestion is the best
tradeoff here? (That is: remove all hierarchy from the test tree other
than the leaf directories, add a -c++XY suffix to all new test names,
and include machine-readable comments in test files indicating which
paragraphs of which standards are covered.)

+1, I think this is a reasonable approach, though I do wonder if we
will want some extra hierarchy for really large sections like [expr]
or [dcl.dcl] or not.

~Aaron

Exploded-with-symlinks and Exploded-with-copies are probably way too costly
to maintain. Would the Flat-by-section model imply that if a section was
renamed from X to Y, then some tests would be in X/ and some tests in Y/ ?
Additionally what should be done if a test can be written against multiple
versions ? (Pick the oldest version/the latest released version/the latest
draft)

Bruno