Testing "normal" cross-compilers versus GPU backends

This note arose from ⚙ D12506 Distinguish "native" from "native-arch" in lit testing but the reviewers
felt that we needed a broader audience, because the proposed patch
really didn't solve the entire problem and we had no better ideas.

Mehdi Amini needs to build LLVM with just a GPU backend, and still have
"ninja check" Just Work. Commits r243958-243960 tried to accomplish
that; however they are too big a hammer, and cause much simpler cross
environments (like mine) to exclude big chunks of very useful tests
(including my favorite, DebugInfo).

FYI, my main cross environment is building on X86 Windows but using
a default target triple for PS4 (which is also X86).

I experimented with building LLVM with just the ARM backend (running on
an X86 workstation) and setting the default triple to some ARM value.
"ninja check" worked fine (without Mehdi's series of commits), so the
normal kind of cross-compiler environment seems to be happy with how
things were set up originally.

Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
the default triple to "amdgcn--amdhsa", and getting 200-some failures.

(This does make me wonder about AMDGPU testing in general; how does that
work? The only places I see lit checks for AMDGPU are in the usual
target-dependent places.)

Mehdi's solution was:
- In lit.cfg, change the existing "native" feature definition from
  "host-triple == target-triple" to also check "and the corresponding
  backend is included."(**)
- Make piles of tests that seemed inapplicable to GPUs depend on the
  "native" feature (through REQUIRES: or in the lit.local.cfg).
- Build LLVM with just the GPU backend, and not set a target triple
  (i.e., it's set to the host triple, typically an X86-something).(*)
  Thus lit.cfg sees matching host and target triples, but the X86
  backend is missing, and so the "native" feature is not set.

[The "native" feature was invented to identify environments where
JIT would work properly. The "host-triple == target-triple" condition
isn't exactly right, but it works well enough.]

The major problem is that these new "native" dependencies are incorrect.
For example the DebugInfo tests don't really require it; they work fine
as long as the default triple has the corresponding backend included,
as my ARM-on-X86 experiment demonstrated.

So, we need some solution that Mehdi can use in a GPU-backend-testing
workflow, but that does not so drastically interfere with normal
cross-compiler testing. Does anybody have any suggestions?

Thanks,
--paulr

(*) This configuration seems inherently wrong; the default target triple
ought to require having the corresponding backend included, I think.
(**) Which makes this modification superfluous.

Hi Paul,

Thanks for the summary!

This note arose from https://urldefense.proofpoint.com/v2/url?u=http-3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8eo6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but the reviewers
felt that we needed a broader audience, because the proposed patch
really didn't solve the entire problem and we had no better ideas.

Mehdi Amini needs to build LLVM with just a GPU backend, and still have
"ninja check" Just Work. Commits r243958-243960 tried to accomplish
that; however they are too big a hammer, and cause much simpler cross
environments (like mine) to exclude big chunks of very useful tests
(including my favorite, DebugInfo).

FYI, my main cross environment is building on X86 Windows but using
a default target triple for PS4 (which is also X86).

I experimented with building LLVM with just the ARM backend (running on
an X86 workstation) and setting the default triple to some ARM value.
"ninja check" worked fine (without Mehdi's series of commits), so the
normal kind of cross-compiler environment seems to be happy with how
things were set up originally.

Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
the default triple to "amdgcn--amdhsa", and getting 200-some failures.

(This does make me wonder about AMDGPU testing in general; how does that
work? The only places I see lit checks for AMDGPU are in the usual
target-dependent places.)

I don’t understand this interrogation about how do you do testing in general. The same way you don’t process tests/CodeGen/X86/* with the ARM backend, you can’t process any random IR through these backends.

IMO, the problem is in general about tests that are written without specifying a triple, that will be executed with the default triple.

Most of these tests were written with X86 (or ARM) in mind, and there is no guarantee that they will behave as intended with every possible triple.
The DataLayout for instance has to be the one from the target, and is not portable.
I think a "portable backend test” is pretty rare in general.

You can run all the tests by setting the default triple to X86 and compiling in the X86 backend, but that’s just a trick to make the tests happy.
Alternatively, and this is what I tried to do, blacklisting these tests that “pretends” to be “portable” but are not.

Mehdi's solution was:
- In lit.cfg, change the existing "native" feature definition from
"host-triple == target-triple" to also check "and the corresponding
backend is included."(**)

I agree that we could remove the condition on the host backend included and deemed it a unsupported, but the build system needs to reject this configuration.

- Make piles of tests that seemed inapplicable to GPUs depend on the
"native" feature (through REQUIRES: or in the lit.local.cfg).

Nitpick: the GPU is just an example, any other backends can be affected.
It seems that these test are “lying” about the target they will be able to run on (like if they would run on “anything”).

- Build LLVM with just the GPU backend, and not set a target triple
(i.e., it's set to the host triple, typically an X86-something).(*)
Thus lit.cfg sees matching host and target triples, but the X86
backend is missing, and so the "native" feature is not set.

[The "native" feature was invented to identify environments where
JIT would work properly. The "host-triple == target-triple" condition
isn't exactly right, but it works well enough.]

The major problem is that these new "native" dependencies are incorrect.
For example the DebugInfo tests don't really require it; they work fine
as long as the default triple has the corresponding backend included,
as my ARM-on-X86 experiment demonstrated.

Are they are guarantee’d to work with a default triple set to any of the possible (in-tree) backend?
(I don’t know enough about these tests, which is definitively why I included them in the “big hammer” solution)

Thanks,

From: Mehdi Amini [mailto:mehdi.amini@apple.com]
Sent: Wednesday, September 02, 2015 7:10 PM
To: Robinson, Paul
Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

Hi Paul,

Thanks for the summary!

>
> This note arose from Proofpoint Targeted Attack Protection
3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but the
reviewers
> felt that we needed a broader audience, because the proposed patch
> really didn't solve the entire problem and we had no better ideas.
>
> Mehdi Amini needs to build LLVM with just a GPU backend, and still have
> "ninja check" Just Work. Commits r243958-243960 tried to accomplish
> that; however they are too big a hammer, and cause much simpler cross
> environments (like mine) to exclude big chunks of very useful tests
> (including my favorite, DebugInfo).
>
> FYI, my main cross environment is building on X86 Windows but using
> a default target triple for PS4 (which is also X86).
>
> I experimented with building LLVM with just the ARM backend (running on
> an X86 workstation) and setting the default triple to some ARM value.
> "ninja check" worked fine (without Mehdi's series of commits), so the
> normal kind of cross-compiler environment seems to be happy with how
> things were set up originally.
>
> Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
> the default triple to "amdgcn--amdhsa", and getting 200-some failures.
>
> (This does make me wonder about AMDGPU testing in general; how does that
> work? The only places I see lit checks for AMDGPU are in the usual
> target-dependent places.)

I don’t understand this interrogation about how do you do testing in
general. The same way you don’t process tests/CodeGen/X86/* with the ARM
backend, you can’t process any random IR through these backends.

You said you had 200+ failures with AMDGPU. Are the AMD folks simply
tolerating the 200 failures, and you don't want to? I should hope there
is more to it than that.

IMO, the problem is in general about tests that are written without
specifying a triple, that will be executed with the default triple.

Most of these tests were written with X86 (or ARM) in mind, and there is
no guarantee that they will behave as intended with every possible triple.
The DataLayout for instance has to be the one from the target, and is not
portable.
I think a "portable backend test” is pretty rare in general.

It depends on what the test is trying to do. I'm sure it is quite common
for IR tests to behave essentially the same way regardless of target.
We have lots of tests (the ones you chose to mark "native") that had been
working fine with ARM, X86, PPC, SPARC, and whatever.

The fact that they don't work with your backend is different from saying
those tests can't possibly work for any cross-compiler. But the latter
is what your patch implemented, and it is preventing useful testing.
--paulr

From: Mehdi Amini [mailto:mehdi.amini@apple.com]
Sent: Wednesday, September 02, 2015 7:10 PM
To: Robinson, Paul
Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

Hi Paul,

Thanks for the summary!

This note arose from Proofpoint Targeted Attack Protection

3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but the
reviewers

felt that we needed a broader audience, because the proposed patch
really didn't solve the entire problem and we had no better ideas.

Mehdi Amini needs to build LLVM with just a GPU backend, and still have
"ninja check" Just Work. Commits r243958-243960 tried to accomplish
that; however they are too big a hammer, and cause much simpler cross
environments (like mine) to exclude big chunks of very useful tests
(including my favorite, DebugInfo).

FYI, my main cross environment is building on X86 Windows but using
a default target triple for PS4 (which is also X86).

I experimented with building LLVM with just the ARM backend (running on
an X86 workstation) and setting the default triple to some ARM value.
"ninja check" worked fine (without Mehdi's series of commits), so the
normal kind of cross-compiler environment seems to be happy with how
things were set up originally.

Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
the default triple to "amdgcn--amdhsa", and getting 200-some failures.

(This does make me wonder about AMDGPU testing in general; how does that
work? The only places I see lit checks for AMDGPU are in the usual
target-dependent places.)

I don’t understand this interrogation about how do you do testing in
general. The same way you don’t process tests/CodeGen/X86/* with the ARM
backend, you can’t process any random IR through these backends.

You said you had 200+ failures with AMDGPU. Are the AMD folks simply
tolerating the 200 failures, and you don't want to? I should hope there
is more to it than that.

Well, I don’t know, they might just run `ninja check` with the default triple set to X86?
(which I would consider being working around a buggy test suite)

IMO, the problem is in general about tests that are written without
specifying a triple, that will be executed with the default triple.

Most of these tests were written with X86 (or ARM) in mind, and there is
no guarantee that they will behave as intended with every possible triple.
The DataLayout for instance has to be the one from the target, and is not
portable.
I think a "portable backend test” is pretty rare in general.

It depends on what the test is trying to do. I'm sure it is quite common
for IR tests to behave essentially the same way regardless of target.

IR tests != backend test (I may miss your point here, it’s late…).

We have lots of tests (the ones you chose to mark "native") that had been
working fine with ARM, X86, PPC, SPARC, and whatever.

The fact that they don't work with your backend is different from saying
those tests can't possibly work for any cross-compiler.

I believe this is far from what I said, or at least from what I had in mind.

But the latter is what your patch implemented, and it is preventing useful testing.

I don’t disagree with that statement, and I agree that it should be fixed.
It doesn’t mean that I think the previous situation was better though.
The current over-conservative state seems at least more correct than having only a subset of the targets that can pass `ninja check`.

Are you attending the LLVM bay area social tonight by any chance?

Best,

The Hexagon compiler (that we ship) only comes as a cross-compiler, and we don't include any other targets in it. We set the default target triple to hexagon-..., and the "make check" works. (I think we may have had to XFAIL some tests, but it was not widespread.)

The problem in the AMDGPU case seems to be that there are some features that the backed does not support (will not support?), and any test that relies on them will fail. In theory, each target could have a corresponding set of limitations, and that set could be different for each such target. The theoretical solution would then be to identify the set of all features and annotate each test (that does not specify the target triple) to list all the required ones. Each target could mark some features as "unsupported" and then only the fully supported tests would run. In practice this is a lot of work and I have no idea if this problem is common enough to warrant it.

If this is specific to AMDGPU, then maybe the solution could also be specific? For example, the tests that are not expected to work on that target could just be XFAILed with a comment explaining it. It's ugly, but it's quick.

-Krzysztof

I'm not very familiar with the "native" feature, so I'm not sure what the impact is there, but it seems like setting the default target triple to something that does not exist could be a way for a target to require that the proper triple is provided explicitly. This way any test that does not explicitly contain a matching triple would be ignored on that target.
This would still exclude large number of potentially useful tests, but maybe it's a limitation of the testing that the target would be willing to tolerate?

-Krzysztof

>
>
>
>> From: Mehdi Amini [mailto:mehdi.amini@apple.com]
>> Sent: Wednesday, September 02, 2015 7:10 PM
>> To: Robinson, Paul
>> Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
>> Subject: Re: Testing "normal" cross-compilers versus GPU backends
>>
>> Hi Paul,
>>
>> Thanks for the summary!
>>
>>>
>>> This note arose from Proofpoint Targeted Attack Protection
>> 3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>> ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
>> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but the
>> reviewers
>>> felt that we needed a broader audience, because the proposed patch
>>> really didn't solve the entire problem and we had no better ideas.
>>>
>>> Mehdi Amini needs to build LLVM with just a GPU backend, and still have
>>> "ninja check" Just Work. Commits r243958-243960 tried to accomplish
>>> that; however they are too big a hammer, and cause much simpler cross
>>> environments (like mine) to exclude big chunks of very useful tests
>>> (including my favorite, DebugInfo).
>>>
>>> FYI, my main cross environment is building on X86 Windows but using
>>> a default target triple for PS4 (which is also X86).
>>>
>>> I experimented with building LLVM with just the ARM backend (running on
>>> an X86 workstation) and setting the default triple to some ARM value.
>>> "ninja check" worked fine (without Mehdi's series of commits), so the
>>> normal kind of cross-compiler environment seems to be happy with how
>>> things were set up originally.
>>>
>>> Mehdi reports building LLVM with the X86 and AMDGPU backends, setting
>>> the default triple to "amdgcn--amdhsa", and getting 200-some failures.
>>>
>>> (This does make me wonder about AMDGPU testing in general; how does that
>>> work? The only places I see lit checks for AMDGPU are in the usual
>>> target-dependent places.)
>>
>> I don’t understand this interrogation about how do you do testing in
>> general. The same way you don’t process tests/CodeGen/X86/* with the ARM
>> backend, you can’t process any random IR through these backends.
>
> You said you had 200+ failures with AMDGPU. Are the AMD folks simply
> tolerating the 200 failures, and you don't want to? I should hope there
> is more to it than that.

Well, I don’t know, they might just run `ninja check` with the default triple set to X86?
(which I would consider being working around a buggy test suite)

I always enable AMDGPU and X86 when I build, so I've never run into
this problem.

-Tom

From: Tom Stellard [mailto:tom@stellard.net]
Sent: Thursday, September 03, 2015 7:31 AM
To: Mehdi Amini
Cc: Robinson, Paul; llvm-dev@lists.llvm.org; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

>
> >
> >
> >
> >> From: Mehdi Amini [mailto:mehdi.amini@apple.com]
> >> Sent: Wednesday, September 02, 2015 7:10 PM
> >> To: Robinson, Paul
> >> Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
> >> Subject: Re: Testing "normal" cross-compilers versus GPU backends
> >>
> >> Hi Paul,
> >>
> >> Thanks for the summary!
> >>
> >>>
> >>> This note arose from
Proofpoint Targeted Attack Protection
> >> 3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> >>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> >> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but
the
> >> reviewers
> >>> felt that we needed a broader audience, because the proposed patch
> >>> really didn't solve the entire problem and we had no better ideas.
> >>>
> >>> Mehdi Amini needs to build LLVM with just a GPU backend, and still
have
> >>> "ninja check" Just Work. Commits r243958-243960 tried to accomplish
> >>> that; however they are too big a hammer, and cause much simpler
cross
> >>> environments (like mine) to exclude big chunks of very useful tests
> >>> (including my favorite, DebugInfo).
> >>>
> >>> FYI, my main cross environment is building on X86 Windows but using
> >>> a default target triple for PS4 (which is also X86).
> >>>
> >>> I experimented with building LLVM with just the ARM backend (running
on
> >>> an X86 workstation) and setting the default triple to some ARM
value.
> >>> "ninja check" worked fine (without Mehdi's series of commits), so
the
> >>> normal kind of cross-compiler environment seems to be happy with how
> >>> things were set up originally.
> >>>
> >>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
setting
> >>> the default triple to "amdgcn--amdhsa", and getting 200-some
failures.
> >>>
> >>> (This does make me wonder about AMDGPU testing in general; how does
that
> >>> work? The only places I see lit checks for AMDGPU are in the usual
> >>> target-dependent places.)
> >>
> >> I don’t understand this interrogation about how do you do testing in
> >> general. The same way you don’t process tests/CodeGen/X86/* with the
ARM
> >> backend, you can’t process any random IR through these backends.
> >
> > You said you had 200+ failures with AMDGPU. Are the AMD folks simply
> > tolerating the 200 failures, and you don't want to? I should hope
there
> > is more to it than that.
>
> Well, I don’t know, they might just run `ninja check` with the default
triple set to X86?
> (which I would consider being working around a buggy test suite)
>

I always enable AMDGPU and X86 when I build, so I've never run into
this problem.

-Tom

Tom, presumably your default target triple is X86-ish? And the only
tests to exercise the AMDGPU backend are those that explicitly specify
a triple for AMDGPU?

Mehdi, assuming that's what Tom does, your stated goal was to be able to
run tests *without* including the X86 backend, so Tom's solution won't
work for you (restating just for confirmation).

Krzysztof suggested much the same thing that I think you are currently
doing, which is deliberately configure a default triple but exclude the
corresponding backend. I expect we can detect that situation in lit.cfg
and exclude tests on that basis, rather than 'native'. It would solve
the problem for my case (host triple != target triple, although the arch
parts of the triple do match) and the "normal" cross-compiler case (e.g.
host = X86, backend + target triple = ARM).

I'm going to play around with that and see what I can do to make it work.

> >
> >>
> >> IMO, the problem is in general about tests that are written without
> >> specifying a triple, that will be executed with the default triple.
> >>
> >> Most of these tests were written with X86 (or ARM) in mind, and there
is
> >> no guarantee that they will behave as intended with every possible
triple.
> >> The DataLayout for instance has to be the one from the target, and is
not
> >> portable.
> >> I think a "portable backend test” is pretty rare in general.
> >
> > It depends on what the test is trying to do. I'm sure it is quite
common
> > for IR tests to behave essentially the same way regardless of target.
>
> IR tests != backend test (I may miss your point here, it’s late…).

Right, sorry, lost focus for a moment there... nevertheless it is still
the case that many tests exercise functionality that is not particularly
target-centric and these should be run for any target that actually
supports that functionality. For example, the DebugInfo tests should
be run for any target that supports emitting debug info. Whether a
target supports debug info is orthogonal to its native-ness. As written
the top-level DebugInfo tests should work fine as long as the default
triple's backend is included, and that backend supports debug info.

If your backend doesn't support debug info, then it's reasonable to
exclude those from your testing; but we can't do that at the cost of
excluding those from testing other backends that *do* support the feature,
even if that testing runs in a cross-compiler environment.

In this particular example, we'd be setting things up so that DebugInfo
is excluded for the wrong reason (not based on some more abstract idea
of the feature-set of the target) but as Krzysztof observes, getting
a feature-oriented set of conditions would be a huge task.

>
>
> > We have lots of tests (the ones you chose to mark "native") that had
been
> > working fine with ARM, X86, PPC, SPARC, and whatever.
> >
> > The fact that they don't work with your backend is different from
saying
> > those tests can't possibly work for any cross-compiler.
>
> I believe this is far from what I said, or at least from what I had in
mind.
>
> > But the latter is what your patch implemented, and it is preventing
useful testing.
>
> I don’t disagree with that statement, and I agree that it should be
fixed.
> It doesn’t mean that I think the previous situation was better though.
> The current over-conservative state seems at least more correct than
having only a subset of the targets that can pass `ninja check`.
>
> Are you attending the LLVM bay area social tonight by any chance?

Unfortunately no, I have a prior commitment.
--paulr

> From: Tom Stellard [mailto:tom@stellard.net]
> Sent: Thursday, September 03, 2015 7:31 AM
> To: Mehdi Amini
> Cc: Robinson, Paul; llvm-dev@lists.llvm.org; NAKAMURA Takumi
> Subject: Re: Testing "normal" cross-compilers versus GPU backends
>
> >
> > >
> > >
> > >
> > >> From: Mehdi Amini [mailto:mehdi.amini@apple.com]
> > >> Sent: Wednesday, September 02, 2015 7:10 PM
> > >> To: Robinson, Paul
> > >> Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
> > >> Subject: Re: Testing "normal" cross-compilers versus GPU backends
> > >>
> > >> Hi Paul,
> > >>
> > >> Thanks for the summary!
> > >>
> > >>>
> > >>> This note arose from
> Proofpoint Targeted Attack Protection
> > >> 3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> > >>
> ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> > >> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but
> the
> > >> reviewers
> > >>> felt that we needed a broader audience, because the proposed patch
> > >>> really didn't solve the entire problem and we had no better ideas.
> > >>>
> > >>> Mehdi Amini needs to build LLVM with just a GPU backend, and still
> have
> > >>> "ninja check" Just Work. Commits r243958-243960 tried to accomplish
> > >>> that; however they are too big a hammer, and cause much simpler
> cross
> > >>> environments (like mine) to exclude big chunks of very useful tests
> > >>> (including my favorite, DebugInfo).
> > >>>
> > >>> FYI, my main cross environment is building on X86 Windows but using
> > >>> a default target triple for PS4 (which is also X86).
> > >>>
> > >>> I experimented with building LLVM with just the ARM backend (running
> on
> > >>> an X86 workstation) and setting the default triple to some ARM
> value.
> > >>> "ninja check" worked fine (without Mehdi's series of commits), so
> the
> > >>> normal kind of cross-compiler environment seems to be happy with how
> > >>> things were set up originally.
> > >>>
> > >>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
> setting
> > >>> the default triple to "amdgcn--amdhsa", and getting 200-some
> failures.
> > >>>
> > >>> (This does make me wonder about AMDGPU testing in general; how does
> that
> > >>> work? The only places I see lit checks for AMDGPU are in the usual
> > >>> target-dependent places.)
> > >>
> > >> I don’t understand this interrogation about how do you do testing in
> > >> general. The same way you don’t process tests/CodeGen/X86/* with the
> ARM
> > >> backend, you can’t process any random IR through these backends.
> > >
> > > You said you had 200+ failures with AMDGPU. Are the AMD folks simply
> > > tolerating the 200 failures, and you don't want to? I should hope
> there
> > > is more to it than that.
> >
> > Well, I don’t know, they might just run `ninja check` with the default
> triple set to X86?
> > (which I would consider being working around a buggy test suite)
> >
>
> I always enable AMDGPU and X86 when I build, so I've never run into
> this problem.
>
> -Tom

Tom, presumably your default target triple is X86-ish? And the only
tests to exercise the AMDGPU backend are those that explicitly specify
a triple for AMDGPU?

Correct, the default triple is x86-ish, and all the AMDGPU backend tests
specify a triple.

-Tom

From: Tom Stellard [mailto:tom@stellard.net]
Sent: Thursday, September 03, 2015 7:31 AM
To: Mehdi Amini
Cc: Robinson, Paul; llvm-dev@lists.llvm.org; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

From: Mehdi Amini [mailto:mehdi.amini@apple.com]
Sent: Wednesday, September 02, 2015 7:10 PM
To: Robinson, Paul
Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

Hi Paul,

Thanks for the summary!

This note arose from

Proofpoint Targeted Attack Protection

3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-

ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e

o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but

the

reviewers

felt that we needed a broader audience, because the proposed patch
really didn't solve the entire problem and we had no better ideas.

Mehdi Amini needs to build LLVM with just a GPU backend, and still

have

"ninja check" Just Work. Commits r243958-243960 tried to accomplish
that; however they are too big a hammer, and cause much simpler

cross

environments (like mine) to exclude big chunks of very useful tests
(including my favorite, DebugInfo).

FYI, my main cross environment is building on X86 Windows but using
a default target triple for PS4 (which is also X86).

I experimented with building LLVM with just the ARM backend (running

on

an X86 workstation) and setting the default triple to some ARM

value.

"ninja check" worked fine (without Mehdi's series of commits), so

the

normal kind of cross-compiler environment seems to be happy with how
things were set up originally.

Mehdi reports building LLVM with the X86 and AMDGPU backends,

setting

the default triple to "amdgcn--amdhsa", and getting 200-some

failures.

(This does make me wonder about AMDGPU testing in general; how does

that

work? The only places I see lit checks for AMDGPU are in the usual
target-dependent places.)

I don’t understand this interrogation about how do you do testing in
general. The same way you don’t process tests/CodeGen/X86/* with the

ARM

backend, you can’t process any random IR through these backends.

You said you had 200+ failures with AMDGPU. Are the AMD folks simply
tolerating the 200 failures, and you don't want to? I should hope

there

is more to it than that.

Well, I don’t know, they might just run `ninja check` with the default

triple set to X86?

(which I would consider being working around a buggy test suite)

I always enable AMDGPU and X86 when I build, so I've never run into
this problem.

-Tom

Tom, presumably your default target triple is X86-ish? And the only
tests to exercise the AMDGPU backend are those that explicitly specify
a triple for AMDGPU?

This is how we used to do it as well, and I assume this is how most of backends out of the main CPUs are dealing with the situation.
This is what I consider a bug and was trying to solve (admittedly not in an optimal way).

Mehdi, assuming that's what Tom does, your stated goal was to be able to
run tests *without* including the X86 backend, so Tom's solution won't
work for you (restating just for confirmation).

Yes, or alternatively I expect the test suite to pass whatever the default triple is.

Krzysztof suggested much the same thing that I think you are currently
doing, which is deliberately configure a default triple but exclude the
corresponding backend.

You and Takumi were considering this as an unsupported configuration before, and I tend to agree with that (this is the configuration I’m using for our tests but it was not intentional to leave the default triple unset).

I expect we can detect that situation in lit.cfg
and exclude tests on that basis, rather than 'native'. It would solve
the problem for my case (host triple != target triple, although the arch
parts of the triple do match) and the "normal" cross-compiler case (e.g.
host = X86, backend + target triple = ARM).

I'm going to play around with that and see what I can do to make it work.

IMO, the problem is in general about tests that are written without
specifying a triple, that will be executed with the default triple.

Most of these tests were written with X86 (or ARM) in mind, and there

is

no guarantee that they will behave as intended with every possible

triple.

The DataLayout for instance has to be the one from the target, and is

not

portable.
I think a "portable backend test” is pretty rare in general.

It depends on what the test is trying to do. I'm sure it is quite

common

for IR tests to behave essentially the same way regardless of target.

IR tests != backend test (I may miss your point here, it’s late…).

Right, sorry, lost focus for a moment there... nevertheless it is still
the case that many tests exercise functionality that is not particularly
target-centric and these should be run for any target that actually
supports that functionality. For example, the DebugInfo tests should
be run for any target that supports emitting debug info.

I’m not sure that “debug info” support is all or nothing.
As an extreme example, I know targets that support debug info but do not support function calls, what if your "debug info” test involves these?

Also, I’m not a DebugInfo expert, but when a front-end generated them, aren’t they dependent on the DataLayout? Hence the target?
It means that many DebugInfo test could fail with a backend that would have a significantly different DataLayout and won’t expect some stuff the way they are played down.

Whether a
target supports debug info is orthogonal to its native-ness. As written
the top-level DebugInfo tests should work fine as long as the default
triple's backend is included, and that backend supports debug info.

If your backend doesn't support debug info, then it's reasonable to
exclude those from your testing; but we can't do that at the cost of
excluding those from testing other backends that *do* support the feature,
even if that testing runs in a cross-compiler environment.

In this particular example, we'd be setting things up so that DebugInfo
is excluded for the wrong reason (not based on some more abstract idea
of the feature-set of the target) but as Krzysztof observes, getting
a feature-oriented set of conditions would be a huge task.

Agree: the predicate I wrote is not correct, and I don’t see a trivial “good” solution. This is why any compromise or intermediate solution that is good enough to support whatever use-case people here have, including your, seems appropriate to implement.

Let me know I can help coding something.

From: Mehdi Amini [mailto:mehdi.amini@apple.com]
Sent: Thursday, September 03, 2015 3:26 PM
To: Robinson, Paul
Cc: Tom Stellard; llvm-dev@lists.llvm.org; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

>
>
>
>> From: Tom Stellard [mailto:tom@stellard.net]
>> Sent: Thursday, September 03, 2015 7:31 AM
>> To: Mehdi Amini
>> Cc: Robinson, Paul; llvm-dev@lists.llvm.org; NAKAMURA Takumi
>> Subject: Re: Testing "normal" cross-compilers versus GPU backends
>>
>>>
>>>>
>>>>
>>>>
>>>>> From: Mehdi Amini [mailto:mehdi.amini@apple.com]
>>>>> Sent: Wednesday, September 02, 2015 7:10 PM
>>>>> To: Robinson, Paul
>>>>> Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
>>>>> Subject: Re: Testing "normal" cross-compilers versus GPU backends
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Thanks for the summary!
>>>>>
>>>>>>
>>>>>> This note arose from
>> Proofpoint Targeted Attack Protection
>>>>> 3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>>>>>
>>
ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
>>>>> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but
>> the
>>>>> reviewers
>>>>>> felt that we needed a broader audience, because the proposed patch
>>>>>> really didn't solve the entire problem and we had no better ideas.
>>>>>>
>>>>>> Mehdi Amini needs to build LLVM with just a GPU backend, and still
>> have
>>>>>> "ninja check" Just Work. Commits r243958-243960 tried to
accomplish
>>>>>> that; however they are too big a hammer, and cause much simpler
>> cross
>>>>>> environments (like mine) to exclude big chunks of very useful tests
>>>>>> (including my favorite, DebugInfo).
>>>>>>
>>>>>> FYI, my main cross environment is building on X86 Windows but using
>>>>>> a default target triple for PS4 (which is also X86).
>>>>>>
>>>>>> I experimented with building LLVM with just the ARM backend
(running
>> on
>>>>>> an X86 workstation) and setting the default triple to some ARM
>> value.
>>>>>> "ninja check" worked fine (without Mehdi's series of commits), so
>> the
>>>>>> normal kind of cross-compiler environment seems to be happy with
how
>>>>>> things were set up originally.
>>>>>>
>>>>>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
>> setting
>>>>>> the default triple to "amdgcn--amdhsa", and getting 200-some
>> failures.
>>>>>>
>>>>>> (This does make me wonder about AMDGPU testing in general; how does
>> that
>>>>>> work? The only places I see lit checks for AMDGPU are in the usual
>>>>>> target-dependent places.)
>>>>>
>>>>> I don’t understand this interrogation about how do you do testing in
>>>>> general. The same way you don’t process tests/CodeGen/X86/* with the
>> ARM
>>>>> backend, you can’t process any random IR through these backends.
>>>>
>>>> You said you had 200+ failures with AMDGPU. Are the AMD folks simply
>>>> tolerating the 200 failures, and you don't want to? I should hope
>> there
>>>> is more to it than that.
>>>
>>> Well, I don’t know, they might just run `ninja check` with the default
>> triple set to X86?
>>> (which I would consider being working around a buggy test suite)
>>>
>>
>> I always enable AMDGPU and X86 when I build, so I've never run into
>> this problem.
>>
>> -Tom
>
> Tom, presumably your default target triple is X86-ish? And the only
> tests to exercise the AMDGPU backend are those that explicitly specify
> a triple for AMDGPU?

This is how we used to do it as well, and I assume this is how most of
backends out of the main CPUs are dealing with the situation.
This is what I consider a bug and was trying to solve (admittedly not in
an optimal way).

> Mehdi, assuming that's what Tom does, your stated goal was to be able to
> run tests *without* including the X86 backend, so Tom's solution won't
> work for you (restating just for confirmation).

Yes, or alternatively I expect the test suite to pass whatever the default
triple is.

Um, that may be asking a bit much...

> Krzysztof suggested much the same thing that I think you are currently
> doing, which is deliberately configure a default triple but exclude the
> corresponding backend.

You and Takumi were considering this as an unsupported configuration
before, and I tend to agree with that (this is the configuration I’m using
for our tests but it was not intentional to leave the default triple
unset).

Right, intuitively it doesn't make sense. Is it actually useful to build a
GPU compiler that will crash unless you ask it to generate GPU code? Seems
to me it should default to producing GPU code.

> I expect we can detect that situation in lit.cfg
> and exclude tests on that basis, rather than 'native'. It would solve
> the problem for my case (host triple != target triple, although the arch
> parts of the triple do match) and the "normal" cross-compiler case (e.g.
> host = X86, backend + target triple = ARM).
>
> I'm going to play around with that and see what I can do to make it
work.
>
>>
>>
>>>>
>>>>>
>>>>> IMO, the problem is in general about tests that are written without
>>>>> specifying a triple, that will be executed with the default triple.
>>>>>
>>>>> Most of these tests were written with X86 (or ARM) in mind, and
there
>> is
>>>>> no guarantee that they will behave as intended with every possible
>> triple.
>>>>> The DataLayout for instance has to be the one from the target, and
is
>> not
>>>>> portable.
>>>>> I think a "portable backend test” is pretty rare in general.
>>>>
>>>> It depends on what the test is trying to do. I'm sure it is quite
>> common
>>>> for IR tests to behave essentially the same way regardless of target.
>>>
>>> IR tests != backend test (I may miss your point here, it’s late…).
>
> Right, sorry, lost focus for a moment there... nevertheless it is still
> the case that many tests exercise functionality that is not particularly
> target-centric and these should be run for any target that actually
> supports that functionality. For example, the DebugInfo tests should
> be run for any target that supports emitting debug info.

I’m not sure that “debug info” support is all or nothing.
As an extreme example, I know targets that support debug info but do not
support function calls, what if your "debug info” test involves these?

Then as part of getting the test suite to work for you, you would need to
disable that particular test for your target. It sounds like this kind of
thing is exactly what the Hexagon folks did, and it seems quite reasonable.
(And in fact I see two DebugInfo tests marked XFAIL: hexagon.)

Also, I’m not a DebugInfo expert, but when a front-end generated them,
aren’t they dependent on the DataLayout? Hence the target?

Not really. DebugInfo tests primarily care what the DWARF description
looks like, not so much what the generated code looks like, and there
are 100 or so DebugInfo tests that work across lots of targets.

Many DebugInfo tests are in target-dependent directories as artifacts of
how the tests are implemented, rather than because they are truly
target-dependent. The ones I've tripped over tend to be target-specific
because they generate assembler and the assembler syntax varies too much
across targets to be worth making the CHECKs tolerant enough.

There are fine details that would be DataLayout dependent, but 99% of
the DebugInfo tests aren't checking things that are at such a low level.

It means that many DebugInfo test could fail with a backend that would
have a significantly different DataLayout and won’t expect some stuff the
way they are played down.

> Whether a
> target supports debug info is orthogonal to its native-ness. As written
> the top-level DebugInfo tests should work fine as long as the default
> triple's backend is included, and that backend supports debug info.
>
> If your backend doesn't support debug info, then it's reasonable to
> exclude those from your testing; but we can't do that at the cost of
> excluding those from testing other backends that *do* support the
feature,
> even if that testing runs in a cross-compiler environment.
>
> In this particular example, we'd be setting things up so that DebugInfo
> is excluded for the wrong reason (not based on some more abstract idea
> of the feature-set of the target) but as Krzysztof observes, getting
> a feature-oriented set of conditions would be a huge task.

Agree: the predicate I wrote is not correct, and I don’t see a trivial
“good” solution. This is why any compromise or intermediate solution that
is good enough to support whatever use-case people here have, including
your, seems appropriate to implement.

Let me know I can help coding something.

I have experimented with implementing the thing Takumi and I think should
be a configuration error. :slight_smile: Basically it takes the same kind of approach
that I did in D12506, except it checks for the existence of the target that
matches the default triple. If that target exists then 'llc' with no triple
will succeed, and it looks like the bulk of the tests that you disabled are
in that category. I'm not especially happy about this tactic, though.

The Hexagon precedent is interesting; Krzysztof said they set the default
triple, and didn't have to xfail all that much stuff. Searching the tree,
I find exactly 7 individual tests marked XFAIL: hexagon, plus it disables
all of ExecutionEngine, and turns off the 'object-emission' feature.

I'm curious if you would try setting the default triple to match your
target, and see what /kinds/ of tests fail. The raw number is much less
interesting than in the categories.

Thanks,
--paulr

If the lack of the default triple was to become a feature, the compiler could simply request specific arch/cpu settings and exit gracefully. The default triple could be set to something that explicitly states the absence of a particular default target (e.g. triple=none, or something like that). This may not be useful for popular targets (that are used as hosts), but it might be for embedded targets that come in a variety of different flavors. The default choice may be a lot less obvious in those cases. As an added bonus, if users have to explicitly specify target details, they would be immune to potential changes of the defaults if they were to upgrade the compiler.

-Krzysztof

Relatedly, I'd really like it if there was some way to run all of those generic tests for *more* than just the default triple. It's quite unfortunate that you cannot easily run, say, the DebugInfo tests for both X86 and Sparc at one go.

It'd be great if there was some sort of 'list of triples to run the generic tests on' build option. If that existed, it'd solve both my wish (to be able to run the tests for multiple such triples -- ideally I think it'd *default* to running them on each configured-in target!), and Mehdi's desire to not run them at all.

Relatedly, I'd really like it if there was some way to run all of those
generic tests for *more* than just the default triple. It's quite
unfortunate that you cannot easily run, say, the DebugInfo tests for both
X86 and Sparc at one go.

It'd be great if there was some sort of 'list of triples to run the
generic tests on' build option. If that existed, it'd solve both my wish
(to be able to run the tests for multiple such triples -- ideally I think
it'd *default* to running them on each configured-in target!),

(making it the default would make check-all a bit slow, I would imagine -
but yes, having an easy option "ninja check-all-llvm-targets" or something
like that could be handy (maybe this could generalize to Clang tests too,
not sure - many of them don't have a fixed triple and it'd be nice to be
able to reproduce failures from an apple buildbot on linux, etc))

and Mehdi's desire to not run them at all.

Just by setting the set of targets to empty? Makes sense.

Modifying lit itself to support running a “generic” test multiple times, iterating through some specified set of triples, seems like a rather more ambitious feature than I’m willing to sign up for. You guys want such a thing, knock yourselves out.

But does this suggest a groundswell of support for configurations where the default triple’s target is NOT included in the build? And having the LLVM tests tolerate that configuration? That patch I have 90% done, and it seems to work okay; my only real objection is that it endorses a pretty counter-intuitive kind of configuration.

(And if an iterate-through-triples patch supersedes what I did, that’s fine too—but my immediate goal is to re-enable cross-compiler testing of these generic tests, which currently isn’t happening at all.)

–paulr

From: Mehdi Amini [mailto:mehdi.amini@apple.com]
Sent: Thursday, September 03, 2015 3:26 PM
To: Robinson, Paul
Cc: Tom Stellard; llvm-dev@lists.llvm.org; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

From: Tom Stellard [mailto:tom@stellard.net]
Sent: Thursday, September 03, 2015 7:31 AM
To: Mehdi Amini
Cc: Robinson, Paul; llvm-dev@lists.llvm.org; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

From: Mehdi Amini [mailto:mehdi.amini@apple.com]
Sent: Wednesday, September 02, 2015 7:10 PM
To: Robinson, Paul
Cc: llvm-dev@lists.llvm.org; tom@stellard.net; NAKAMURA Takumi
Subject: Re: Testing "normal" cross-compilers versus GPU backends

Hi Paul,

Thanks for the summary!

This note arose from

Proofpoint Targeted Attack Protection

3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-

ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e

o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e= but

the

reviewers

felt that we needed a broader audience, because the proposed patch
really didn't solve the entire problem and we had no better ideas.

Mehdi Amini needs to build LLVM with just a GPU backend, and still

have

"ninja check" Just Work. Commits r243958-243960 tried to

accomplish

that; however they are too big a hammer, and cause much simpler

cross

environments (like mine) to exclude big chunks of very useful tests
(including my favorite, DebugInfo).

FYI, my main cross environment is building on X86 Windows but using
a default target triple for PS4 (which is also X86).

I experimented with building LLVM with just the ARM backend

(running

on

an X86 workstation) and setting the default triple to some ARM

value.

"ninja check" worked fine (without Mehdi's series of commits), so

the

normal kind of cross-compiler environment seems to be happy with

how

things were set up originally.

Mehdi reports building LLVM with the X86 and AMDGPU backends,

setting

the default triple to "amdgcn--amdhsa", and getting 200-some

failures.

(This does make me wonder about AMDGPU testing in general; how does

that

work? The only places I see lit checks for AMDGPU are in the usual
target-dependent places.)

I don’t understand this interrogation about how do you do testing in
general. The same way you don’t process tests/CodeGen/X86/* with the

ARM

backend, you can’t process any random IR through these backends.

You said you had 200+ failures with AMDGPU. Are the AMD folks simply
tolerating the 200 failures, and you don't want to? I should hope

there

is more to it than that.

Well, I don’t know, they might just run `ninja check` with the default

triple set to X86?

(which I would consider being working around a buggy test suite)

I always enable AMDGPU and X86 when I build, so I've never run into
this problem.

-Tom

Tom, presumably your default target triple is X86-ish? And the only
tests to exercise the AMDGPU backend are those that explicitly specify
a triple for AMDGPU?

This is how we used to do it as well, and I assume this is how most of
backends out of the main CPUs are dealing with the situation.
This is what I consider a bug and was trying to solve (admittedly not in
an optimal way).

Mehdi, assuming that's what Tom does, your stated goal was to be able to
run tests *without* including the X86 backend, so Tom's solution won't
work for you (restating just for confirmation).

Yes, or alternatively I expect the test suite to pass whatever the default
triple is.

Um, that may be asking a bit much…

It already works for >95% of the suite :slight_smile:
The rest is working (and useful) only for “some” (unspecified) cross compiler configuration.

Krzysztof suggested much the same thing that I think you are currently
doing, which is deliberately configure a default triple but exclude the
corresponding backend.

You and Takumi were considering this as an unsupported configuration
before, and I tend to agree with that (this is the configuration I’m using
for our tests but it was not intentional to leave the default triple
unset).

Right, intuitively it doesn't make sense. Is it actually useful to build a
GPU compiler that will crash unless you ask it to generate GPU code? Seems
to me it should default to producing GPU code.

Correct me if I’m wrong:

You’re viewing this from the “clang” point of view. A default triple is needed because the command line interface does not require to specify it.

I see LLVM as a library or compiler *framework* in the first place, and clang is just a use case as another.

When you build a compiler using LLVM as a library: 1) it does not have to be a command line compiler, 2) the interface does not have to make optional the target selection

Most GPU compilers are embedded in the driver (they compile shaders on-demand during host program execution). The driver can detect the hardware and initialize LLVM with the right triple.

We build LLVM as a shared library, we then build multiple compiler that will link to this library to CodeGen to various backend. The compiler is responsible to select and initialize the appropriate backend, we *never* rely on the default triple, and I don’t even see how we could.

You could also see LLVM as a system library that can have multiple clients, each client responsible of its own initialization.

I expect we can detect that situation in lit.cfg
and exclude tests on that basis, rather than 'native'. It would solve
the problem for my case (host triple != target triple, although the arch
parts of the triple do match) and the "normal" cross-compiler case (e.g.
host = X86, backend + target triple = ARM).

I'm going to play around with that and see what I can do to make it

work.

IMO, the problem is in general about tests that are written without
specifying a triple, that will be executed with the default triple.

Most of these tests were written with X86 (or ARM) in mind, and

there

is

no guarantee that they will behave as intended with every possible

triple.

The DataLayout for instance has to be the one from the target, and

is

not

portable.
I think a "portable backend test” is pretty rare in general.

It depends on what the test is trying to do. I'm sure it is quite

common

for IR tests to behave essentially the same way regardless of target.

IR tests != backend test (I may miss your point here, it’s late…).

Right, sorry, lost focus for a moment there... nevertheless it is still
the case that many tests exercise functionality that is not particularly
target-centric and these should be run for any target that actually
supports that functionality. For example, the DebugInfo tests should
be run for any target that supports emitting debug info.

I’m not sure that “debug info” support is all or nothing.
As an extreme example, I know targets that support debug info but do not
support function calls, what if your "debug info” test involves these?

Then as part of getting the test suite to work for you, you would need to
disable that particular test for your target. It sounds like this kind of
thing is exactly what the Hexagon folks did, and it seems quite reasonable.
(And in fact I see two DebugInfo tests marked XFAIL: hexagon.)

It seems conceptually wrong to me, for the reason I already exposed.
It should go the other way (whitelist instead of blacklist)

Also, I’m not a DebugInfo expert, but when a front-end generated them,
aren’t they dependent on the DataLayout? Hence the target?

Not really. DebugInfo tests primarily care what the DWARF description
looks like, not so much what the generated code looks like,

My question is less will the CHECK matches than “will the backend be able to generate code with invalid debug information (ex: pointer size, etc.) or just crash?”

and there
are 100 or so DebugInfo tests that work across lots of targets.

Many DebugInfo tests are in target-dependent directories as artifacts of
how the tests are implemented, rather than because they are truly
target-dependent. The ones I've tripped over tend to be target-specific
because they generate assembler and the assembler syntax varies too much
across targets to be worth making the CHECKs tolerant enough.

There are fine details that would be DataLayout dependent, but 99% of
the DebugInfo tests aren't checking things that are at such a low level.

It means that many DebugInfo test could fail with a backend that would
have a significantly different DataLayout and won’t expect some stuff the
way they are played down.

Whether a
target supports debug info is orthogonal to its native-ness. As written
the top-level DebugInfo tests should work fine as long as the default
triple's backend is included, and that backend supports debug info.

If your backend doesn't support debug info, then it's reasonable to
exclude those from your testing; but we can't do that at the cost of
excluding those from testing other backends that *do* support the

feature,

even if that testing runs in a cross-compiler environment.

In this particular example, we'd be setting things up so that DebugInfo
is excluded for the wrong reason (not based on some more abstract idea
of the feature-set of the target) but as Krzysztof observes, getting
a feature-oriented set of conditions would be a huge task.

Agree: the predicate I wrote is not correct, and I don’t see a trivial
“good” solution. This is why any compromise or intermediate solution that
is good enough to support whatever use-case people here have, including
your, seems appropriate to implement.

Let me know I can help coding something.

I have experimented with implementing the thing Takumi and I think should
be a configuration error. :slight_smile: Basically it takes the same kind of approach
that I did in D12506, except it checks for the existence of the target that
matches the default triple. If that target exists then 'llc' with no triple
will succeed, and it looks like the bulk of the tests that you disabled are
in that category. I'm not especially happy about this tactic, though.

Why aren’t you happy about that?

The Hexagon precedent is interesting; Krzysztof said they set the default
triple, and didn't have to xfail all that much stuff. Searching the tree,
I find exactly 7 individual tests marked XFAIL: hexagon, plus it disables
all of ExecutionEngine, and turns off the 'object-emission' feature.

I'm curious if you would try setting the default triple to match your
target, and see what /kinds/ of tests fail. The raw number is much less
interesting than in the categories.

Failing tests attached, let me know which ones you’d like me to investigate.

Tests:

failing_tests (11.8 KB)

The Hexagon precedent is interesting; Krzysztof said they set the default
triple, and didn't have to xfail all that much stuff. Searching the tree,
I find exactly 7 individual tests marked XFAIL: hexagon, plus it disables
all of ExecutionEngine, and turns off the 'object-emission' feature.

I'm curious if you would try setting the default triple to match your
target, and see what /kinds/ of tests fail. The raw number is much less
interesting than in the categories.

Failing tests attached, let me know which ones you’d like me to investigate.

Tests:
<failing_tests>

(note I forced enable the “native” feature on this run)

So, just looking at the first one in that list for an example, running:
llc -mtriple amdgcn test/CodeGen/Generic/2002-04-14-UnexpectedUnsignedType.ll
you get an error of:
error: unsupported call to function bar in foo

...because apparently AMDGPU simply doesn't support function calls. At all.

That is a rather unusual feature to be missing, and I'd imagine is causing a fair number of the other nominally-generic tests to fail too...

>>> Krzysztof suggested much the same thing that I think you are currently
>>> doing, which is deliberately configure a default triple but exclude
the
>>> corresponding backend.
>>
>> You and Takumi were considering this as an unsupported configuration
>> before, and I tend to agree with that (this is the configuration I’m
using
>> for our tests but it was not intentional to leave the default triple
>> unset).
>
> Right, intuitively it doesn't make sense. Is it actually useful to
build a
> GPU compiler that will crash unless you ask it to generate GPU code?
Seems
> to me it should default to producing GPU code.

Correct me if I’m wrong:

You’re viewing this from the “clang” point of view. A default triple is
needed because the command line interface does not require to specify it.

I see LLVM as a library or compiler *framework* in the first place, and
clang is just a use case as another.

When you build a compiler using LLVM as a library: 1) it does not have to
be a command line compiler, 2) the interface does not have to make
optional the target selection

Most GPU compilers are embedded in the driver (they compile shaders on-
demand during host program execution). The driver can detect the hardware
and initialize LLVM with the right triple.

We build LLVM as a shared library, we then build multiple compiler that
will link to this library to CodeGen to various backend. The compiler is
responsible to select and initialize the appropriate backend, we *never*
rely on the default triple, and I don’t even see how we could.

You could also see LLVM as a system library that can have multiple
clients, each client responsible of its own initialization.

If you want to write your tests as unit tests linked against the library,
I have no problem with that.

If you want to take advantage of the existing tests, the bulk of them are
written using command-line tools, some of which make use of the default
triple. So, you need to configure your builds appropriately. It's not
about clang, it's about the command-line tools used to implement the tests.
If you don't like how the tests work, you don't have to use them.

>>> I expect we can detect that situation in lit.cfg
>>> and exclude tests on that basis, rather than 'native'. It would solve
>>> the problem for my case (host triple != target triple, although the
arch
>>> parts of the triple do match) and the "normal" cross-compiler case
(e.g.
>>> host = X86, backend + target triple = ARM).
>>>
>>> I'm going to play around with that and see what I can do to make it
>> work.
>>>
>>>>
>>>>
>>>>>>
>>>>>>>
>>>>>>> IMO, the problem is in general about tests that are written
without
>>>>>>> specifying a triple, that will be executed with the default
triple.
>>>>>>>
>>>>>>> Most of these tests were written with X86 (or ARM) in mind, and
>> there
>>>> is
>>>>>>> no guarantee that they will behave as intended with every possible
>>>> triple.
>>>>>>> The DataLayout for instance has to be the one from the target, and
>> is
>>>> not
>>>>>>> portable.
>>>>>>> I think a "portable backend test” is pretty rare in general.
>>>>>>
>>>>>> It depends on what the test is trying to do. I'm sure it is quite
>>>> common
>>>>>> for IR tests to behave essentially the same way regardless of
target.
>>>>>
>>>>> IR tests != backend test (I may miss your point here, it’s late…).
>>>
>>> Right, sorry, lost focus for a moment there... nevertheless it is
still
>>> the case that many tests exercise functionality that is not
particularly
>>> target-centric and these should be run for any target that actually
>>> supports that functionality. For example, the DebugInfo tests should
>>> be run for any target that supports emitting debug info.
>>
>> I’m not sure that “debug info” support is all or nothing.
>> As an extreme example, I know targets that support debug info but do
not
>> support function calls, what if your "debug info” test involves these?
>
> Then as part of getting the test suite to work for you, you would need
to
> disable that particular test for your target. It sounds like this kind
of
> thing is exactly what the Hexagon folks did, and it seems quite
reasonable.
> (And in fact I see two DebugInfo tests marked XFAIL: hexagon.)

It seems conceptually wrong to me, for the reason I already exposed.
It should go the other way (whitelist instead of blacklist)

I think the default expectation for a new target is that it should support
basically all target-independent LLVM features, and there are tests for lots
of those features. Therefore if a target fails to support something, the
owner of the target is responsible for making sure those tests are disabled
appropriately.

It sounds like you think targets should get to pick and choose which things
they want to support, and then have to go do work to explicitly enable the
target-independent tests for the things they choose to support.

We disagree on this point, and I think historically my view is where the
community is coming from; this is just my opinion however and I could
easily be wrong, but it is clearly how the test suite operates. If you
want to reorganize the test suite along different lines, you should start
your own llvm-dev discussion about that.

>
>>
>> Also, I’m not a DebugInfo expert, but when a front-end generated them,
>> aren’t they dependent on the DataLayout? Hence the target?
>
> Not really. DebugInfo tests primarily care what the DWARF description
> looks like, not so much what the generated code looks like,

My question is less will the CHECK matches than “will the backend be able
to generate code with invalid debug information (ex: pointer size, etc.)
or just crash?”

The target's address size is given by the DataLayout, which all targets
are required to define, so I'm completely failing to understand the problem.
Why would the debug info be invalid?

Offhand I can't think of anything the target has to do specially to support
debug info, it should Just Work. The details of the object-file format
matter more than the target itself, in my experience, and even that matters
relatively little. If your target supports emitting arbitrary data, and
assembler labels/object-file relocations, then your debug info should be fine.

DebugInfo /tests/ end up "depending" on the target only because there's no
convenient way to see what the debug info looks like, without producing
either an assembly file or an object file. Producing either of those things
requires passing data through the target. The details of the target itself
are largely irrelevant (although as I've mentioned before, details of the
assembler syntax might matter for writing actual CHECK lines).

If your target can't produce an object file, then you should turn off
'object-emission' (Hexagon does this, for example). Other than that, I don't
see any reason why debug-info tests can't in principle be target-neutral.

> I have experimented with implementing the thing Takumi and I think
should
> be a configuration error. :slight_smile: Basically it takes the same kind of
approach
> that I did in D12506, except it checks for the existence of the target
that
> matches the default triple. If that target exists then 'llc' with no
triple
> will succeed, and it looks like the bulk of the tests that you disabled
are
> in that category. I'm not especially happy about this tactic, though.

Why aren’t you happy about that?

Because it takes what would seem to be a configuration error and makes it
something we explicitly tolerate. If people conclude that it should be
tolerated, then that's fine.

Hmm I'm told GPUs as a class typically don't support calls (everything has
to be inlined).

Perhaps the more appropriate direction here is to invent a "gpu" config flag
and mark some things XFAIL: gpu.
--paulr