Clang option to provide list of target-subarchs.

There are at least four clang frontends for offloading to accelerators:
1 Cuda clang 2 OpenMP 3 HCC and 4 OpenCL. These frontends will
want to embed object code for multiple offload targets into a single
application binary to provide portability across different subarchitectures
(e.g. sm_35, sm_50) and across different architectures (e.g nvptx64,amdgcn).

Problem: Different frontends are using different flags to provide a
list of subarchitectures. For example, cuda clang repeats the flag
“--cuda-gpu-arch=sm_35 --cuda-gpu-arch=sm_50” and HCC uses
“--amdgpu-target=gfx701 --amdgpu-target=gfx802”.

We propose a common clang flag to provide a list of target
subarchitectures called “target-subarchs”. For example,

--target-subarchs=sm_35,sm_50,gfx701,gfx802

In discussions with HCC and OpenMP maintainers, we believe a new
flag name would have these requirements:
   end in “s” because it is a list;
   not have vendor specific names like cuda and amd;
   not contain “gpu” because offloading may extend to non-gpu archs;
   avoid “arch” by itself so as not to be confused with first field of a triple;
   and not collide with existing flags to allow both options.

"--target-subarchs" satisfies all the above. Comments?

Greg Rodgers

Adding Justin as well.

Overall this seems reasonable to me depending on the actual patch :slight_smile:

-eric

In principle this sounds fine to me. Some questions/comments:

* You can specify --cuda-gpu-arch multiple times, and each time adds a
new CUDA arch.

How is this going to work with --target-subarchs? Is there going to
be a --no-target-subarchs flag to disable subarchs? What will the
semantics of this be, exactly?

The semantics of flags that mean "compile for this one subarch" and
"don't compile for this one subarch" seem a lot more straightforward
than flags that deal in lists. What are your thoughts about making it
work that way instead?

* I don't think "target subarch" is a great name. I don't think most
people think of GPU architectures as "sub-architectures"; "subarch"
implies to me that there are different GPU archs for each CPU arch,
which obviously isn't the case.

Similarly, what problem are we solving by putting "target" in the flag
name? We already have e.g. -march; it's not -mtarget-arch.

"--offload-arch", maybe?

* As discussed offline, we'll need to continue supporting the existing
flags, e.g. --cuda-gpu-arch, probably forever.

* There are a bunch of other flags we may want to harmonize. For
example, I believe CUDA and opencl both have separate
flush-denormals-to-zero flags.

Thank you for the feedback.

How is this going to work with --target-subarchs? Is there going to be a --no-target-subarchs flag to disable subarchs? What will the semantics of this be, exactly?

The large number of subarchs expected makes an inclusive only flag desirable and an exclusive flag impractical. Also, since subarchs will age more quickly than archs, who knows what old crufty subarchs you would get with an exclusion flag. We expect that the runtime will match the most appropriate subarch.
  
As is currently done with --cuda-gpu-arch, we expect that the triple for the arch will be implied from the context. That is, if one specifies --target-subarchs="sm_50,gfx702", the software will generate the triples "nvptx64-nvidia-cuda" and "amdgcn--cuda" from the subarchs. Collisions (different archs) for the same subarch are unlikely and indicate a poor choice of subarch names. For example, AMD should never choose sm_ prefix for its subarchs.

... than flags that deal in lists. What are your thoughts about making it work that way instead?

The semantics of repeating a flag for each desired object verses a list does ease typing, which may not be justification enough. But when they get lost and separated in long option lists, it could be frustrating. Using a list, improves readability of scripts. As we said, existing flags would still be supported.

what problem are we solving by putting "target" in the flag name? We already have e.g. -march; it's not -mtarget-arch. "--offload-arch", maybe?

There are no problems solved with the word "target". The genesis for me of this name is the association with OpenMP target pragmas used for offloading. target is a noun and offload is a verb. We desire a list of objects that end in s . I am ok with archs instead of subarchs because it continues to imply some relationship with the arch field of the triple.

I am ok with "--offload-archs" . If anyone has an issue with --offload-archs, please raise them here.

Thank you

Greg

The large number of subarchs expected makes an inclusive only flag desirable and an exclusive flag impractical.

Sorry, I think I wasn't clear about what I meant.

I did not mean that --no-target-subarchs=X would enable all known
subarchs other than X. That would, as you say, cause problems due to
the large number of subarchs we support. Instead, I meant that
--target-subarchs=X,Y --no-target-subarchs=Y,Z would build only
subarch X.

This is similar to many other flags in clang. See for example -Wfoo
and -Wno-foo. These can override each other; the last one wins.

This is important so that scripts can refine the compile flags
provided by other scripts. We currently do this inside of Google with
--cuda-gpu-arch -- it is not a contrived use-case.

For basically the same reasons that we allow -Wfoo and -Wno-foo to
appear in the same command line invocation, I think we should allow
--offload-archs=X and --no-offload-archs=X to appear in the same
invocation.