The Trouble with Triples

[echristo]

Hi Daniel,

(from the context, you might have meant ‘tuple’ where you’ve written ‘triple’. I’m answering based on the assumption you meant ‘triple’)

I did mean what I wrote.

The GNU triple is already used as a way of encoding a large amount of the target data in a string but unfortunately, while this data is passed throughout LLVM, it isn’t reliable because GNU triples are ambiguous and inconsistent. For example, in GCC toolchains mips-linux-gnu probably means a MIPS target on Gnu/Linux but anything beyond that (ISA revision, default ABI, multilib layout, etc.) is up to the person who built the toolchain and may change over time. Another example is that Debian’s definition for i386-linux-gnu has been i486 and i586 at various points in time.

Sorta…

The proposed TargetTuple is a direct replacement for the GNU triple and is intended to resolve this ambiguity and move away from a string-based implementation (we need to keep a string serialization though, see below). Essentially, I’m trying to push the ambiguity out of the internals and give the distributor control of how the ambiguity is resolved for their environment. Once that is done, we’ll be able to rely on the TargetTuple for information about the target such as ABI’s, architecture revisions, endianness, etc.

This is pretty vague.

I agree that we should open up the API to specify the appropriate data and that is something that TargetTuple will acquire during step 4 and 7 of the plan (mostly step 7 where compiler/tool options begin mutating the target tuple). I don’t agree with keeping the GNU triple around though for two main reasons. The first is that most people believe that GNU triples accurately describe the target and there will be a strong temptation to inappropriately base logic on them. The second is that the meaning of the triple varies between toolchain builds and over time and there is a significant potential for bugs where different parts of the toolchain use different meanings for the same GNU triple (due to rebuilding or switching toolchains, or moving objects from system to system). We ought to resolve the ambiguity once and then stick to that interpretation.

The string serialization I mentioned above is useful for LLVM-IR as part of a direct replacement for the ‘target triple’ statement. We could split this statement up into smaller pieces but the migration to target tuples is already difficult so I think it would be best to do a direct replacement first and redesign the IR statements later if we want to. The serialization is also useful for command line options on internal tools such as llc to give us precise control over our tests that the GNU triple can’t deliver. This will be particularly important when distributors can apply their own disambiguations to GNU triples. The serialization may also be useful as part of a C API but I haven’t given the C API much thought beyond preserving the current API.

My first impression of using this serialization as is that it’s something I’m against. Keep in mind that being able to parse the string can’t invoke a target backend to handle the rest of the parsing. It’d need to be as generic as a DataLayout if you want to do this sort of thing and I’m entirely uncertain this is possible for the goals you (and I) have in mind here.

Hopefully, that helps clear up your concerns. Let me know if there’s anything that still seems strange.

Not really. I don’t see much of a sketch on what you have in mind for your “TargetTuple” here other than “it’ll be a bunch of things together”.

Let me be clear, I do agree with you that the Triple by itself is insufficient for what we want long term in the backends, however, we won’t be able to get rid of it completely. It’s too ingrained into how cross compilation is done as a base. It is, however, possible to design an API that includes the Triple and the relevant information to augment sufficiently. My vision for this is an API that has a base part that is going to be generic across all targets (think the current arguments to the TargetMachine constructor), and additional target specific information that can be passed in via user customization (i.e. command line options etc).

My suggestion on a route forward here is that we should look at the particular
API and areas of the backend that you’re having an issue with and figure out
how to best communicate the data you’d like to the appropriate area. I realize
this probably seems a little vague and handwavy, but I don’t know what areas
you’ve been having problems with lately. I’ll absolutely help with this effort if
you need assistance or guidance in any way.

The MIPS specific problems are broad and varied. Some of the bigger ones are:

• Building clang on a 32-bit Debian and a 64-bit MIPS processor produces a compiler that cannot target the native system. The release packages work around this by ‘cross-compiling’ from the host triple to the target triple which are different strings (mips-linux-gnu vs mips64-linux-gnu) but have the same meaning.
• It’s not possible to produce a clang that can generate code for both 32-bit and 64-bit MIPS without one of them needing a -target option to change the GNU triple. This is because we based the logic on the triple and lack anything else to use.

I blame the mips backend for this one. We can do -m32/-m64 just fine for x86 as an example. Some backends have this problem, others don’t.

• Various details (ELF headers, label prefixes, exception personality, JIT target, etc.) depend on the ABI and OS Distribution rather than just 32-bit vs 64-bit

Sure?

• It’s not possible to implement clang in a way that can support all of mips-linux-gnu’s possible meanings. mips-mti-linux-gnu, and mips-img-linux-gnu have the same problem to a lesser degree

I’m really not sure what any of these things are bringing up. You haven’t actually said what communication problem you’re trying to solve between the user and the compiler here. How about we start this from another perspective? Can you give some examples of what you’d like to do to communicate the information you think you need to various parts of the backend and how you’d like to communicate it?

I promise I’m not trying to be (on purpose at least) particularly dense here, but I just don’t have enough information to work with here. I agree that we probably have an API problem - some of which I solved for the mips backend at one point using MCOptions (which I don’t really like as a general solution), but a more general solution that’ll work and be cleaner is definitely a direction I’d like us to go.

-eric

[Renato_Golin1]

This is pretty vague.

Let em give you one concrete example where I want to use the TargetTuple.

Today, we parse the triple into a Triple object. Then, we parse all
command lines, and add special -target-feature flags. This logic is
duplicated for all tools that accept the same command line options,
because there isn't a single source of information.

My first step was to create the TargetParser, which does the string
parsing, resolving to legal internal representations, that can be use
to not only validate, but also transfer knowledge to other classes,
like TargetTuple, about what the target is. As it stands, TargetParser
has a variable internal representation (enums and tables, that will be
generated by tablegen later), so it's not safe to rely on them between
revisions, but it's ok because no one is. We now use that class to
parse Clang's options, assembler directives, etc.

My first goal with the TargetTuple is to build it with a Triple (which
also uses TargetParser to parse the architecture part), and then let
all methods I'm currently keeping on TargetParser to set the feature
flags. Once that's done, we can start adding a more stable API and use
methods for the most common/important flags (such as hasVFP(),
isARMv7(), etc) and let the accessors return the internal unreliable
value for direct comparisons using the enum values.

Ex:
  if (Tuple.getCPUFeatures() & ARM::HWDIV) // has hardware divide.

That should not be used to build TargetMachine and others, but just
for local enquiries, and possibly not all of it should be exposed.
That's why I want to have separate (maybe redundant) ways: direct
access to const values, and accessor methods for what's *really*
public.

From there, it should be straightforward to have a simple method:

  Tuple.getTargetFeatures(Features);

which will expose every thing that Clang does today, based on its own
string parsing. Once that's done, than Clang will just need to know
what kind of argument goes into what kind of field:

  const Arg *CPUArg = Args.getLastArg(options::OPT_mcpu_EQ);
  Tuple.setCPU(CPUArg);
  const Arg *FPUArg = Args.getLastArg(options::OPT_mfpu_EQ);
  Tuple.setFPU(FPUArg);

since Tuple depends on the architecture, setting CPU, FPU, Extensions
should use the arch-specific parser, which will use arch-specific
Tuple.setFeatures() -> Tuple.setARMFeatures() after setting the CPU,
FPU, etc.

My first impression of using this serialization as is that it's something
I'm against. Keep in mind that being able to parse the string can't invoke a
target backend to handle the rest of the parsing. It'd need to be as generic
as a DataLayout if you want to do this sort of thing and I'm entirely
uncertain this is possible for the goals you (and I) have in mind here.

I'm ok with adding a condensed extra argument with some key target
features. Changing the triple will disrupt too much of the past, and
goes back to the mud ripples effect we were talking earlier.

But even that features string has to be small, condensed, and will
never represent everything. We should not expect that we can encode
every bit of information, nor should we strive to put as much as
possible.

We have to assume that the user will pass the same arguments to llc,
lli, llvm-mc and other tools as (s)he passed to clang. If the command
line parsing (at least the target-specific part) is common, than we
won't even need the extra features field.

Let me be clear, I do agree with you that the Triple by itself is
insufficient for what we want long term in the backends, however, we won't
be able to get rid of it completely. It's too ingrained into how cross
compilation is done as a base.

Precisely.

My vision for this is an API that has a base part that is
going to be generic across all targets (think the current arguments to the
TargetMachine constructor), and additional target specific information that
can be passed in via user customization (i.e. command line options etc).

That's how I see it, too.

cheers,
--renato

[Daniel_Sanders]

There will be no string representation of all options, as that would
be impossible, or at least, highly undesirable, especially in the IR.

I just want to double check we're saying the same thing here.
We will need a string serialization of the TargetTuple so that we can store it in the IR and read it back. Whether this is in one piece (e.g. 'target tuple = "...") or multiple pieces (e.g. 'target arch = "..."', 'target endian = "..."') doesn't matter too much at first but I can see the benefits of the latter being the end point. If we do choose that multiple pieces then I'd like to pass through the former first to keep the early steps of the migration to a TargetTuple as simple and mechanical as possible.

[Daniel_Sanders]

From: Renato Golin [mailto:renato.golin@linaro.org]
Sent: 30 July 2015 00:11
To: Eric Christopher
Cc: Daniel Sanders; LLVM Developers Mailing List (llvmdev@cs.uiuc.edu); Jim
Grosbach (grosbach@apple.com)
Subject: Re: The Trouble with Triples

> This part doesn't seem obvious from the direction the patches are going.

Until now, most of what he has done was to refactor the Triple class,
with no functional change, and to create a thin layer around the
Triple (called Tuple) and pass those instead. This is on par with that
premise.

The current patch is the first one to actually have some more
substantial change, so it's good that you stopped it now, before we
start breaking everything.

Definitely. It's better to be discussing it now rather than (currently) 30 big patches later when I reach the patches that change the IR.

Maybe, knowing what it is now, if you could have another quick look at
the patch, and see if the new light has helped understand the patch
for what it will be. Maybe it's still not good enough, so then we'll
have to resort to a new round of design discussions.

> Definitely don't want this in the middle end at all. That all can be part of
> the TargetMachine/TargetSubtargetInfo interface.

Ah, yes! But the TargetMachine (& pals) are created from information
from the Triple and the other bits that front-ends keep for
themselves.

TargetMachine and many of the others actually hold a llvm::Triple object as their representation of this information. It was a std::string until recently but most users of the member were re-parsing it with llvm::Triple before using it.

So, in theory, if the Tuple is universal, creating them with a Tuple
(instead of a Triple+stuff) will free the front-ends of keeping the
rest of the info on their own, and TargetMachine/SubTargetInfo/etc
will be more homogeneous across different tools / front-ends than it
is today.

Another strong point is: we're not trying to change any other machine
description / API. This is just about the user options and defaults,
that are used to *create* machine descriptions.

>> The decision to create a new class (Tuple) is because Triple already
>> has a legacy-heavy meaning, which should not change.
>
> Agreed with at least the "because" part.

There was also the name. Triple is very far from the truth.

But changing the Triple class could cause ripples in the mud that
would be hard to see at first, and hard to change later, after people
started relying on it.

The final goal is that the Triple class would end up as being nothing
more than a Triple *parser*, with the current legacy logic, setting up
the Tuple fields and using them to select the rest of the default
fields.

> OK. What's the general class design look like then? The text from the
> original mail was fairly confusing then as I thought he was doing something
> that you say he isn't doing

Daniel, can you send your current plan for the Tuple class?

cheers,
--renato

Sure. I'll need some time to transfer it to email and I'm debugging a regression in the 3.7.0 release at the moment so I'll send another email later.

[Renato_Golin1]

We will need a string serialization of the TargetTuple so that we can store it in the IR and read it back.

Why does it have to be in the IR? If every tool that deals with IR has
the same options and they mean the same thing I don't see why we'd
need that.

Whether this is in one piece (e.g. 'target tuple = "...") or multiple pieces (e.g. 'target arch = "..."', 'target endian = "..."') doesn't matter too much at first but I can see the benefits of the latter being the end point. If we do choose that multiple pieces then I'd like to pass through the former first to keep the early steps of the migration to a TargetTuple as simple and mechanical as possible.

Assuming we really need it, wouldn't it work if we put in metadata?
That way, it would bloat the IR of targets that really needed it and
not the ones that don't, at the same time as being non-critical to
parsing and validating the IR, so you could safely drop some
information without breaking the file.

This would allow more flexibility across targets, if they need
different types of storage, and would allow targets that do need the
info to complement the missing info with their own defaults. So we
could reuse old IR with new IR and still make it work.

cheers,
--renato

[Daniel_Sanders]

From: Renato Golin [mailto:renato.golin@linaro.org]
Sent: 30 July 2015 15:07
To: Daniel Sanders
Cc: Eric Christopher; LLVM Developers Mailing List (llvmdev@cs.uiuc.edu);
Jim Grosbach (grosbach@apple.com)
Subject: Re: The Trouble with Triples

> We will need a string serialization of the TargetTuple so that we can store it
in the IR and read it back.

Why does it have to be in the IR? If every tool that deals with IR has
the same options and they mean the same thing I don't see why we'd
need that.

My main reason is that the GNU triple is currently in the IR and I'm trying not to make design changes in the early stages of this work.

I haven't fully researched why the triple is there but one reason is to make backends target the same target as the frontends. Consider the case where clang is used to generate IR and then llc is used to compile it. We currently use the information in the IR to set the GNU triple, the target features, etc.

One case where tool options isn't sufficient is ModuleLinker::run(). It currently compares the GNU triples to determine whether the IR modules are link-compatible. I'm sure I've seen a couple others but they're not springing to mind at the moment.

> Whether this is in one piece (e.g. 'target tuple = "...") or multiple pieces
(e.g. 'target arch = "..."', 'target endian = "..."') doesn't matter too much at
first but I can see the benefits of the latter being the end point. If we do
choose that multiple pieces then I'd like to pass through the former first to
keep the early steps of the migration to a TargetTuple as simple and
mechanical as possible.

Assuming we really need it, wouldn't it work if we put in metadata?
That way, it would bloat the IR of targets that really needed it and
not the ones that don't, at the same time as being non-critical to
parsing and validating the IR, so you could safely drop some
information without breaking the file.

This would allow more flexibility across targets, if they need
different types of storage, and would allow targets that do need the
info to complement the missing info with their own defaults. So we
could reuse old IR with new IR and still make it work.

cheers,
--renato

That makes sense to me. What would we do if the metadata is absent for some reason?

[Renato_Golin1]

I haven't fully researched why the triple is there but one reason is to make backends target the same target as the frontends. Consider the case where clang is used to generate IR and then llc is used to compile it. We currently use the information in the IR to set the GNU triple, the target features, etc.

This already works well, at least for ARM, without necessarily relying
on the target triple in the IR file. Try to strip the target line then
pass it back again via command line. Everything works fine.

I don't know why the triple ended up in the IR, but I think it was the
need to identify the back-end it was tailored to, not necessarily
every detail of the compiler choices. It could easily have been "arm",
"mips", etc. but I guess it ended up as a triple because it was
easier. Most of what triples encode are already present in IR as
function attributes, lowering choices (PCS, ABI, etc), and type sizes.

That makes sense to me. What would we do if the metadata is absent for some reason?

Nothing.

What you're aiming for is just giving IR more information about
something that can be passed through command line options. Right now,
that's how people use it, so most of the cases, the IR info will be
redundant. Worse still, it could create conflicts if the options are
not completely similar (or have slightly different meanings), which is
bad.

If we're only talking about ARM, I'd put no more than
arch/cpu/fpu/extension in the IR as metadata, to be used in case there
is no other information in the command line, but with the latter
always overriding the behaviour. Mips probably has some other stuff,
x86 has less, as CPU names are more unique than in ARM. Maybe putting
the ABI (GNU, EABI, Darwin) would be a good thing too, to help linking
the objects with the right libraries.

But the lack of information can, and should, be overridden by command
line options. In case there's none, defaults should be used. In that
case, the compiler shouldn't do anything smart on its own, and let the
user see warnings, errors, breakages, etc. We can't control the world
outside the compiler.

cheers,
--renato

[Daniel_Sanders]

After re-reading your last couple emails with the benefit of sleep, that all makes sense to me. The user/driver would be responsible for ensuring each tool is called with the same options, while each tool is responsible for using the same TargetTuple in all its calls to LLVM. The only bit I'm unsure about w.r.t to not having a serialization in the IR is the ModuleLinker where we currently have code to ensure that all the modules are compatible (by comparing the triple). Presumably, this code was added because there was a real possibility of having incompatible modules (i.e. different triples) so we shouldn't rely on the tool options here. For this the serialization via metadata makes sense to me. We can then compare and merge TargetTuples with target-appropriate leniency and/or strictness.

Step 5 is the relevant step of the plan.

5. Replace serialized Triples with serialized TargetTuples in LLVM-IR.
  a. Maintain backwards compatibility with IR using triples, at least for a while.

I don't think we need to change the plan. We're just changing the serialization format from a string to metadata. I'll need to read up on the metadata before I can define a serialization.

[Renato_Golin1]

Exactly. That's why I suggested to not touch the triple and to add the
rest as metadata. So that previous versions of Clang could still
compile and link new IR and vice-versa.

The triple still plays the part of the bare minimum in the new scheme,
and that's not going away. But if we have more metadata, you can have
better warnings/errors when linking IR.

Each target will be able to choose whatever they think it's
appropriate and the rest will not have to cope with attributes they
don't understand, not the bloating that this would make.

cheers,
--renato

[Daniel_Sanders]

The triple still plays the part of the bare minimum in the new scheme,
and that's not going away. But if we have more metadata, you can have
better warnings/errors when linking IR.

I just want to check one more detail here. Suppose the IR is generated on one host and linked on another, and that these two hosts have different meanings for the triple (whether by accident or intention). I would expect the metadata to specify all values in the TargetTuple for this to behave the same way as generating and linking on one host. Do you agree?

The root of my concern is that the compatibility check could incorrectly say two modules are compatible because the meaning of the triple used to initialize the TargetTuple changed.

[echristo]

We will need a string serialization of the TargetTuple so that we can store it in the IR and read it back.

Why does it have to be in the IR? If every tool that deals with IR has
the same options and they mean the same thing I don’t see why we’d
need that.

I agree with the need for serializing information and options into the module.

Whether this is in one piece (e.g. 'target tuple = “…”) or multiple pieces (e.g. ‘target arch = “…”’, ‘target endian = “…”’) doesn’t matter too much at first but I can see the benefits of the latter being the end point. If we do choose that multiple pieces then I’d like to pass through the former first to keep the early steps of the migration to a TargetTuple as simple and mechanical as possible.

Assuming we really need it, wouldn’t it work if we put in metadata?
That way, it would bloat the IR of targets that really needed it and
not the ones that don’t, at the same time as being non-critical to
parsing and validating the IR, so you could safely drop some
information without breaking the file.

Metadata won’t work. Metadata is defined to not affect code correctness and this obviously does.

-eric

[Renato_Golin1]

That's a good point... But I think we're going around in circles already...

Should we try something more efficient? Like a hangout or something?

cheers,
--renato

[mehdi_amini]

Is it true for Module level Metadata and not only for instruction attached one?

Thanks,

[resistor]

There are examples of module-level metadata that impact the correctness of the generated code in the LangRef today:

http://llvm.org/docs/LangRef.html#module-flags-metadata
http://llvm.org/docs/LangRef.html#objective-c-garbage-collection-module-flags-metadata
http://llvm.org/docs/LangRef.html#automatic-linker-flags-module-flags-metadata
http://llvm.org/docs/LangRef.html#c-type-width-module-flags-metadata

—Owen

[echristo]

Wow, those are terrible. I remember the first as it was a hack for dealing with LTO and being unable to specify code generation flags. It definitely was not meant to affect whether or not correct output happened.

The rest appear to be even worse and should not have gone in. I can prepare a patch to remove them if you’d like.

-eric

[echristo]

In case it wasn’t obvious this is me both being sad, mostly joking, and a little serious about it. I realize that apparently the ship has sailed here, but these should probably be first class IR citizens rather than building off of the “named metadata” extension.

That last one definitely should go away in some means.

-eric

[resistor]

The problem with this direction is that clients need to be able to extend the IR with their own additional annotations according to their own schemas. We cannot practically bake all of those in as first-class IR citizens.
To throw in another example, SPIR 1.2 uses module metadata for required-for-correctness purposes: https://www.khronos.org/files/opencl-spir-12-provisional.pdf

—Owen

[resistor]

As does NVVM: http://docs.nvidia.com/cuda/nvvm-ir-spec/index.html#global-property-annotation-chapter-11

—Owen

[Daniel_Sanders]

(from the context, you might have meant ‘tuple’ where you’ve written ‘triple’. I’m answering based on the assumption you meant ‘triple’)

[Renato_Golin1]

I feel your pain. We have our own variants of VFP, PCS, extensions,
ILPXX, IEEE support, CPUs with the same revision belonging to two
different architectures and vice-versa. Not to mention that we have
two back-ends, and one architecture (ARMv8 is split between the two).
And all the brand names that people mad up during the years
(StrongARM, XScale, Krait, Tegra) which map to custom features, etc.

But the way we're solving this, via out TargetParser, is to create
tables of default/allowed values, so that users will often get what
they want, even if they don't specify most of it. I imagine that MIPS
would also benefit from a TargetParser. And, if we keep the complexity
in the parser and the driver, we won't need to store much in the IR,
IFF they have the same meaning everywhere. So, whatever we spit out in
the IR has to be necessary and sufficient to get the meaning "just
right".

--renato