[RFC] Make Lanai backend non-experimental

Dear LLVM community,

We wanted to discuss the possibility of moving the Lanai backend from an experimental status to a regular backend. During the initial upstreaming of the Lanai backend (llvm-dev thread “[RFC] Lanai backend”, http://lists.llvm.org/pipermail/llvm-dev/2016-February/095118.html) it was discussed that we could move out of experimental status after a few months of successful life within LLVM.

The Lanai backend is actively maintained, it has an official ELF number assigned; we fix bugs and keep improving it. Therefore, we think the buildbots should build it. We are OK with changes being submitted as long as they don’t break the codegen tests - we’ll examine the effects on runtime internally and will send out patches to fix things if problems occur.

Best regards,

Jacques & Lanai compiler dev team

I don't see why not. LGTM.

Thanks!
--renato

Same here.

-Krzysztof

Oops - meant to send this to the full list.

  MartinO

Presumably if my out-of-tree backend was to be pushed to LLVM, it too would be considered experimental.

Yes. Though, not all out-of-tree back-ends end up upstream for
different reasons.

A few basic rules to get accepted are if:
* the target exists and can be easily purchased / emulated for
investigating problems,
* there are official documents / specs published by the project /
company that maintains the targets,
* there is a reasonable community maintaining the rest of the system
(firmware, OS, other tools, etc),
* enough people commit themselves to maintain the LLVM back-end to
avoid bit-rot,
* the back-end is free of contentious features that would mean
breaking every other target.

For example:
* a PIC back-end ticks many of the boxes, but it doesn't have a
community around it and would very quickly bit-rot,
* the CHERI back-end has an active community and interested
developers, but it would disrupt the IR too much,
* the BPF back-end has been added mostly as-is and is actively
maintained, so gained the status of non-experimental.

The main difference between experimental and official is that they get
built by default. So, if you don't specify them at CMake time, they
will be built and their tests will execute, which is a big win for the
back-end maintainers (as they get free testing), but it's also a big
burden on other developers if their tests are not independent enough
and keep breaking due to unrelated changes.

Under those conditions, it's as easy to remove a back-end that is too
much trouble and no one cares about, as it to add a new experimental
back-end or promote it to official. It all depends on the community
around it and how much they value the LLVM back-end.

But what is involved in making a particular target, feature or back-end experimental other than simply agreement? Are there particular build processes or '-Dmacroname's to add, or how does somebody like me edit my out-of-tree sources so that it would obey the "is experimental" rules?

I'm assuming you already have an implementation, for example, in
GitHub. The easiest way to get things started is to point people at
the repository and make sure it works on trunk.

The second step, then, would be to merge that into the tree and
propose to include it as an experimental target. It will take a few
months until the process is complete and then you'll be able to check
it out and work straight from LLVM's repository.

You'll have to change the CMake to force it to be built
(-DLLVM_TARGETS_TO_BUILD) until such time that you have shown that you
care about the target and that it hasn't failed (best if you have a
buildbot showing the history always green).

Once all that is done, you can do as Jacques did and request it to be
built by default (aka. non-experimental, aka. official back-end).

After that change, though, you're permanently enlisted to maintain it
even more intensely. Not just your buildbot will break, but other
people's too, and they're probably a lot faster than yours, so you
need to keep an eye on them, especially if they break due to unrelated
commits.

That is the contract that you have to sign to have your target
supported, and if you break that contract (let it bit-rot, disable
tests that should be passing, remove parts of the code, or just
neglects bug reports), your back-end *will* be removed.

Hope that helps,
--renato

The IR disruptions for CHERI aren’t that major (the main thing is that we allow alloca’s in different address spaces, but that’s not too difficult to upstream), but it does fail to tick one box that you didn’t list:

* The ISA is stable and mature.

The CHERI ISA is now evolving a lot less quickly than it was a year ago, but we do still make changes that break legacy code and so it would not be helpful to have a compiler with a release cycle that is completely decoupled from our FPGA / QEMU releases. It would be annoying for everyone upstream to have something in tree where the ISA is a moving target.

A few things that we’ve done are definitely ‘research quality’ code and wouldn’t meet upstream’s code quality standards either.

David

The IR disruptions for CHERI aren’t that major (the main thing is that we allow alloca’s in different address spaces, but that’s not too difficult to upstream),

I remember things like inttoptr / ptrtoint were major pains to you and
could be even experienced in the front-end (constant folding, etc).
Has that changed recently?

but it does fail to tick one box that you didn’t list:

* The ISA is stable and mature.

Indeed! That was part of the "documents published", but I agree, one
can "publish" anything these days. :slight_smile:

A few things that we’ve done are definitely ‘research quality’ code and wouldn’t meet upstream’s code quality standards either.

Some "research quality" internals can be seen in any back-end if you
look closely enough, but having most of it being flaky is indeed, a
big problem that will show up in breakages for unrelated changes.

cheers,
--renato

Hi Renato

We would like this feature upstreamed

-Matt

Agree with Pete.

I’d add (because the community will have to maintain it somehow):

  • good code quality and documentation
  • good testing (small tests, documented, with good coverage).

Do you have an in-tree target that would make use of it? If so, I’d be happy to start disentangling it from our tree (I think it’s in a fairly self-contained set of changes). We only support one alloca address space per module currently, because for our use we’re simply changing the address space of the entire stack, rather than individual allocations, would that be enough for you or do you need something more general?

David

Hey,

That sounds good. We are:
* Open-sourcing functional simulator to verify execution;
* Publishing ISA spec document that could serve as official reference
- the previously published documentation is out-of-date;

I've created a patch (https://reviews.llvm.org/D22530) that adds Lanai
as one of the default targets.

Thanks,

Jacques

Hi Jacques,

Thanks for the effort! What's the time frame for this?

Pete's point is an important one, and I think this could be an
important example on how we keep back-ends in or out.

It seems you guys are doing the right thing, but it would be good if
you could do that first, so that we can use Lanai as an example to all
future targets.

Following this thread, I'm adding a new section to the developer's
policy to outline the criteria for new targets:

https://reviews.llvm.org/D22753

cheers,
--renato

Hey,

That sounds good. We are:
* Open-sourcing functional simulator to verify execution;
* Publishing ISA spec document that could serve as official reference
- the previously published documentation is out-of-date;

Thanks Jacques. Thats great! I look forward to trying it out.

Cheers,
Pete

Presumably if my out-of-tree backend was to be pushed to LLVM, it too would be considered experimental.

Yes. Though, not all out-of-tree back-ends end up upstream for
different reasons.

A few basic rules to get accepted are if:

  • the target exists and can be easily purchased / emulated for
    investigating problems,
  • there are official documents / specs published by the project /
    company that maintains the targets,

Most of your list makes a lot of sense to me Renato, but these two items don’t really.

I think a large number of targets in tree would be extremely hard to purchase and directly investigate problems. As examples:

  • Hexagon
  • SystemZ
  • XCore

I’m sure it is theoretically possible for me to acquire such a system and test on it, but the amount of time and energy it would take make it a practical impossibility.

Even for fairly pedestrian backends such as AArch64 and PowerPC, we routinely have people ask active developers on those backends to do the triaging of issues. I’ve never seen this be a problem in practice, and I’m not convinced that it is something we should actually require to have an upstream backend.

I also worry that we have backends for targets that don’t have any (real) published ISA spec. I’m not sure there is a formal “spec” per-se for BPF. And perhaps there is a spec for AMDGPU or Hexagon, but even when I’ve asked developers on those backends about problems, they’ve not pointed be at the published ISA documentation.

And I don’t think any of these backends are bad or should be removed. I rather like several of them. =] I just think that sufficient community involvement makes the availability of hardware/emulator “nice to have” but not essential, and it makes the ISA documentation substantially less important.

The rest of your list below though, I think is a great list.

Hi Chandler,

I think you have good points. Maybe we could make some hard-lined
rules and others as "nice-to-have".

The biggest problem is the community behind it, outside of LLVM. If
the community is strong, and they care about LLVM support, than we can
keep their back-ends in tree and not have to worry about them.

IIRC, our *only* rule for a long time has been "keep up or give up".
Now, how do we know when a target is good for moving out of
experimental? And what is the meaning of experimental anyway?

Maybe we should separate the discussion from the actual change. So, if
you could comment on the review D22753 about which of the points you'd
consider mandatory and which are nice-to-have, that'd probably be
easiest.

About the Lanai back-end being official, I have no reservations. But I
was the only one to say anything, so I'll wait for others to have
their say on the review before I put my approval.

Just a few unrelated comments below... :slight_smile:

Even for fairly pedestrian backends such as AArch64 and PowerPC, we
routinely have people ask active developers on those backends to do the
triaging of issues.

I think that's most for specific environment than anything else.

Really, you can get sub-$100 AArch64 1-day delivery and a *very*
decent open source emulator (QEMU), both system and user level.

And I don't think any of these backends are bad or should be removed. I
rather like several of them. =] I just think that sufficient community
involvement makes the availability of hardware/emulator "nice to have" but
not essential, and it makes the ISA documentation substantially less
important.

I also don't think emulators replace the ISA in any way, but I get
what you mean. Being nice-to-have, either one would be better than
nothing, but having both would be really optimal.

Between the two, I'd personally prefer the ISA (if it was correct and
complete) than a simulator. But that's me.

cheers,
--renato

From: "Renato Golin via llvm-dev" <llvm-dev@lists.llvm.org>
To: "Chandler Carruth" <chandlerc@google.com>
Cc: "LLVM Developers" <llvm-dev@lists.llvm.org>
Sent: Monday, July 25, 2016 3:25:40 PM
Subject: Re: [llvm-dev] [RFC] Make Lanai backend non-experimental

Hi Chandler,

I think you have good points. Maybe we could make some hard-lined
rules and others as "nice-to-have".

The biggest problem is the community behind it, outside of LLVM. If
the community is strong, and they care about LLVM support, than we
can
keep their back-ends in tree and not have to worry about them.

IIRC, our *only* rule for a long time has been "keep up or give up".
Now, how do we know when a target is good for moving out of
experimental? And what is the meaning of experimental anyway?

Maybe we should separate the discussion from the actual change. So,
if
you could comment on the review D22753 about which of the points
you'd
consider mandatory and which are nice-to-have, that'd probably be
easiest.

About the Lanai back-end being official, I have no reservations. But
I
was the only one to say anything, so I'll wait for others to have
their say on the review before I put my approval.

It's been actively maintained, and does not seem to have caused any particular difficulties for the rest of us. It is fine by me too.

Just a few unrelated comments below... :slight_smile:

> Even for fairly pedestrian backends such as AArch64 and PowerPC, we
> routinely have people ask active developers on those backends to do
> the
> triaging of issues.

I think that's most for specific environment than anything else.

Really, you can get sub-$100 AArch64 1-day delivery and a *very*
decent open source emulator (QEMU), both system and user level.

I see your point, but there can also be substantial overhead in person-hours (often which far exceeds the actual hardware costs). Especially for embedded/emulated systems, the turn-around time on builds can be really long. Setting up software environments on these systems can also be time consuming. Learning about the hardware can take a long time.

-Hal

I agree, but it isn’t completely unreasonable to expect a simulator to be available.

-Chris

> Presumably if my out-of-tree backend was to be pushed to LLVM, it too
would be considered experimental.

Yes. Though, not all out-of-tree back-ends end up upstream for
different reasons.

A few basic rules to get accepted are if:
* the target exists and can be easily purchased / emulated for
investigating problems,
* there are official documents / specs published by the project /
company that maintains the targets,

Most of your list makes a lot of sense to me Renato, but these two items
don't really.

I think a large number of targets in tree would be *extremely* hard to
purchase and directly investigate problems. As examples:

- Hexagon
- SystemZ
- XCore

I'm sure it is theoretically possible for me to acquire such a system and
test on it, but the amount of time and energy it would take make it a
practical impossibility.

Even for fairly pedestrian backends such as AArch64 and PowerPC, we
routinely have people ask active developers on those backends to do the
triaging of issues. I've never seen this be a problem in practice, and I'm
not convinced that it is something we should actually require to have an
upstream backend.

As I mentioned in the original thread "[llvm-dev] [RFC] Lanai backend", it
seems to me that the "can I run the code coming out of the backend?"
property is more about enabling LLVM's users to do interesting things with
the backend, rather than about LLVM's internal maintenance.

http://lists.llvm.org/pipermail/llvm-dev/2016-February/095214.html

I'm not sure how that affects the decision.

My gut (and I'm actually not sure if I exactly consider this a pro or a con
personally) is that is sounds Stallman-esque to require backends to answer
"yes" to "can I run the code coming out of the backend?". If you squint it
is sort of like an anti-tivoization requirement, which would be a very
ironic requirement for a BSD-licensed project.

One example that personally convinces me that "can I run the code coming
out of the backend?" should not be a requirement is the following: if Apple
is willing to develop their mobile GPU backend(s?) upstream, but still keep
their actual driver stack and hardware locked down, I think it would be a
net win for the community to bring it into tree.

-- Sean Silva

I’m sure it is theoretically possible for me to acquire such a system and test on it, but the amount of time and energy it would take make it a practical impossibility.

Even for fairly pedestrian backends such as AArch64 and PowerPC, we routinely have people ask active developers on those backends to do the triaging of issues. I’ve never seen this be a problem in practice, and I’m not convinced that it is something we should actually require to have an upstream backend.

As I mentioned in the original thread “[llvm-dev] [RFC] Lanai backend”, it seems to me that the “can I run the code coming out of the backend?” property is more about enabling LLVM’s users to do interesting things with the backend, rather than about LLVM’s internal maintenance.

http://lists.llvm.org/pipermail/llvm-dev/2016-February/095214.html

I’m not sure how that affects the decision.

MHO, but I almost completely disagree with you. I agree that the question comes down to asking how much benefit to the community there is to merging the backend, balances vs how much cost there is (your “internal maintenance” point).

It is unquestionably easier for a contributor to land their backend in-tree than to maintain it out-of-tree. This is because landing it in tree shifts the maintenance burden from the contributor to the community. If there is low value to the community, then this is a "bad deal” for the project as a whole, since there is only so much attention to go around.

My gut (and I’m actually not sure if I exactly consider this a pro or a con personally) is that is sounds Stallman-esque to require backends to answer “yes” to “can I run the code coming out of the backend?”. If you squint it is sort of like an anti-tivoization requirement, which would be a very ironic requirement for a BSD-licensed project.

This seems like some weird FUD or mud slinging argument, I’m not sure what to make of it.

-Chris