RFC: Graduate CIRCT to monorepo?

Hi all,

The CIRCT project, focused on MLIR abstractions for hardware development, has been developed as an LLVM incubator project for almost the last two years. Original proposal In that time, we’ve held weekly open design meetings with between 20 and 30 people meeting notes and CIRCT has become the basis for both commercial and academic projects, e.g. chisel-circt Several presentations at the last LLVM developer meeting give more detail on our progress [keynote] (2021 LLVM Dev Mtg “CIRCT: Lifting hardware development out of the 20th century” - YouTube) - lightning talks - [graph regions] (2021 LLVM Dev Mtg “Representing Concurrency with Graph Regions in MLIR” - YouTube)

After some discussions, the CIRCT developers have come to feel that there would be significant benefits to being part of the Monorepo. Firstly, although we spend effort to track closely with LLVM HEAD, there is a non-trivial cost to reacting to changes after-the-fact. By living in the monorepo, we hope to stay more up-to-date with semi-automatic changes and refactoring in MLIR that may not necessarily be breaking. Secondly, we believe that this will ease the burden on downstream projects, particularly those that depend on both CIRCT and other MLIR out-of-tree projects and live in “LLVM version hell”. CIRCT has reached the point where many projects are leveraging it and would benefit from a ‘rock-solid’ upstream. Thirdly, CIRCT has been driving several aspects of MLIR that have little internal in-tree usage (e.g. graph regions, parallel computation, scalability, python extensions) and having CIRCT live in the monorepo would make these features more immediately exercisable.

The goal of this RFC is to identify potential barriers to living in the monorepo and identify ways to reduce these barriers.

Issues we are aware of
As an LLVM incubator project, CIRCT follows most aspects of the LLVM development process (coding style, code of conduct, configure and build process). However, there are some differences.

  1. CIRCT development generally occurs at HEAD and we expect that this would continue as part of the repo. This would add a significant amount of code using MLIR (approximately 12 dialects + related transformations). We’d need to get some agreement around breaking changes here. Ideally, we’d have a process similar to how Flang works, where MLIR buildbots also configure CIRCT, enabling breakages to be caught early. However, this will probably mean that MLIR developers will also have to build CIRCT on a regular basis and be willing to update CIRCT (or at least coordinate changes). Setting expectations here is certainly important.
  2. CIRCT leverages more github infrastructure than the rest of LLVM. We use github issues to track development. In moving to the monorepo, we’d need to identify a way to migrate our current issues as part of the merge.
  3. CIRCT uses github pull requests to manage code reviews with automated precommit testing through github actions Precommit testing enforces the LLVM coding style prior to checkin, largely making it a non-issue during reviews. Nightly builds are also run against LLVM HEAD to identify build breakages. We’ve found this combination to be quite effective although it is different than Phabricator. Given the lack of movement on the monorepo adopting github pull requests, identifying a way forward here is important. If necessary, we can adopt Phabricator, although this seems like a step backwards in some ways from processes that we have proven effective. It would also require a PR import path, or migration over time. Regardless, even if we allowed PRs in the monorepo in a limited way, we’d need a migration path for existing PRs (although these should probably be scrubbed first).
  4. CIRCT depends on many external packages (various EDA tools, python, tcl, CapnProto). All of these dependencies are optional and basic unit tests are possible without them.

By far the simplest way to merge into the monorepo would be to have CIRCT live as a toplevel project, including the existing history as a single merge commit as recent projects (e.g. BOLT) have done. This would reduce the impact on most existing LLVM/MLIR users who would largely not change their existing flow. The git repo is approximately 1.4 GB.

Comments and Feedback very welcome, particularly if you feel that you would be significantly negatively impacted by this change.

Steve (on behalf of the CIRCT community)

5 Likes

First off, congratulations! It’s great to hear that CIRCT has matured enough to graduate.

To date, updating Flang has for me been a relatively small source of pain, but that’s mostly because Flang’s usage of MLIR is small (~1 dialect and associated transformations) and Flang’s strong adherence to writing idiomatic MLIR. I’m concerned that CIRCT’s twelve (!!) dialects would put a significant burden on MLIR developers especially since MLIR is still not that stable.

Having some sort of process for deprecating APIs or coordinating big changes would be nice. I’m not familiar with the rest of the projects in LLVM but maybe something can be copied from there.

3 Likes

That’s exciting! :slight_smile:

I’m not familiar with the CIRCT codebase, but in general since there is an expectation that the Monorepo continues to build at HEAD it seems that this would effectively be shifting the work to the MLIR developers instead. I’m interested to understand better how much of an impact it would be and how can we ensure to minimize the burden on MLIR development.
I think we (MLIR developers) would also likely expect that CIRCT keeps up with the most recent “good practice” as they evolve within MLIR (for example adopting things like “declarative assembly”, etc.): we’ll need some level of agreement on how to manage technical debt, updating away from deprecated APIs, etc.

I have personally strong concerns about diverging the monorepo development flow, IMO this goes against the whole concept of maintaining the project as a cohesive monorepo in the first place.
I believe it was known when the incubator was started that diverging from the LLVM project’s workflow would make it challenging / painful to graduate into the monorepo.

1 Like

This is a place where the intention of CIRCT has been to look like the dialects in-tree. That said, I’m sure there are isolated places where this isn’t the case, since we don’t have automated ways of ensuring it. One thing that we could undertake as part of this effort is a concerted effort to review the code that is there to correct obviously diverging usage.

1 Like

Certainly… The goal is specifically to avoid dumping a huge amount of additional work on the wider MLIR community. However, I think one thing that would help everyone living downstream of MLIR is a more considered approach to breaking changes. This is probably a topic that goes beyond CIRCT, but I think everyone (in-tree and out-of-tree) would benefit from more structure to the rolling out of big changes. One option might be to move towards “stable periods” and “merge windows” in an effort to isolate big changes.

I think that’s already happening…

Me too. It made alot of sense at the time to experiment with github pull requests. Now that we have a largely successful experiment that shows that adopting pull requests can work, and where the pain points are, I think we (the LLVM community) a left with a hard choice between imperfect alternatives. I’m personally a little disappointed at some of the shortcomings of pull requests and wish that there was a clearer path to improving them. I don’t see Phabricator as being any better, but it seems that until the pain of maintaining it becomes untenable, there’s little appetite for change. CIRCT may simply have to accept moving back to Phab/buildbot as the price of moving into the monorepo.

1 Like

Thanks for starting this rolling! I think there is a lot of value moving CIRCT to the monorepo, not just for CIRCT, but also for MLIR. A few thoughts in no particular order:

CIRCT has an active community, often 30-40 people at the open design meeting spread between academia, industry, and hobbiests. There is considerable interest in the hardware space in consolidating compiler infrastructure. CIRCT’s ODM also serves as a place for cross pollination between projects and has encouraged several projects to try building on CIRCT/MLIR. Because of this interest, CIRCT is serving as infrastructure for an increasing number of projects, and, transitively, MLIR is also.

CIRCT is used for production compilers and infrastructure. Just from my corner of the project, we are building and will be shipping numerous processor designs through CIRCT-based compilers. Other people in the community have other production tooling and use-cases.

As Steve said, CIRCT has driven some aspects of MLIR, and not having heavy users of those features in tree has made it harder to keep them well tuned (e.g. canonicalization and graph-region issues of last year). To the extent MLIR intends to be compiler infrastructure, in-tree CIRCT provides a forcing-function to prevent over-fitting to ML pipelines.

My sense of the consensus of the ODM discussions is that there is a strong preference for github PRs, even with their limitations, but no one considers moving Phabricator a blocking issue. precommit testing through github actions is, I think, one of the things which would be most-missed changing over.

Thanks for getting the ball rolling on this discussion Steve. I agree with the comments above. Here’s one thought about the number of CIRCT dialects and the burden being added to core MLIR maintainers:

We had discussed carving out a clear separation in CIRCT between “production” dialects and passes (e.g. ExportVerilog and the HW/Comb/Seq/SV dialects) and “experimental” dialects (e.g. StaticLogic and FSM). Perhaps this RFC can be a forcing function to do so. If the CIRCT community was able to make such a delineation, would there be a path forward by upstreaming just the “production” dialects?

This seems like a potential improvement to me. There would be less surface area for MLIR developers to maintain, and any added efforts to keep CIRCT building would be scoped to parts used by industry in production. The parts of CIRCT that have reached this level of maturity can enjoy the benefits of being in the monorepo. If there are any modernization efforts required from the CIRCT community before the dialects/passes can be accepted upstream, this limits the scope of what needs to be audited/updated.

For the remaining experimental dialects/passes, I don’t think this is going to make life much harder. The community can keep using llvm/circt to incubate those ideas, and if it comes time to graduate a dialect to the monorepo’s CIRCT codebase, that could follow similar steps as adding any new dialect to upstream MLIR (RFC, discussion, maintenance commitments, etc.).

The main concern I see with having the CIRCT codebase split between the monorepo and llvm/circt is having one foot in the GitHub PR workflow, and one foot in the Phabricator workflow. CIRCT developers who care about both production and experimental things would have to use both, which could be annoying. Personally, having used both, this doesn’t seem like the end of the world.

Are there other concerns with such a split? Could this be a way to make it more palatable to start accepting parts of CIRCT upstream?

Awesome: no concern for me then!

I’d really try to avoid a big split here. I think we should strive to move development upstream rather than splitting our efforts. If we need a way to distinguish expectations about experimental code (i.e.: it’s not widely built and tested, or robust) then I think makes more sense to guard this with a configuration option.

That is fair. That sounds like a good option to me.

One question I have — from the perspective of only having worked on the CIRCT repo and not upstream LLVM/MLIR — is what the release process conventions are/should be for non-LLVM subprojects in the LLVM monorepo. I assume that the MLIR subproject doesn’t branch or release with LLVM.

Are most of the MLIR-based projects built from forks of the LLVM repo, owned by individual companies/organizations? Or are there sometimes shared release branch for each subproject that gets maintained in the upstream LLVM monorepo?

I’m just curious what approaches different groups have tried and whether they’ve learned any best practices for maintaining a non-LLVM, MLIR project within the LLVM monorepo.

Today, in CIRCT, SiFive is just pushing up tags prefixed with sifive/, but I’m wondering if we’ll need to come up with a new release process when moving into the LLVM monorepo.

I’m very happy to see this, thank you for the great summary Stephen!

I’m not up to date on how big git repos are, but this is surprisingly large. Are there some large binary files checked in or something?

I agree with Stephen here. My qualitative sense is that the production parts of CIRCT are overall higher quality than the dialects in the MLIR repository itself (which has accreted a bunch of researchy stuff). It would be good for an experienced MLIR person with fresh eyes to look at the concrete proposal whenever it comes though of course.

FWIW, I agree.

Yes, I think this is a good idea. Instead of graduating the entire repo, we can graduate the stable parts of it, and leave the more experimental pieces (e.g. FSM) in the CIRCT repo, or potentially just delete them if they aren’t used.

I think this is an important choice CIRCT-developers-who-want-to-integrate need to decide, and I only see two reasonable paths here: Either we agree that CIRCT should move to phabricator, or we should wait until the path forward with PRs is discussed and committed. If the LLVM community wants to move to PRs, then it could be great for CIRCT to be a leading experiment in that direction, but we don’t have a committed plan here and it is still controversial.

I don’t want to see the monorepo processes divided unless it is part of a committed transition.

-Chris

2 Likes

MLIR is released with LLVM, but it does not mean much as we don’t really have “release testers” and a process around it. Some people have been trying to use it from the release, but it is still evolving so fast that most are trying to keep up with the main branch (at various frequencies). Other have been branching at an arbitrary point and never updated after that.

It should be possible for a subproject to opt-out of the release branch entirely I think?

Perhaps @stephenneuendorffer accidentally included his build directory or the llvm submodule in that count? I just did a fresh clone of circt.git and it looks like it’s 28 MiB total (17 MiB in the .git/ directory).

Ah, I see. I think CIRCT will be in a similar boat with things evolving fast enough that the LLVM release cadence won’t be sufficient for most users.

It puzzled me to… I went back and looked more closely and it includes the llvm submodule as well :slight_smile:

I have to be honest, I think the monorepo is getting too big and we have to be careful about what we decide to add to it going forward. My personal opinion is that all sub-projects should be a piece of a single larger product e.g. A complete C/C++ (and related languages) toolchain.

Once we enter the territory where we are composing multiple different products out of different slices of the monorepo, I think the negatives of the monorepo outweigh the positives.

So my main question is what are the alternatives? I understand that having CIRCT in the same repo as llvm makes it easier to keep up-to-date with changes to LLVM and MLIR, but is there some other way to achieve this without all the other projects having to take on the burden of a larger git checkout, more commit traffic, more CI noise, etc.

And I don’t mean for this to sound like I’m saying CIRCT would be a burden to the LLVM project, because it works the other way too. If we add CIRCT we are burdening it with all of the noise from the other LLVM sub-projects, which are mostly unrelated. I’m concerned that adding CIRCT to the monorepo will make both projects a little worse off, and I don’t want that, so I would really like to hear what other options there are besides adding CIRCT to the monorepo.

3 Likes

I fully agree with @tstellar. Same concers.

I share the same concern (monorepo is getting too big), too.

1 Like

I’m strongly against having CIRCT in LLVM’s repo. There’s no engineering reason to do so.
The code is not tightly integrated with LLVM, and will never be.
Plus, if a CIRCT buildbot fails after a commit to LLVM’s code, we can’t ask the LLVM dev to fix the issue, right?

If anything, I would remove MLIR (and Bolt) from the repo. These tools are not tightly integrated with LLVM and there’s no engineering reason to keep them in the same repo. It brings extra overhead for everyone without a good reason.

For example, Alive2 lives in a separate repo. It is tightly integrated with LLVM, much more than MLIR or CIRCT. Even so, we live happily outside of the monorepo. The build breaks like once or twice a month because of changes in LLVM, sure. But it is our responsibility to catch up, I’m not going to unload that work on the LLVM dev that broke our build. And Alive2 directly benefits LLVM devs by catching bugs automatically. So having all devs contributing to it makes a lot more sense than fixing CIRCT buildbots.

One thing to keep in mind is that not all devs have fancy workstations. A couple of months ago, with my old computer, I had trouble committing during peak hours because there are so many commits that I couldn’t rebase & push in time before someone else would commit before me. So keeping adding more projects will only make things worse.

The number of downstream LLVM/MLIR projects is huge. Let’s not bring any one them in, please. Because if you bring one you have to bring them all. Let’s not abuse the monorepo. If something can be improved to make the life of downstream folks better, let’s improve, otherwise let’s remove MLIR from the monorepo.

2 Likes

Flang is in tree. It goes from Flang → MLIR → LLVM. There is hope/discussion that maybe Clang will do something similar in the future.