Flang landing in the monorepo - next Monday!

Hi All,

As discussed before Christmas, this is a reminder that we intend to
merge flang on the 13th January (next Monday) before the llvm-10 branch.
At the moment I'm proposing to do it at 10am GMT. I can be flexible on
this point if it requires close coordination with anyone in another
timezone, just let me know.

Previous discussion was in [llvm-dev] Flang landing in the monorepo
<http://lists.llvm.org/pipermail/llvm-dev/2019-December/137661.html>.

The current plan of action is summarized at
<https://github.com/flang-compiler/f18/issues/876>.

The result will look a lot like the recent MLIR merge from 24th Dec:
<https://github.com/llvm/llvm-project/commits/0f0d0ed1c78>, commit
0f0d0ed1c78 in the llvm-project monorepo.

Once it is done I'll mail the list. If you want to coordinate with me,
please get in touch.

Regards,

- Peter

Hey Peter - would you be able to link to/describe the history on this process/decision. I can find one old thread where this was first proposed ( http://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html ) with some general positive responses and a lot of questions.

I see there’s a flang-dev list, though I’m not sure where code review is happening (is there flang-commits or equivalent?) as there’s not much chatter on the mailing list that I can see.

& I’m a bit concerned about flang, already a project with no LLVM API usage, it seems, and a community that doesn’t currently look like it has much overlap with the LLVM developer community (a very rough glance at flang-dev participants, but I could be wrong - the LLVM community is large, to be sure) - I’m concerned that this might create a not very well integrated community, as we saw in the past with lldb, for instance.

  • Dave

Hey Peter - would you be able to link to/describe the history on this
process/decision. I can find one old thread where this was first proposed (
http://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html ) with
some general positive responses and a lot of questions.

I see there's a flang-dev list, though I'm not sure where code review is
happening (is there flang-commits or equivalent?) as there's not much
chatter on the mailing list that I can see.

There is a flang-commits. We have everything set up to use Phabricator
as MLIR did.

& I'm a bit concerned about flang, already a project with no LLVM API
usage, it seems, and a community that doesn't currently look like it has
much overlap with the LLVM developer community (a very rough glance at
flang-dev participants, but I could be wrong - the LLVM community is large,
to be sure) - I'm concerned that this might create a not very well
integrated community, as we saw in the past with lldb, for instance.

I think I mentioned this before but some people are waiting for the move
to be able to actually participate in the Flang development. So the
current github contributions do not necessarily reflect the LLVM
developers that will be involved.

On flang-dev [0] we have long term LLVM developers and people that join
the LLVM developers community with the merge of Flang but even those
already actively participated at the LLVM-Dev!

The API usage is a valid concern, among others, which will require code
changes in the near future.

[0] http://lists.llvm.org/pipermail/flang-dev/2019-December/author.html

Thanks,
  Johannes

Hey Peter - would you be able to link to/describe the history on this
process/decision. I can find one old thread where this was first proposed (
http://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html ) with
some general positive responses and a lot of questions.

I see there's a flang-dev list, though I'm not sure where code review is
happening (is there flang-commits or equivalent?) as there's not much
chatter on the mailing list that I can see.

There is a flang-commits. We have everything set up to use Phabricator
as MLIR did.

& I'm a bit concerned about flang, already a project with no LLVM API
usage, it seems, and a community that doesn't currently look like it has
much overlap with the LLVM developer community (a very rough glance at
flang-dev participants, but I could be wrong - the LLVM community is large,
to be sure) - I'm concerned that this might create a not very well
integrated community, as we saw in the past with lldb, for instance.

I think I mentioned this before but some people are waiting for the move
to be able to actually participate in the Flang development. So the
current github contributions do not necessarily reflect the LLVM
developers that will be involved.

I'll add to this that a number of us who have been involved with LLVM for a long time are involved in guiding the Flang project. While it's certainly true that a few of the people working hard on Flang are new to the LLVM community, there is a reasonable overlap with the existing LLVM community in the overall effort.

On flang-dev [0] we have long term LLVM developers and people that join
the LLVM developers community with the merge of Flang but even those
already actively participated at the LLVM-Dev!

The API usage is a valid concern, among others, which will require code
changes in the near future.

[0] http://lists.llvm.org/pipermail/flang-dev/2019-December/author.html

The other thing that I'll add here is that the existing Flang development has been broad and not deep - the development started with the lexical analysis (and preprocessor), then the semantic analysis, and so on. It is only very recently that any work on actual code generation has begun, and there, the initial focus has been on targeting LLVM via MLIR. I'm not saying that there aren't more places to use LLVM APIs (e.g., filesystem abstractions, ADT, etc.) -- there are -- but that's something we're planning to improve over time.

At a high level, however, the whole point of this Flang project is produce an LLVM-community-integrated Fortran frontend for LLVM. The current developers of the project, which is a growing community, are committed to realizing that outcome and are open to feedback on code structure, API usage, and other aspects of integrating with the rest of the LLVM project.

-Hal

Thanks,
  Johannes

Hi Hal,


Hey Peter - would you be able to link to/describe the history on this
process/decision. I can find one old thread where this was first proposed (
[http://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html](http://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html) ) with
some general positive responses and a lot of questions.

I see there's a flang-dev list, though I'm not sure where code review is
happening (is there flang-commits or equivalent?) as there's not much
chatter on the mailing list that I can see.

There is a flang-commits. We have everything set up to use Phabricator
as MLIR did.

& I'm a bit concerned about flang, already a project with no LLVM API
usage, it seems, and a community that doesn't currently look like it has
much overlap with the LLVM developer community (a very rough glance at
flang-dev participants, but I could be wrong - the LLVM community is large,
to be sure) - I'm concerned that this might create a not very well
integrated community, as we saw in the past with lldb, for instance.

I think I mentioned this before but some people are waiting for the move
to be able to actually participate in the Flang development. So the
current github contributions do not necessarily reflect the LLVM
developers that will be involved.

I’ll add to this that a number of us who have been involved with LLVM for a long time are involved in guiding the Flang project. While it’s certainly true that a few of the people working hard on Flang are new to the LLVM community, there is a reasonable overlap with the existing LLVM community in the overall effort.

On flang-dev [0] we have long term LLVM developers and people that join
the LLVM developers community with the merge of Flang but even those
already actively participated at the LLVM-Dev!

The API usage is a valid concern, among others, which will require code
changes in the near future.

[0] [http://lists.llvm.org/pipermail/flang-dev/2019-December/author.html](http://lists.llvm.org/pipermail/flang-dev/2019-December/author.html)

The other thing that I’ll add here is that the existing Flang development has been broad and not deep - the development started with the lexical analysis (and preprocessor), then the semantic analysis, and so on. It is only very recently that any work on actual code generation has begun, and there, the initial focus has been on targeting LLVM via MLIR. I’m not saying that there aren’t more places to use LLVM APIs (e.g., filesystem abstractions, ADT, etc.) – there are – but that’s something we’re planning to improve over time.

At a high level, however, the whole point of this Flang project is produce an LLVM-community-integrated Fortran frontend for LLVM. The current developers of the project, which is a growing community, are committed to realizing that outcome and are open to feedback on code structure, API usage, and other aspects of integrating with the rest of the LLVM project.

I added you to the original thread when I was replying because I knew you were around this effort, it might be nice to see if you can respond to the other thread too.

A lot of my concerns are echoed by David here and I have additional concerns as well. I don’t think the project is ready for inclusion into the main llvm tree as I don’t see the point right now. There’s nothing llvm about it.

-eric

Hi Hal,


Hey Peter - would you be able to link to/describe the history on this
process/decision. I can find one old thread where this was first proposed (
[http://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html](http://lists.llvm.org/pipermail/llvm-dev/2019-February/130497.html) ) with
some general positive responses and a lot of questions.

I see there's a flang-dev list, though I'm not sure where code review is
happening (is there flang-commits or equivalent?) as there's not much
chatter on the mailing list that I can see.

There is a flang-commits. We have everything set up to use Phabricator
as MLIR did.

& I'm a bit concerned about flang, already a project with no LLVM API
usage, it seems, and a community that doesn't currently look like it has
much overlap with the LLVM developer community (a very rough glance at
flang-dev participants, but I could be wrong - the LLVM community is large,
to be sure) - I'm concerned that this might create a not very well
integrated community, as we saw in the past with lldb, for instance.

I think I mentioned this before but some people are waiting for the move
to be able to actually participate in the Flang development. So the
current github contributions do not necessarily reflect the LLVM
developers that will be involved.

I’ll add to this that a number of us who have been involved with LLVM for a long time are involved in guiding the Flang project. While it’s certainly true that a few of the people working hard on Flang are new to the LLVM community, there is a reasonable overlap with the existing LLVM community in the overall effort.

On flang-dev [0] we have long term LLVM developers and people that join
the LLVM developers community with the merge of Flang but even those
already actively participated at the LLVM-Dev!

The API usage is a valid concern, among others, which will require code
changes in the near future.

[0] [http://lists.llvm.org/pipermail/flang-dev/2019-December/author.html](http://lists.llvm.org/pipermail/flang-dev/2019-December/author.html)

The other thing that I’ll add here is that the existing Flang development has been broad and not deep - the development started with the lexical analysis (and preprocessor), then the semantic analysis, and so on. It is only very recently that any work on actual code generation has begun, and there, the initial focus has been on targeting LLVM via MLIR. I’m not saying that there aren’t more places to use LLVM APIs (e.g., filesystem abstractions, ADT, etc.) – there are – but that’s something we’re planning to improve over time.

At a high level, however, the whole point of this Flang project is produce an LLVM-community-integrated Fortran frontend for LLVM. The current developers of the project, which is a growing community, are committed to realizing that outcome and are open to feedback on code structure, API usage, and other aspects of integrating with the rest of the LLVM project.

I added you to the original thread when I was replying because I knew you were around this effort, it might be nice to see if you can respond to the other thread too.

A lot of my concerns are echoed by David here and I have additional concerns as well. I don’t think the project is ready for inclusion into the main llvm tree as I don’t see the point right now. There’s nothing llvm about it.

To elaborate a bit because this almost assuredly comes off poorly (and I apologize):

I am in favor of having a flang front end in tree. I have concerns about the design of flang versus other front ends, the lack of llvm based library use, and a number of other things that I tried to enumerate in previous emails. I don’t know if anything has changed and the responses I got back originally were “we’re going to do it anyway” so it didn’t leave much room for engagement.

Thanks.

-eric

Hi Eric, David, all

I work with Peter at Arm in a team of 5 engineers working on F18, and will continue as it becomes llvm-project/flang. We depend on Fortran support in (PGI/Nvidia) flang for our commercial Fortran compiler and will move over to llvm-project/flang (I’ll call it Flang from now on) when it is ready so we are committed long-term to the technology and the community. We have a further 12 or so engineers working on LLVM core codegen, mostly for Arm’s SVE architecture, and we have a long-term involvement and commitment to the LLVM project as a whole.

I totally agree with your points that Flang as it stands is not fully integrated into LLVM. Hal has explained the issue about Flang not using LLVM APIs for codegen so hopefully you will agree that will get better over time. The next large development in Flang will be to add the lowering code from the syntax tree to MLIR, which is now an LLVM API. This code is being prototyped in a separate branch of Flang [1], but Eric at Nvidia will start to land the changes soon.

I think your point on usage of LLVM data structures and support libraries is spot on and we need to take that on board. Over time we want Flang to use more of the common LLVM infrastructure and I think that being a part of the monorepo should help this to happen.

At Arm, we have worked on integrating Flang into Clang’s driver [2] [3] [4] and we are trying to follow the existing structure of the driver and to take advantage of common functionality as much as possible in that exercise. I should also say that we are working on a patch to Flang that adds testing using llvm-lit/FileCheck in the style of llvm projects [5]. We’ll be adding all tests for the new driver as lit tests. We are also working internally on porting over the existing Flang tests that use bespoke scripts to using lit (this code has not made it to github yet.)

As a community, we are also trying as much as possible to follow the LLVM coding style in Flang [6] and other design principles like modularity so each part of the compiler can be tested in isolation.

Our rationale for contributing Flang to LLVM at this point rather than waiting is so that we can work on integrating it more closely with LLVM as a community effort, in tree. The longer we develop Flang as a separate entity, the more its community ethos, its processes, its code style and structure, etc. will likely diverge from the LLVM norms. We think that the eventual inclusion Flang would be harder on that side even though the code itself may make more use of LLVM APIs and datastructures. I don’t think there is a right or wrong answer but this is the position we support at Arm and we look forward to working with the community to improve Flang in the ways you suggest.

Yours

Rich

[1] https://github.com/schweitzpgi/f18/tree/f18

[2] https://reviews.llvm.org/D63607

[3] https://github.com/flang-compiler/f18/pull/759

[4] https://github.com/flang-compiler/f18/pull/762 and https://github.com/flang-compiler/f18/pull/763

[5] https://github.com/flang-compiler/f18/pull/861

[6] https://github.com/flang-compiler/f18/blob/master/documentation/C%2B%2Bstyle.md

Hi Eric, David, all

I work with Peter at Arm in a team of 5 engineers working on F18, and will continue as it becomes llvm-project/flang. We depend on Fortran support in (PGI/Nvidia) flang for our commercial Fortran compiler and will move over to llvm-project/flang (I’ll call it Flang from now on) when it is ready so we are committed long-term to the technology and the community. We have a further 12 or so engineers working on LLVM core codegen, mostly for Arm’s SVE architecture, and we have a long-term involvement and commitment to the LLVM project as a whole.

I totally agree with your points that Flang as it stands is not fully integrated into LLVM. Hal has explained the issue about Flang not using LLVM APIs for codegen so hopefully you will agree that will get better over time. The next large development in Flang will be to add the lowering code from the syntax tree to MLIR, which is now an LLVM API. This code is being prototyped in a separate branch of Flang [1], but Eric at Nvidia will start to land the changes soon.

I think your point on usage of LLVM data structures and support libraries is spot on and we need to take that on board. Over time we want Flang to use more of the common LLVM infrastructure and I think that being a part of the monorepo should help this to happen.

At Arm, we have worked on integrating Flang into Clang’s driver [2] [3] [4] and we are trying to follow the existing structure of the driver and to take advantage of common functionality as much as possible in that exercise. I should also say that we are working on a patch to Flang that adds testing using llvm-lit/FileCheck in the style of llvm projects [5]. We’ll be adding all tests for the new driver as lit tests. We are also working internally on porting over the existing Flang tests that use bespoke scripts to using lit (this code has not made it to github yet.)

As a community, we are also trying as much as possible to follow the LLVM coding style in Flang [6] and other design principles like modularity so each part of the compiler can be tested in isolation.

Our rationale for contributing Flang to LLVM at this point rather than waiting is so that we can work on integrating it more closely with LLVM as a community effort, in tree. The longer we develop Flang as a separate entity, the more its community ethos, its processes, its code style and structure, etc. will likely diverge from the LLVM norms. We think that the eventual inclusion Flang would be harder on that side even though the code itself may make more use of LLVM APIs and datastructures. I don’t think there is a right or wrong answer but this is the position we support at Arm and we look forward to working with the community to improve Flang in the ways you suggest.

I’ll say, FWIW - that I tend to agree with this, that I’m happy to see this developed further in tree, even if it’s not LLVM-y right now. Having more of the revision history in-tree, bisectable, etc, seems good to me & my biggest concern is around community cross pollination to make sure it doesn’t end up isolated. I know some of the names of the folks working across f18/flang and LLVM & hope that continues/grows (& of course this shouldn’t be dependent on who’s names I recognize or don’t - the community is large enough that there are going to be different fairly non-overlapping subgroups, etc).

(just chiming in here to say I’m personally not opposed to flang going in-tree as-is - my reason for the earlier email was to express a few broad concerns I’d heard from the community, get some other folks in here for visibility & make sure this wasn’t just a “it’s happening because no one’s paying attention” kind of thing)

(as for style - at least a few things that stood out at a glance: a bunch of else-after-return, and a lot of bracing on single line blocks (technically that latter one isn’t written in the style guide, but it’s pretty pervasive across the LLVM project))

(Disclaimer: Not taking sides, I have no stake in Flang or Fortran).

I think David's comparison to LLDB is interesting. It started very
distant, and then with time merged with the codebase and now it's a
full fledged LLVM project.

The current Flang (F18) is meant to be much closer to LLVM than the
previous one, and the whole mindset was afaik to keep it that way. In
the same way that LLDB once was.

I do agree with Hal that there is a small but significant overlap of
communities, and I do agree with Rick that the longer it stays
separate, the harder it will be for that sub-community.

But LLDB started at a time where "being an LLVM project" was mostly
about being in our SVN repo. There was no mono-repo, and the cost of
being there was lower-ish.

I believe that can be solved with build semantics (CMake stuff?), not
a big problem, so the main issue is about community: will the project
move fast enough towards LLVM integration or will it dangle with
downstream implementations and be mostly useless upstream?

I think we have enough people that care about Flang publicly putting
their names forward. This makes me assume the former, so I'm
personally ok with the merge.

If it happens before the current branch or not, will depend on how
fast they're willing to work towards a working Fortran front-end.

I'd assume that, once it's in, the next release (July/20) would have
to have something minimally decent. But that's not a strong opinion,
so, salt & pepper.

cheers,
--renato

FYI to everyone: If you have things that you would like to see done
before a merge of Flang, please reply with as many details as you have
time to provide (and if you have things that you would like to see done
soon, but you're comfortable with them happening after the merge, please
share those items as well). Our Flang community technical call on Monday
morning will be dedicated to forming a concrete plan from any
yet-unaddressed items and making sure that we have a checklist to address.

Thanks again,

Hal

Thanks Hal, having a plan of action will alleviate my concerns completely.

-eric

Hi,

Why is it targeting pre-llvm10 branching? Are we expecting to ship anything in LLVM 10? If so I would have expected it to land much much earlier to be able to stabilize during the development phase long before the branching point.

I was wondering about this too. I'm not necessarily opposed, but
wouldn't it make sense to target landing soon *after* the branch
instead, to give it time to integrate, stabilize, and then ship in the
next release?

Usually, yes.

I'm guessing this is due to downstream implementations wanting to pick
up the current release as base target (which can be done regardless,
but...).

In theory we can hide flang by having to explicitly name it on the
build command, so merging now or later won't make massive practical
change.

I personally don't like rushing things like that, but I'm not strongly
opposed to either, if everyone else is on board with the rush.

cheers,
--renato

Hi all

Thanks for all the replies and engagement on this issue.

First point, given the state of discussions today I would like to propose that we don't start the merge at 10:00 GMT on Monday 13th as proposed and we delay by at least 24 hours until after the scheduled F18 technical call on Monday afternoon.

In order to help compile a plan of action, I've tried to compile a list of the concerns that folks have raised so far.

I think all these have been addressed (please correct me if you think otherwise)
1. Audit trail/visibility of code review [Addressed by Peter - code has been reviewed [a] to F18 coding guidelines [b].
2. Long-term viability of Flang community and overlap with existing LLVM community [Hopefully Hal and Johannes replies and Greg's and Pat's and my reply demonstrate long-term commitment to Flang after upstreaming]
3. Compatibility of license [Addressed by Steve, a recent update has made the licenses compatible [c]]
4. No use of LLVM APIs and so no connection to the project [Addressed by Hal and me - it is the natural next step in development as Flang starts to generate MLIR. Nvidia are working on this now.]

I think these are acknowledged right now and we are actively working on fixes:
5. No use of lit in the regression tests [Arm is working on this]
6. Need to refactor parts of clang driver that can be shared with flang into a separate library [Arm is working on this, but plans to implement a simple driver first before refactoring to better understand the opportunities to do so. See Peter's RFC [d] ]
7. No use of LLVM utilities or standard data structures (Vector, etc.) [I think Pat and Eric have patches ready to go for this?]
  
I think these are only acknowledged, with the intention to remediate post merge, but no concrete plan or owner at this point:
8. Simple deviations from the LLVM coding style
    a. Separating public headers into include/flang
    b. Syntactical things like braces on single line statements, comments on end of namespaces, etc.
    c. .cc file extensions rather than .cpp
9. Bigger deviations from the LLVM coding style that are harder to fix
    a. Early exits and not using else after return, etc.
10. Flang not supporting all the same C++ compilers as the rest of the project (even taking into account C++17 requirement)

As things stand we are not proposing to remediate these (issues 5 and greater) in the code before it is merged to LLVM next week. These issues will be worked on after commit.

Does anyone feel passionately that any of the above points need to be addressed in the code before we can merge?
Does anyone feel there is anything missing from that list?

Given Mehdi and Hans' query about why this needs to go into LLVM10. Unless anyone has any good reason, perhaps we could postpone until after the branch, perhaps by one week to January 20th? That would give us some more time to bottom out what changes need to be made before we can accept the code and give us time to make any changes that block inclusion (assuming they are suitably sized to do in this timescale). As Renato suggests, that would also give us the LLVM11 branching point as a natural deadline to have addressed the larger concerns in our list.

What do people think to that proposal?

Regards
Rich

[a] https://github.com/flang-compiler/f18/pulls
[b] https://github.com/flang-compiler/f18/blob/master/documentation/C%2B%2Bstyle.md
[c] https://github.com/flang-compiler/f18/commit/e06be2faa64a52471b3cfb2829dc05888236aa68
[d] http://lists.llvm.org/pipermail/cfe-dev/2019-June/062669.html

What is the latest/current proposed merge commit showing what will be pushed to the monorepo?

Please see [0] and [1].

If you want a one-liner to obtain the master branch as it would look after the merge, this will create a directory called llvm-project-post-merge:

git clone --single-branch -b rewritten-history-v4-llvm-project-merge https://github.com/peterwaller-arm/f18 llvm-project-post-merge

(single-branch is necessary to avoid pulling down all the other branches I have in there).

Can you make sure to prune/repack the repo before pushing?

My understanding is that when I say "git push", object negotiation happens between my git client and github, and the objects github needs are sent there. I will only be pushing one ref (master), so I don't expect prune will have an effect. I don't see any way to control the repacking on the github side.

It's not github, but gitlab's docs say they repack after all pushes: https://docs.gitlab.com/ee/administration/housekeeping.html

I imagine github do the same. I can't find anything in their docs to that effect that I can link to. I've written to github support to them to ask what to expect. As a quick experiment I compared the size of the packfiles after a clone of each:

840M llvm-project-single-branch/.git
805M f18-post-merge-single-branch/.git

Hmm, the rewritten history branch, with both llvm and f18, is smaller than f18 alone. This suggests to me that the repositories aren't repacked at the same rate.

Please correct me if I'm wrong. I can repack for good measure, but I don't think it will affect anyone other than me.

Are the license header updated to be the LLVM license?
The test don't seem to be lit-based testing: is this part of the TODO list?

Yes on both counts. Please see Richard Barton's email, points 3 and 5 in [2] where he collects a list of concerns like this.

[0] https://github.com/flang-compiler/f18/pull/854#issuecomment-572578180

[1] https://github.com/peterwaller-arm/f18/commits/rewritten-history-v4-llvm-project-merge

[2] http://lists.llvm.org/pipermail/llvm-dev/2020-January/138103.html

Correction: I was trying to say "LLVM+18" is bigger than "LLVM alone",
which is surprising, but could be explained if the former was repacked
by github more recently than the latter :slight_smile:

(Which I could imagine, because pushing 3,000 commits might happen to
trigger some packing heuristics)

Hi all

Apologies, I somehow combined two points in my list while copying from my editor to my email client! Below is the list as I intended, with changes in bold.

Ta
Rich

I think all these have been addressed (please correct me if you think otherwise)

  1. Audit trail/visibility of code review [Addressed by Peter - code has been reviewed [a] to F18 coding guidelines [b].
  2. Long-term viability of Flang community and overlap with existing LLVM community [Hopefully Hal and Johannes replies and Greg’s and Pat’s and my reply demonstrate long-term commitment to Flang after upstreaming]
  3. Compatibility of license [Addressed by Steve, a recent update has made the licenses compatible [c]]
  4. No use of LLVM APIs and so no connection to the project [Addressed by Hal and me - it is the natural next step in development as Flang starts to generate MLIR. Nvidia are working on this now.]

I think these are acknowledged right now and we are actively working on fixes:
5. No use of lit in the regression tests [Arm is working on this]
6. Need to refactor parts of clang driver that can be shared with flang into a separate library [Arm is working on this, but plans to implement a simple driver first before refactoring to better understand the opportunities to do so. See Peter’s RFC [d] ]
7. No integration into the LLVM build system/Cmake [I think Pat and Eric have patches ready to go for this?]

I think these are only acknowledged, with the intention to remediate post merge, but no concrete plan or owner at this point:
8. No use of LLVM utilities or standard data structures
9. Simple deviations from the LLVM coding style
a. Separating public headers into include/flang
b. Syntactical things like braces on single line statements, comments on end of namespaces, etc.
c. .cc file extensions rather than .cpp
10. Bigger deviations from the LLVM coding style that are harder to fix
a. Early exits and not using else after return, etc.
11. Flang not supporting all the same C++ compilers as the rest of the project (even taking into account C++17 requirement)

What is the latest/current proposed merge commit showing what will be pushed to the monorepo?

Please see [0] and [1].

Thanks!

If you want a one-liner to obtain the master branch as it would look after the merge, this will create a directory called llvm-project-post-merge:

git clone --single-branch -b rewritten-history-v4-llvm-project-merge https://github.com/peterwaller-arm/f18 llvm-project-post-merge

(single-branch is necessary to avoid pulling down all the other branches I have in there).

Can you make sure to prune/repack the repo before pushing?

My understanding is that when I say “git push”, object negotiation happens between my git client and github, and the objects github needs are sent there. I will only be pushing one ref (master), so I don’t expect prune will have an effect. I don’t see any way to control the repacking on the github side.

It’s not github, but gitlab’s docs say they repack after all pushes: https://docs.gitlab.com/ee/administration/housekeeping.html

GitHub does not. When we migrated to the monorepo, I asked them to do a one-time repack and the repo shrunk from 1.14GB to 650MB.
Usually with normal git operations, repacking is not necessary / useful, but with large history rewrite it isn’t necessarily the case (in the monorepo migration, it was svn2git)

I imagine github do the same. I can’t find anything in their docs to that effect that I can link to. I’ve written to github support to them to ask what to expect. As a quick experiment I compared the size of the packfiles after a clone of each:

840M llvm-project-single-branch/.git
805M f18-post-merge-single-branch/.git

Hmm, the rewritten history branch, with both llvm and f18, is smaller than f18 alone. This suggests to me that the repositories aren’t repacked at the same rate.

Please correct me if I’m wrong. I can repack for good measure, but I don’t think it will affect anyone other than me.

It seems that the way you processed the repo stayed fairly well packed right now, so there does not seem to be a need for anything here.
When I rewrote the MLIR history the impact of repacking was non negligible (~15% when comparing the size of the .git folder before/after repacking).

Are the license header updated to be the LLVM license?
The test don’t seem to be lit-based testing: is this part of the TODO list?

Yes on both counts. Please see Richard Barton’s email, points 3 and 5 in [2] where he collects a list of concerns like this.

[0] https://github.com/flang-compiler/f18/pull/854#issuecomment-572578180

[1] https://github.com/peterwaller-arm/f18/commits/rewritten-history-v4-llvm-project-merge

[2] http://lists.llvm.org/pipermail/llvm-dev/2020-January/138103.html

Looks great!

Thanks,