moving the clang-omp merge along

Andrey Bokhanko expressed interest in getting the clang-omp merge done in time for the 3.5 release but wants guidance on the process. I suggested starting with the creation a new clang-omp branch upstream rebased on clang trunk for generation of merge patch. Unfortunately merging the current changes from the clang-omp (based on clang 3.4) to a clang-omp (based on clang trunk) looks very difficult as selective patches have been committed into clang trunk from clang-omp and don’t appear to have been kept synchronized with the current changes from upstream. Does anyone know if these new files from previous commits out of clang-omp contain any local changes which haven’t been back ported to clang-omp? It would seem that postponing this merge will just make the process harder as time goes on if selective merges from clang-omp into clang trunk continue in the interim. Hopefully the folks who did the original selective commits would help detangle this mess.
Jack

I would strongly recommend that you get your current branch in sync with clang-TOT as a first step. Once this done, you should separate individual patches and submit them for review. Based on previous history, the community is unlikely to accept a single massive change set.

p.s. The tone of your last sentence is less than ideal. These are the folks actually working on getting the work you are interested merged into upstream. You should thank them, not critique them. (I’m not one of them, btw.)

Philip

I think a bunch of the confusion comes from phrasing this as a merge. It
isn't.

Andrey and others are working hard to contribute the functionality to
Clang, but the github project you're referencing is not a branch of Clang,
and the code there which isn't also in Clang's trunk hasn't been
contributed to Clang and isn't even covered by the LLVM project... This
means that it hasn't been code reviewed, etc.

The Intel folks are contributing small incremental patches to Clang and
they are being reviewed in the normal process. This is IMO the best way to
add new features to Clang and has been standard practice for some time. I
don't think there is some other process which should be followed instead
because the release date is getting close, and I also don't think there is
any other process that we *could* follow given that the branch isn't part
of the Clang project... If the contributions arrive, get reviewed, and are
committed in time for the release, awesome. If not, there will be another
release after that. =]

    Andrey Bokhanko expressed interest in getting the clang-omp
    merge done in time for the 3.5 release but wants guidance on the
    process.

I think a bunch of the confusion comes from phrasing this as a merge. It isn't.

Andrey and others are working hard to contribute the functionality to Clang, but the github project you're referencing is not a branch of Clang, and the code there which isn't also in Clang's trunk hasn't been contributed to Clang and isn't even covered by the LLVM project... This means that it hasn't been code reviewed, etc.

The Intel folks are contributing small incremental patches to Clang and they are being reviewed in the normal process. This is IMO the best way to add new features to Clang and has been standard practice for some time. I don't think there is some other process which should be followed instead because the release date is getting close,

I second this. If you look at the OMP reviews so far they've uncovered significant improvements pre-commit that wouldn't have been pointed out after the fact.

and I also don't think there is any other process that we *could* follow given that the branch isn't part of the Clang project... If the contributions arrive, get reviewed, and are committed in time for the release, awesome. If not, there will be another release after that. =]

If anything it would have been nice to have discussed the overall design of the OMP frontend earlier in development. The fact the patches are landing broadly as written in order to keep it moving is already something of a concession :slight_smile:

Alp.

Philip,
From observing many merges in FSF gcc over the years, it is crazy to take a new branch, selectively pull in small sections and then take long breaks where the two start to rapidly fork. If a branch is to be merged, the process should at least be scheduled such that the process will take place over a known period of time so attempts can be made to keep the two in sync or at least keep track of where the two have begun to diverge. At the moment, there are quite a few files introduced from clang-omp that are no longer in sync and the svn web browser access doesn’t seem to allow you to easily view the commit history on individual files to see if they have been changed since the original commits.
Jack

Philip,
Also, it would be quite unfortunate if the previous partial merge of clang-omp, which I assume was done to make the -fopenmp/-lgomp support in clang trunk more usable, in the end became a major impediment towards getting the clang-omp changes into clang trunk in time for 3.5’s release.
Jack

Philip,
    From observing many merges in FSF gcc over the years, it is crazy to take a new branch, selectively pull in small sections

"Crazy"? Selectively pulling in small chunks is the only realistic way to deal with the task.

and then take long breaks where the two start to rapidly fork. If a branch is to be merged, the process should at least be scheduled such that the process will take place over a known period of time so attempts can be made to keep the two in sync or at least keep track of where the two have begun to diverge. At the moment, there are quite a few files introduced from clang-omp that are no longer in sync and the svn web browser access doesn't seem to allow you to easily view the commit history on individual files

Jack, are you seriously talking about taking on a merge of this scale using the *SVN web interface*.. and then complaining about that to Philip?

There are tools for that: Consider using clang's own format/tooling/refactoring together with git to track upstream changes and automate the sync work for your out-of-tree branch.

It's no wonder you're getting stuck if you were trying to coordinate the effort with WebSVN, after all the one thing you can be sure of is that upstream code won't stay still in the LLVM project.

Alp.

As far as I can tell the commit history of the OPENMP support into clang trunk is represented
by the following commits…

https://github.com/llvm-mirror/clang/commit/9e6a299419ec5b0a4b08ae13d2c4b1170293222b
https://github.com/llvm-mirror/clang/commit/bf9865fc78da180dfca153535e2dee932d023a20
https://github.com/llvm-mirror/clang/commit/2c692c3166dd5d2f667c06cd0b0d47fbea391aa6
https://github.com/llvm-mirror/clang/commit/3390ed3341a0848f37333d55bb34bcff4b9364c2
https://github.com/llvm-mirror/clang/commit/1a751e402717efb1b1321ab310700ce79a302da8
https://github.com/llvm-mirror/clang/commit/60efcfe13b442a0e0b5f9d11a590eb8955687271
https://github.com/llvm-mirror/clang/commit/1d8bd5a9a45f2ce1e362d66c3ca6512b8422a6a5
https://github.com/llvm-mirror/clang/commit/0fbbbd5db8ee5ada8ddd0c401f8057d7d0f4e75f
https://github.com/llvm-mirror/clang/commit/5e162947b3757e51724e7522385a9fdf4511b1b7
https://github.com/llvm-mirror/clang/commit/fad839f4742722fb8346b9677c99f895988ca766
https://github.com/llvm-mirror/clang/commit/fce71a317d3df0b6aac8c2332798842ab9c1d6c2
https://github.com/llvm-mirror/clang/commit/c5ad6c52b912fd63cd772f922aab402b12021c16
https://github.com/llvm-mirror/clang/commit/18414560e68c02039c2574d86c2f884a33b04de8
https://github.com/llvm-mirror/clang/commit/bfa38388de6a8e39aa29284442650f49e3d2d713
https://github.com/llvm-mirror/clang/commit/9292dbd49443621fad5e63ec530bc136561294cd
https://github.com/llvm-mirror/clang/commit/f1e95d44ac2449a5c4285a5754ac033c72b114f6
https://github.com/llvm-mirror/clang/commit/ef95faf680799e3eb69eeddf7630a0cb4a0237f9
https://github.com/llvm-mirror/clang/commit/552b0f284f9d8b6b70ade5545744f82930aad0f9
https://github.com/llvm-mirror/clang/commit/f6e4bb6620b307c877046456b64f26a64dd13250
https://github.com/llvm-mirror/clang/commit/a100391cc47d7e9db05cc239b1919b958da71cb0
https://github.com/llvm-mirror/clang/commit/c814cd660a16bd49b5c9d09b73a56e8b60e99605
https://github.com/llvm-mirror/clang/commit/a465dc43f18a62a73536fb51fbc75172ef4fbbc3
https://github.com/llvm-mirror/clang/commit/567ad56d9f6860293112974319a079c560c31f68
https://github.com/llvm-mirror/clang/commit/68711dd20d21899e758c5f207dacd0cae8c4342e
https://github.com/llvm-mirror/clang/commit/fcd77d033b1a8a688001b8a87ccefc622fb09cc3
https://github.com/llvm-mirror/clang/commit/b293a9a424d50da3ff676493917c134d3789bd21
https://github.com/llvm-mirror/clang/commit/fec65ff9a65e73091d9f8117e6a072b79a6b8363
https://github.com/llvm-mirror/clang/commit/f343a494807eaba4cdc0470ad40719699dc752f3
https://github.com/llvm-mirror/clang/commit/316202f49ae868466ba214b770349dc211d07262
https://github.com/llvm-mirror/clang/commit/a3eb9f5811b85ce1b5158ad6691ea0a61dbb247d
https://github.com/llvm-mirror/clang/commit/67fafb973cfbd0ac08f3177a344e5cdd4fee4ae6
https://github.com/llvm-mirror/clang/commit/2c9a039c9bdfb19c769cd0224e30fc66e5ad49d4
https://github.com/llvm-mirror/clang/commit/0661f6d435fd4f159d7383291816eb90a621cc2d
https://github.com/llvm-mirror/clang/commit/9e4d5da66ba56954f8fa64f59d28f4d2335fd37f
https://github.com/llvm-mirror/clang/commit/15afb8ec13012c6640b683d0f9b831cc08dc3b87
https://github.com/llvm-mirror/clang/commit/271504f86c5b97c62d2fc378b26ece3709f473f1
https://github.com/llvm-mirror/clang/commit/1870c2d2e12b591397f244e9317a5d8fb9ec562d
https://github.com/llvm-mirror/clang/commit/154210ac5cbccd52c070afbd3b1ec86b21afafac
https://github.com/llvm-mirror/clang/commit/8492691ea87c0dc9523b2daf24e6ee7b5cbf442d
https://github.com/llvm-mirror/clang/commit/b23eb1cd646ef10df5e241edae5b31c743f2d9d8
https://github.com/llvm-mirror/clang/commit/7742deb9e03e5129dbe6148f5c9a55af73bd8535
https://github.com/llvm-mirror/clang/commit/7211ea47c1b502966ec02bb023fab59385caa591
https://github.com/llvm-mirror/clang/commit/a582422f20f4c9a7a2ebacabc5b5392b94e3e6b4
https://github.com/llvm-mirror/clang/commit/d195bc38fd424b0c928e3c354038a8ca6e2ccac3
https://github.com/llvm-mirror/clang/commit/d0dbb7e6d4f05f5d0a5978822476897fe3427787
https://github.com/llvm-mirror/clang/commit/543c4ae954f2bce5ac58ed22080f23cbd94794d2
https://github.com/llvm-mirror/clang/commit/0c018357b8bbb1f96bbf622a5807421e626b4228
https://github.com/llvm-mirror/clang/commit/4367829b41e89d2f3dfae94a97af40ffa01c56c9
https://github.com/llvm-mirror/clang/commit/8f1a2db8649eb151ee620273dcf34b700176430f
https://github.com/llvm-mirror/clang/commit/1dc6f745eb19c94527503012d798dc9b9b5ba6da
https://github.com/llvm-mirror/clang/commit/5806bb59d2d19a9b32b739589865d8bb1e2627c5
https://github.com/llvm-mirror/clang/commit/7a8918fe69d419c41956a7245874b0196e03127b
https://github.com/llvm-mirror/clang/commit/4fa7eab771ab8212e1058bd1a91061ff120c8fbb
https://github.com/llvm-mirror/clang/commit/6af701f29be43e49a25ab098c79940ae4cbb69c7
https://github.com/llvm-mirror/clang/commit/c640058aa7f224a71ce3b1d2601d84e1b57f82d3
https://github.com/llvm-mirror/clang/commit/08f8539f0acfc74cc2e04167940bb59a0b582cbf
https://github.com/llvm-mirror/clang/commit/50a70cd11801fd9a700d06e447095249c34c261f

This assumes that everyone has been labeling the comments in their commits with OpenMP.
There was at least one instance where this was misspelled.

All,

To clarify, what I meant is getting OMP runtime library (not clang-omp branch!) as a part of 3.5 release. I specifically referred to this thread: http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-May/thread.html#73025 started by Jack. I thought he means libiomp when saying “openmp support” – apparently, I was mistaken. Hence the confusion.

After so many talks with Chandler, I know very well that upstreaming full OpenMP support is a long way to go. :slight_smile:

Also, the whole desire to put the library into 3.5 release is a responce to the criticism expressed by Chandler in this message: http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20140217/099477.html.

While we are on the topic, let me update on some other things happening in responce to other issues highlighted by Chandler:

  • This library is not being developed as an active part of the LLVM
    community, even if it is checked into SVN as part of the LLVM project and
    under its license. See r197914 where there is a code drop of many months
    worth of development with no change log, attribution, information, or
    other participation in any part of the community.

This is changing, and many developers joining the whole OpenMP in clang support effort. I can say that Michael Wong and many of his colleagues from IBM are involved; Eric Stotzer and his colleagues from TI are involved; Barbara Chapman and her group from UofHouston is involved; Guansong Zhang from AMD is involved. Obviously, Hal Finkel, Chris Bergstrom, Steve Noonan and many others continue to be actively involved as well.

Take a look at the recent activity in openmp-dev mailing list. More to come.

  • There has been essentially zero discussion with the rest of the clang
    or llvm community about this library. There are separate mailing lists
    which have nearly no traffic since the code drop.

Take a look at this month’s messages: http://lists.cs.uiuc.edu/pipermail/openmp-dev/2014-May/thread.html. In general, as more people get interested, more traffic became generated. It’s a chicken and egg problem…

  • There has been no effort to make this library even work properly with
    Clang as a host compiler. See the copious notes that only Clang 3.3 is
    supported, and that not full featured.

It is buildable with clang now. Moreover, regular buildbots, with both gcc and clang, are running:

http://lab.llvm.org:8011/builders/libiomp5-gcc-x86_64-linux-debian
http://lab.llvm.org:8011/builders/libiomp5-clang-x86_64-linux-debian

  • The build system is totally disjoint from LLVM’s, in fact it is an
    entirely custom Perl build system that is unlike anything in use by the
    LLVM project.

We started to work on improving build system. Stay tuned.

  • There are zero tests in the open source repository!!! This is even
    called out in the original submission and on the primary website. We simply
    cannot ship and link against a runtime library which has no tests!

University of Houston contributed OpenUH test suite: http://lists.cs.uiuc.edu/pipermail/openmp-commits/2014-May/000019.html. Sunita Chandrasekaran from the University works on integrating this suite into LLVM test system.

BTW, any advice with how to approach this would be much appreciated!

  • No part of this library has gone through an LLVM release process either,
    not even as a “new” or “beta” project.

Aha! So, you support inclusion of openmp (library, not compiler) in 3.5 as well? :slight_smile:

Andrey

While we are on the topic, let me update on some other things happening in
responce to other issues highlighted by Chandler:

I just wanted to say, I am very happy to see progress starting here. =] All
of my comments below are meant to help the effort move faster and deal
better with the LLVM project.

- This library is not being developed as an active part of the LLVM
community, even if it is checked into SVN as part of the LLVM project and
under its license. See r197914 where there is a code drop of many months
worth of development with *no* change log, attribution, information, or
other participation in any part of the community.

This is changing, and many developers joining the whole OpenMP in clang
support effort. I can say that Michael Wong and many of his colleagues from
IBM are involved; Eric Stotzer and his colleagues from TI are involved;
Barbara Chapman and her group from UofHouston is involved; Guansong Zhang
from AMD is involved. Obviously, Hal Finkel, Chris Bergstrom, Steve Noonan
and many others continue to be actively involved as well.

Take a look at the recent activity in openmp-dev mailing list. More to
come.

Pretty happy about this, and looking forward to more. Some things that
would be helpful (IMO, others may disagree) to start looking into how/when
to adopt or integrate:

- Start more closely following the LLVM developer policy around code review
and small, incremental patches and development.
  - Maybe use code review tools like Phabricator? Dunno what would be
needed to make this work well, probably some stuff...
- Check that the codebase compiles with the same host toolchain baseline as
the rest of the LLVM project[1]
- Maybe work out a plan toward generally conforming with the coding
standards[2] where applicable?

It is buildable with clang now. Moreover, regular buildbots, with both gcc
and clang, are running:

http://lab.llvm.org:8011/builders/libiomp5-gcc-x86_64-linux-debian
http://lab.llvm.org:8011/builders/libiomp5-clang-x86_64-linux-debian

Just want to say, this in particular is fantastic.

- The build system is totally disjoint from LLVM's, in fact it is an
entirely custom Perl build system that is unlike anything in use by the
LLVM project.

We started to work on improving build system. Stay tuned.

Very cool. I would look closely at the compiler-rt and libc++ CMake builds.
Hopefully useful.

- There are *zero tests* in the open source repository!!!! This is even
called out in the original submission and on the primary website. We simply
*cannot* ship and link against a runtime library which has no tests!

University of Houston contributed OpenUH test suite:
http://lists.cs.uiuc.edu/pipermail/openmp-commits/2014-May/000019.html.
Sunita Chandrasekaran from the University works on integrating this suite
into LLVM test system.

BTW, any advice with how to approach this would be *much* appreciated!

I think the right way to go is something along the lines of the ASan (in
compiler-rt) lit tests, but maybe others have a better idea. I agree that
testing here is hard.

- No part of this library has gone through an LLVM release process either,
not even as a "new" or "beta" project.

Aha! So, you support inclusion of openmp (library, not compiler) in 3.5 as
well? :slight_smile:

Hehe, I see what you did there.

I do genuinely support it going through the release process, but I think
that there are two things that need to be pretty solid first:

1) the build system work probably needs to be pretty solid. i'd be happy
with support on par with compiler-rt or libc++ in CMake...
2) the test suite needs to be at least reasonably well integrated so
release testers actually hit it and exercise the code
3) we need a good switch in Clang to use iomp (I know you or someone on the
patch thread worked on this, did it land? if so, done!)

Maybe others have other thoughts, but I think those three are my key ones.

Anyways, thanks for the clarification and all the hard work on this stuff.

Thanks for the summary Andrey.

The underlying problem For the OpenMP *runtime* is really lack of visibility and sidelining on the openmp-dev list.

It's absolutely critical to close down the low-volume openmp-dev list and fold the subscribers into one of the more active mailing lists, either llvm-dev or cfe-dev.

Until that's done the patches won't get review from the mainstream LLVM developer community, build system experts won't join in etc.

Alp.

Alp,
My take on this is that we have a chicken and the egg situation here. As long as the
clang-omp changes haven’t been merged into clang trunk, we will be unable to switch
the default of -fopenmp from -liomp5. So once the clang-omp merge is well underway,
interest in integrating the openmp component into the existing build machinery will
rapidly pick up.
Jack

Andrey,
Thanks for the clarifications. Everyone I asked had been telling me that llvm.org openmp svn was the
development repository for openmp but I thought that was unlikely. Where is that active tree at so we can
monitor for changes that need to be migrated to the llvm.org copy of openmp? In particular, I am interested
in finding the fix failure of…

Testing for “omp_taskyield”:
Generating sources … success
Compiling soures … success
Running test with 8 threads … success …sh: line 1: 87638 Segmentation fault: 11 ./bin/c/ctest_omp_taskyield > bin/c/ctest_omp_taskyield.out
… and verified with 139% certainty

in the OpenMP3.1_Validation test suite on darwin. My understanding is that this is to be fixed in
a new version of openmp but is unclear where to find that copy. Since the upstream developers of
OpenMP3.1_Validation asked for it to be added to the openmp trunk in llvm.org, it would be nice

to fix that for the copy currently checked into llvm.org’s openmp trunk.
I assume you intend to eventually move the clang-omp development completely into clang trunk, no?
That would be a worthwhile goal for clang 3.5 so that you don’t have to constantly rebase the clang-omp
branch to the latest release of clang.
Jack

What he said. :slight_smile: Chandler is doing a better job expressing things the right way, so I’m going to defer to him. Andrey, thanks for the summary. That was very helpful for those of us who haven’t been following closely. p.s. I’m actually thrilled with the idea of having openmp support in clang long term. It just needs to be done right so that it can be maintained going into the future. Philip

From: "Jack Howarth" <howarth.mailing.lists@gmail.com>
To: "Andrey Bokhanko" <andreybokhanko@gmail.com>
Cc: "cfe-dev" <cfe-dev@cs.uiuc.edu>
Sent: Thursday, May 29, 2014 10:36:47 AM
Subject: Re: [cfe-dev] moving the clang-omp merge along

Andrey,
Thanks for the clarifications. Everyone I asked had been telling me
that llvm.org openmp svn was *the*
development repository for openmp but I thought that was unlikely.
Where is that active tree at so we can
monitor for changes that need to be migrated to the llvm.org copy of
openmp? In particular, I am interested
in finding the fix failure of…

Jack,

Let me try to clarify:

- OpenMP language support

   - Intel has released its clang-omp branch to the public. On the clang-omp github page, there are also trunk-rebased repositories that, with some help from Intel, I personally maintain. Nevertheless, this code is not part of the Clang project. We have a process in place, involving extensive code reviews, to ensure implementation quality and maintainability, to help ensure multiple developers are familiar with the different areas of the codebase, etc. that the OpenMP implementation much undergo. This process, considering the limited number of qualified reviewers and the overall quantity of code, is operating at a reasonable pace. Personally, I would love for it to go much faster (recall, *I* maintain the trunk-rebased branch), but that is not necessarily in the best interest of the community. Whether this makes the 3.5 release in total, or takes until 3.6, in the long run, seems to matter very little. There has been steady (and now accelerating) progress toward contributing OpenMP support to Clang itself. Anyone who *needs* OpenMP support now can get it (albeit by applying some extra patches), and the pieces that are in the upstream Clang repository are of notably higher quality than the original code from Intel's clang-omp branch. In the end, everyone will benefit from the improved quality of the implementation.

- OpenMP runtime library

   - This is the piece that is currently on llvm.org as a separate project. The openmp repository on llvm.org *is* *the* repository for this, at least as it exists as an open-source project. I know of no other. Obviously, other entities can have internal branches of any project, as Intel may have in this case, and these internal branches may contain bug fixes not yet contributed to the open source project. When this happens it is unfortunate, and it seems perfectly reasonable to push anyone with such internal fixes to contribute them sooner rather than later. Nevertheless, no one has been withholding otherwise-public code from you -- or, if they have, they've been withholding it from me too :wink: If you'd like to push for such a code release, the openmp-dev list is the appropriate place for this, not here.

I hope this helps.

-Hal

Hal,
I didn’t mean to imply anyone was hiding fixes but whether there multiple repositories for openmp which just weren’t well advertised. In particular, the existence of the https://www.openmprtl.org site and how that site figures in the general transmission of bug fixes between it and the various compiler projects that might use openmp.
Jack

From: "Jack Howarth" <howarth.mailing.lists@gmail.com>
To: "Hal Finkel" <hfinkel@anl.gov>
Cc: "cfe-dev" <cfe-dev@cs.uiuc.edu>, "Andrey Bokhanko" <andreybokhanko@gmail.com>
Sent: Thursday, May 29, 2014 12:12:00 PM
Subject: Re: [cfe-dev] moving the clang-omp merge along

Hal,
I didn't mean to imply anyone was hiding fixes but whether there
multiple repositories for openmp which just weren't well advertised.
In particular, the existence of the https://www.openmprtl.org site
and how that site figures in the general transmission of bug fixes
between it and the various compiler projects that might use openmp.

The releases on the openmprtl are managed by Intel, and the site existed before the library was contributed to the LLVM project, but certainly lag the code in the LLVM openmp repository at this point.

-Hal

Chandler,

Thanks for the encouragement and the additional suggestions! – we can always rely on you to keep us busy. :slight_smile:

  1. we need a good switch in Clang to use iomp (I know you or someone on the patch thread worked on this, did it land? if so, done!)

Yes, this is done – see Alexey’s message.

Andrey

Alp,