libiomp, not libgomp as default library linked with -fopenmp

Project Infrastructure LLVM Dev List Archives
1

All,

I’d like to resurrect the discussion on replacing libgomp with libiomp as the default OpenMP runtime library linked with -fopenmp.

For reference, the previous discussion is accessible there: http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20140217/thread.html#99461

We are very close to getting full OpenMP 3.1 specification supported in clang (only one (!) clause is not implemented yet, and the patch is already sent for review today: http://reviews.llvm.org/D9370). This implementation generates Intel API library calls; thus, it can’t be used with libgomp and it is simply logical to link a compatible runtime (libiomp) instead.

Last time Chandler objected against doing this, and gave a good (and proper!) scolding. Let me quote his objections along with updates on current state of things:

  • This library is not being developed as an active part of the LLVM community, even if it is checked into SVN as part of the LLVM project and under its license. See r197914 where there is a code drop of many months worth of development with no change log, attribution, information, or other participation in any part of the community.

Now it is actively developed by several members of the community. Changes are committed in small patches. See “openmp-commits” mailing list (http://lists.cs.uiuc.edu/pipermail/openmp-commits/) for details.

  • There has been essentially zero discussion with the rest of the clang or llvm community about this library. There are separate mailing lists which have nearly no traffic since the code drop.

Nowadays the traffic, while less than on llvm-dev mailing list (understandably so :-)) is far from being “zero”. Discussions happen all the time. See the list archives for details (http://lists.cs.uiuc.edu/pipermail/openmp-dev/).

  • There has been no effort to make this library even work properly with Clang as a host compiler. See the copious notes that only Clang 3.3 is supported, and that not full featured.

This is fixed.

  • The build system is totally disjoint from LLVM’s, in fact it is an entirely custom Perl build system that is unlike anything in use by the LLVM project.

The build system is full re-written and now cmake-based. The last remaining step (integration into overall llvm build system) is about to be done by any day now (http://lists.cs.uiuc.edu/pipermail/openmp-dev/2015-April/thread.html#500).

  • There are zero tests in the open source repository!!! This is even called out in the original submission and on the primary website. We simply cannot ship and link against a runtime library which has no tests!

UofHouston contributed a comprehensive test suite: http://llvm.org/viewvc/llvm-project/openmp/trunk/testsuite/.

  • No part of this library has gone through an LLVM release process either, not even as a “new” or “beta” project.

The whole library is a part of two last llvm releases: 3.5 and 3.6.

Thus, I believe all of Chandler’s concerns are resolved, and it is finally a good time time to switch to libiomp as default OpenMP library.

Anyone who supports this?

Anyone who disagrees?

Chandler?

Yours,
Andrey Bokhanko

2

From: "Andrey Bokhanko" <andreybokhanko@gmail.com>
To: "cfe-dev" <cfe-dev@cs.uiuc.edu>, "LLVM Developers Mailing List" <llvmdev@cs.uiuc.edu>, "Douglas Gregor"
<dgregor@apple.com>, "Hal Finkel" <hfinkel@anl.gov>, "C Bergström" <cbergstrom@pathscale.com>, "Michael Wong"
<fraggamuffin@gmail.com>, "Alexey Bataev" <a.bataev@gmx.com>
Sent: Thursday, April 30, 2015 8:49:30 AM
Subject: libiomp, not libgomp as default library linked with -fopenmp

All,

I'd like to resurrect the discussion on replacing libgomp with
libiomp as the default OpenMP runtime library linked with -fopenmp.

For reference, the previous discussion is accessible there:
http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20140217/thread.html#99461

We are very close to getting *full* OpenMP 3.1 specification
supported in clang (only one (!) clause is not implemented yet, and
the patch is already sent for review today:
⚙ D9370 [OPENMP] Codegen for 'firstprivate' clause in 'task' directive. ). This implementation generates Intel
API library calls; thus, it can't be used with libgomp and it is
simply logical to link a compatible runtime (libiomp) instead.

To be clear, this is now LLVM's OpenMP runtime (not just Intel's), and has been ported to several platforms in addition to x86 (PowerPC, ARM).

Last time Chandler objected against doing this, and gave a good (and
proper!) scolding. Let me quote his objections along with updates on
current state of things:

- This library is not being developed as an active part of the LLVM
community, even if it is checked into SVN as part of the LLVM
project and under its license. See r197914 where there is a code
drop of many months worth of development with *no* change log,
attribution, information, or other participation in any part of the
community.

Now it is actively developed by several members of the community.
Changes are committed in small patches. See "openmp-commits" mailing
list ( http://lists.cs.uiuc.edu/pipermail/openmp-commits/ ) for
details.

- There has been essentially *zero* discussion with the rest of the
clang or llvm community about this library. There are separate
mailing lists which have nearly no traffic since the code drop.

Nowadays the traffic, while less than on llvm-dev mailing list
(understandably so :-)) is far from being "zero". Discussions happen
all the time. See the list archives for details (
http://lists.cs.uiuc.edu/pipermail/openmp-dev/ ).

- There has been no effort to make this library even work properly
with Clang as a host compiler. See the copious notes that only Clang
3.3 is supported, and that not full featured.

This is fixed.

- The build system is totally disjoint from LLVM's, in fact it is an
entirely custom Perl build system that is unlike anything in use by
the LLVM project.

The build system is full re-written and now cmake-based. The last
remaining step (integration into overall llvm build system) is about
to be done by any day now (
http://lists.cs.uiuc.edu/pipermail/openmp-dev/2015-April/thread.html#500
).

- There are *zero tests* in the open source repository!!!! This is
even called out in the original submission and on the primary
website. We simply *cannot* ship and link against a runtime library
which has no tests!

UofHouston contributed a comprehensive test suite:
http://llvm.org/viewvc/llvm-project/openmp/trunk/testsuite/ .

- No part of this library has gone through an LLVM release process
either, not even as a "new" or "beta" project.

The whole library is a part of two last llvm releases: 3.5 and 3.6.

Thus, I believe all of Chandler's concerns are resolved, and it is
finally a good time time to switch to libiomp as default OpenMP
library.

Anyone who supports this?

I support this. Linking to libgomp will still be available by specifying -fopenmp=libgomp for those who need it. Our runtime supports a libgomp-compatible interface, so that's hopefully not necessary even for those with GCC-compiled OpenMP object files and libraries.

We need to get the build system integration committed and the buildbots updated to compile it, and the test suite setup (the build system integration and test-suite setup have patches under review on the openmp list, so I suspect this will happen soon). Practically, since we don't have OpenMP in the test suite (yet), none of these are really inflexible prerequisites. In my opinion, as soon as the build system integration is done (so that people have an easy way to use LLVM's runtime), we should flip the defaults -- the other things will follow shortly.

-Hal

3

The current test suite status on x86_64-apple-darwin14, with the proposed patch from http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20150427/128219.html applied to current cfe trunk, is quite respectable…

Summary:
S Number of tested Open MP constructs: 62
S Number of used tests: 123
S Number of failed tests: 16
S Number of successful tests: 107
S + from this were verified: 106

Normal tests:
N Number of failed tests: 8
N + from this fail compilation: 1
N + from this timed out 0
N Number of successful tests: 54
N + from this were verified: 54

Orphaned tests:
O Number of failed tests: 8
O + from this fail compilation: 0
O + from this timed out 0
O Number of successful tests: 53
O + from this were verified: 52

for ‘make ctest’ using “-fopenmp=libiomp5 -Xclang -fopenmp=libiomp5 -L/sw/opt/llvm-3.7.0/lib -lm -O3”. Whereas for libgomp, the results are pitiful…

Summary:
S Number of tested Open MP constructs: 62
S Number of used tests: 123
S Number of failed tests: 30
S Number of successful tests: 93
S + from this were verified: 12

Normal tests:
N Number of failed tests: 15
N + from this fail compilation: 0
N + from this timed out 0
N Number of successful tests: 47
N + from this were verified: 6

Orphaned tests:
O Number of failed tests: 15
O + from this fail compilation: 0
O + from this timed out 0
O Number of successful tests: 46
O + from this were verified: 6

when using “-fopenmp=libgomp -Xclang -fopenmp=libgomp -L/sw/lib/gcc5/lib -lm -O3 -Wl,-no_compact_unwind”. So we can’t get much more broken that the current behavior with libgomp.
Jack

4

It doesn't really feel that way. I proposed a cmake patch and the only
person to review or comment was Intel. (This is coming from the person
who ported it to ARM)

To stay more agnostic I'd love to see a non-Intel owner. While Hal may
not be the most active contributor - his reviews are invaluable and
less biased. I don't know if Hal has the time or interest, but I'd
nominate him for "owner". I see Tom wants to assign more owners, but
I'd like to avoid this being an "Intel runtime owned and controlled by
Intel"

5

What results to you get from "-fopenmp=libgomp -Xclang -fopenmp=libgomp" on
the OpenMP3.1_Validation test suite on x86_64 linux? On x86_64 darwin using
libgomp, barely anything passes the verification unlike libiomp5 which only
has one passed test which doesn't verify. Isn't that a serious defect for
libgomp compared to libiomp5?

6

My apologies. I fully agree to switch the libs and I replied to the
wrong thread.

7

I definitely support Andrey’s proposal to move the default to libiomp. Outside of the known ports for x86, ARM and PowerPC, I’m aware of at least one other port in progress. More will likely follow.

Per Hal’s notes, given the forthcoming commits of the CMake build system, flipping the default to libiomp may help spur/accelerate the progress of porting the runtime to new platforms (especially those that are already supported as LLVM backends).

--John D. Leidel

8

     The current test suite status on x86_64-apple-darwin14, with the
proposed patch from
http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20150427/128219.html
applied to current cfe trunk, is quite respectable....

Summary:
S Number of tested Open MP constructs: 62
S Number of used tests: 123
S Number of failed tests: 16
S Number of successful tests: 107
S + from this were verified: 106

Normal tests:
N Number of failed tests: 8
N + from this fail compilation: 1
N + from this timed out 0
N Number of successful tests: 54
N + from this were verified: 54

Orphaned tests:
O Number of failed tests: 8
O + from this fail compilation: 0
O + from this timed out 0
O Number of successful tests: 53
O + from this were verified: 52

for 'make ctest' using "-fopenmp=libiomp5 -Xclang -fopenmp=libiomp5
-L/sw/opt/llvm-3.7.0/lib -lm -O3". Whereas for libgomp, the results are
pitiful....

Summary:
S Number of tested Open MP constructs: 62
S Number of used tests: 123
S Number of failed tests: 30
S Number of successful tests: 93
S + from this were verified: 12

Normal tests:
N Number of failed tests: 15
N + from this fail compilation: 0
N + from this timed out 0
N Number of successful tests: 47
N + from this were verified: 6

Orphaned tests:
O Number of failed tests: 15
O + from this fail compilation: 0
O + from this timed out 0
O Number of successful tests: 46
O + from this were verified: 6

For additional context, the OpenMP3.1_Validation results for libgomp using
the FSF gcc 5.1.0 compiler on x86_64-apple-darwin14 are...

Summary:
S Number of tested Open MP constructs: 62
S Number of used tests: 123
S Number of failed tests: 7
S Number of successful tests: 116
S + from this were verified: 95

Normal tests:
N Number of failed tests: 3
N + from this fail compilation: 0
N + from this timed out 0
N Number of successful tests: 59
N + from this were verified: 48

Orphaned tests:
O Number of failed tests: 4
O + from this fail compilation: 0
O + from this timed out 0
O Number of successful tests: 57
O + from this were verified: 47

So the current results for libiomp5 are very close to matching those for
the FSF gcc compiler with libgomp in terms of the number of passed tests
while libiomp5 already exceeds libgomp for the number of verified tests.

when using "-fopenmp=libgomp -Xclang -fopenmp=libgomp -L/sw/lib/gcc5/lib

9

It looks like openmp/runtime/README.txt could use an update to clarify
this status. The title is "README for the Intel(R) OpenMP* Runtime
Library" and there's a of supported Intel architectures -- this gives
a somewhat different impression than what's described in this thread.

That said, I would also be happy to see the default switched to libiomp.

10

For comparison, the OpenMP3.1_Validation results on x86_64-apple-darwin14 for original clang-omp 3.5.2 are…

11

All,

I’d like to resurrect the discussion on replacing libgomp with libiomp as the default OpenMP runtime library linked with -fopenmp.

Just for the record, I’m really excited to see this. =]

We are very close to getting full OpenMP 3.1 specification supported in clang (only one (!) clause is not implemented yet, and the patch is already sent for review today: http://reviews.llvm.org/D9370). This implementation generates Intel API library calls; thus, it can’t be used with libgomp and it is simply logical to link a compatible runtime (libiomp) instead.

Is there no way to support libgomp here as well? I don’t say this to hold up changing the defaults in any way, just curious. =]

Also, for the record, I’m really excited to see the progress here as well.

Chandler?

Hi! ;]

I totally agree, I think things are way better now. I generally support the direction. I think there are a few things I’d suggest we do as part of the process, but I think these are really small and just about “how” we switch.

  1. I completely agree with the comments some others have made about us needing to make it clear that this isn’t some Intel-only thing, its the LLVM OpenMP runtime. Some suggestions that I think would make sense to help here:
  • I agree with finding some non-Intel folks to add as explicit code owners. I don’t know who has been sufficiently involved, but if Hal makes sense, awesome.
  • Clearly updating the readme and such would be appropriate.
  • I suspect we should change the name of the installed library. ‘libiomp’ is pretty clearly the Intel library. We could continue in the grand tradition of LLVM naming conventions and use ‘libllomp’? Of course, we should install symlinks under the name ‘libiomp’ if needed for existing users to not be broken.
  • Any other changes?

  1. I think we need to update the instructions for checking out LLVM and all the tools to include checking out the openmp project. I’m planning to try it out in a bit.

  2. It would be nice to get at least one boring benchmark into the test-suite that uses OpenMP just so there’s more coverage that the basic stuff all works. In particular, if we could get the benchmark that Phoronix and others keep pointing at, that’d be nice.

Speaking of which, have you checked the performance of some of the basic benchmarks using OpenMP with the two runtimes? Or looked at Clang vs GCC there? I’d be interested to see the numbers.

12

I'm trimming everything - sorry too hard to manage the thread in my
current email client

13

All,

I'd like to resurrect the discussion on replacing libgomp with libiomp as
the default OpenMP runtime library linked with -fopenmp.

Just for the record, I'm really excited to see this. =]

We are very close to getting *full* OpenMP 3.1 specification supported in
clang (only one (!) clause is not implemented yet, and the patch is already
sent for review today: ⚙ D9370 [OPENMP] Codegen for 'firstprivate' clause in 'task' directive.). This
implementation generates Intel API library calls; thus, it can't be used
with libgomp and it is simply logical to link a compatible runtime
(libiomp) instead.

Is there no way to support libgomp here as well? I don't say this to hold
up changing the defaults in any way, just curious. =]

Also, for the record, I'm really excited to see the progress here as well.

Chandler?

Hi! ;]

I totally agree, I think things are way better now. I generally support
the direction. I think there are a few things I'd suggest we do as part of
the process, but I think these are really small and just about "how" we
switch.

1) I completely agree with the comments some others have made about us
needing to make it clear that this isn't some Intel-only thing, its the
LLVM OpenMP runtime. Some suggestions that I think would make sense to help
here:
- I agree with finding some non-Intel folks to add as explicit code
owners. I don't know who has been sufficiently involved, but if Hal makes
sense, awesome.
- Clearly updating the readme and such would be appropriate.
- I suspect we should change the name of the installed library. 'libiomp'
is pretty clearly the Intel library. We could continue in the grand
tradition of LLVM naming conventions and use 'libllomp'? Of course, we
should install symlinks under the name 'libiomp' if needed for existing
users to not be broken.
- Any other changes?

2) I think we need to update the instructions for checking out LLVM and
all the tools to include checking out the openmp project. I'm planning to
try it out in a bit.

3) It would be nice to get at least one boring benchmark into the
test-suite that uses OpenMP just so there's more coverage that the basic
stuff all works. In particular, if we could get the benchmark that Phoronix
and others keep pointing at, that'd be nice.

Attached are two logs, taskbench_clang_3.7.log and taskbench_gcc_5.1.log,

with the output from the task_bench, task_firstpriv_bench and
taskwait_bench benchmarks run on x86_64-apple-darwin14 using an early 2008
MacPro with dual 2.8 GHz quad-Xeon processors and 10 Gb of memory. The
clang 3.7svn compiler was current trunk with the remaining two OPENMP
patches on cfe-commits added to the build. The gcc compiler is release
5.1.0. In both cases, the -O3 -fopenmp flags were used. The
taskbench-1.0-20110715 release was used for the benchmark sources.

Speaking of which, have you checked the performance of some of the basic

taskbench_clang_3.7.log (9.98 KB)

taskbench_gcc_5.1.log (9.68 KB)

14

All,

I'd like to resurrect the discussion on replacing libgomp with libiomp as
the default OpenMP runtime library linked with -fopenmp.

Just for the record, I'm really excited to see this. =]

We are very close to getting *full* OpenMP 3.1 specification supported in
clang (only one (!) clause is not implemented yet, and the patch is already
sent for review today: ⚙ D9370 [OPENMP] Codegen for 'firstprivate' clause in 'task' directive.). This
implementation generates Intel API library calls; thus, it can't be used
with libgomp and it is simply logical to link a compatible runtime
(libiomp) instead.

Is there no way to support libgomp here as well? I don't say this to hold
up changing the defaults in any way, just curious. =]

Also, for the record, I'm really excited to see the progress here as well.

Chandler?

Hi! ;]

I totally agree, I think things are way better now. I generally support
the direction. I think there are a few things I'd suggest we do as part of
the process, but I think these are really small and just about "how" we
switch.

1) I completely agree with the comments some others have made about us
needing to make it clear that this isn't some Intel-only thing, its the
LLVM OpenMP runtime. Some suggestions that I think would make sense to help
here:
- I agree with finding some non-Intel folks to add as explicit code
owners. I don't know who has been sufficiently involved, but if Hal makes
sense, awesome.
- Clearly updating the readme and such would be appropriate.
- I suspect we should change the name of the installed library. 'libiomp'
is pretty clearly the Intel library. We could continue in the grand
tradition of LLVM naming conventions and use 'libllomp'? Of course, we
should install symlinks under the name 'libiomp' if needed for existing
users to not be broken.
- Any other changes?

2) I think we need to update the instructions for checking out LLVM and
all the tools to include checking out the openmp project. I'm planning to
try it out in a bit.

3) It would be nice to get at least one boring benchmark into the
test-suite that uses OpenMP just so there's more coverage that the basic
stuff all works. In particular, if we could get the benchmark that Phoronix
and others keep pointing at, that'd be nice.

Speaking of which, have you checked the performance of some of the basic
benchmarks using OpenMP with the two runtimes? Or looked at Clang vs GCC
there? I'd be interested to see the numbers.

Another set of benchmark numbers are attached in
openmpbench_C_v3_clang-3.7.log and openmpbench_C_v3_gcc_5.1.log using the
schedbench, syncbench and taskbench compiled with the default -O1
optimization using the clang 3.7svn trunk with the two remaining OPENMP
patches from cfe-commits and the gcc 5.1.0 release compiler. The
OMP_NUM_THREADS=4 is set before each set of runs which were performed on
x86_64-apple-darwin14 using a early 2008 MacPro with dual 2.8 GHz quad-core
Xeon processors and 10 Gb of memory. The benchmark is detailed at
https://www.epcc.ed.ac.uk/research/computing/performance-characterisation-and-benchmarking/epcc-openmp-micro-benchmark-suite
.

openmpbench_C_v3_clang-3.7.log (17.9 KB)

openmpbench_C_v3_gcc_5.1.log (18.1 KB)

15

While OpenMP is very explicit in nature - some compilers may not turn
on advanced analysis and become really aggressive with performance
until -O2 or -O3 (all depends on the compiler).

Benchmarks at -O1 may not have any value.. be careful of using weak comparisons.

16

Hal,

17

Jack,

Thanks for running the test suite!

I wonder how a single test passed with libgomp, given that no real OMP runtime calls are generated if clang targets libgomp… My educated guess is that compiler simply ignores all OMP pragmas, a test still compiles and runs correctly (which is a property of OMP specification) and the test says “oh! omp parallel for is working just fine!”.

Andrey

18

Adding openmp-dev list and Andrey [Churbanov].

Andrey, please see Ed’s message below.

Andrey

19

Chandler,

Thanks for the reply – I always included you in libiomp supporters camp; it is good to see I wasn’t mistaken! :wink:

20

It looks like openmp/runtime/README.txt could use an update to clarify
this status. The title is "README for the Intel(R) OpenMP* Runtime
Library" and there's a of supported Intel architectures -- this gives
a somewhat different impression than what's described in this thread.

Of course we'd be very happy to accept patches that update the document to include the relevant support matrices for non-Intel processors (and to change the misleading documentation titles).

Since this *is* a collaborative development, I'd hope that IBM can contribute information
for their machines, since they did the port, and any input from the folks who worked on the ARM code would also be great.

In the more general discussion, I think one critical point was made less forcefully than it merits, which is that the LLVN OpenMP runtime (libiomp5) *is binary compatible* with the GCC one (libgomp); libiomp5 includes the entrypoints used by GCC. Therefore replacing libgomp in the distribution with libiomp5 should not cause problems for people who are using the libgomp interfaces. Their code should continue to link and run without a problem.

This compatibility means that you can link code compiled with gcc -fopenmp against the LLVM runtime (and mix it with code compiled by clang's OpenMP support). What you *cannot* do is compile code with clang/Openmp and link it against the GCC OpenMP runtime.

So, replacing the GCC OpenMP runtime as the default runtime for the clang distribution should not cause any compatibility issues for anyone who was using the libgomp that was previously distributed, but does ensure that the runtime is compatible with LLVM. (Which is the whole point, surely!)

(Full disclosure: There is one incompatibility between the libraries because the LLVM runtime does not support one interface that gcc -openmp uses for a tasking feature that is not in OpenMP 3.1. The interface gcc uses has significant performance implications for code that never uses tasking, and that seemed unreasonable. The same limitation applies to the Intel OpenMP runtime, and we have had no complaints about it yet :-). This incompatibility only arose with gcc 4.8 (IIRC); code compiled by earlier gcc's cannot hit the problem [because they don't support the language feature in question!].

-- Jim

James Cownie <james.h.cownie@intel.com>
SSG/DPD/TCAR (Technical Computing, Analyzers and Runtimes)
Tel: +44 117 9071438