Future of the LLVM OpenMP runtime

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup as an LLVM subproject is sustainable going forward without some form of testing support, automated or otherwise.

The motivation: It's difficult to make changes to the source without some way to see if a commit breaks features or indeed works at all. This has a chilling effect on contributions -- without means to test changes the only verification strategy is visual inspection and finger crossing which isn't really a viable way to develop platform libraries.

My understanding is that the proprietary OpenMP runtime test suite at Intel contains customer code so there's little prospect of getting released for use within LLVM. So we can rule that out.

As for the prospect of growing a test suite organically, there's a very low level of activity in LLVM OpenMP SVN characterised by occasional Intel code drops by Jim and my own work to get ToT tree building and running, with the previous commit being in December. So it's clear that a test suite, even a small one, isn't going to just "show up" through casual effort.

As such I'd like to propose that we set aside a few days, perhaps with developers from Nuanti and Intel and anyone else who wants to lend a hand, to adapt a branch of the libgomp test suite to be hosted externally, that can be used to regression test the LLVM OpenMP runtime. This shouldn't be much work given that the libraries are ABI/API-compatible and will at least show a way forward.

From a licensing point of view it'll be similar to the way we point clang users to external gcc documentation, or the way dragonegg plugs into external gcc builds -- the runtime sources themselves are unaffected other than a workflow change. I don't see alternatives but I'll open the floor to other suggestions keeping in mind that without tests the project is starting to look dead in the water.

(This mail touches on the OpenMP runtime that affects LLVM and clang so I'm copying in those lists.)

Alp.

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup as an LLVM subproject is sustainable going forward without some form of testing support, automated or otherwise.

...

Alp.

I work at a small start up that writes a lot of computationally-intensive code. The guys who work on that use OpenMP, but they're primarily targeting our Linux-based servers, and typically build with GCC. However, a big chunk of that code also goes into the iOS app that I'm responsible for, and thus OpenMP support is very much something we desire.

Using macros that either use OpenMP or GCD, we addressed our biggest hot spots, but only for a very particular class of OpenMP constructs. I'd love to see it fully integrated into Xcode.

Not sure how much I can help, but if you want me to try to build our stuff with it as it moves forward, let me know. Mostly I just wanted to voice my support for the effort.

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup as an LLVM subproject is sustainable going forward without some form of testing support, automated or otherwise.

...

Alp.

I work at a small start up that writes a lot of computationally-intensive code. The guys who work on that use OpenMP, but they're primarily targeting our Linux-based servers, and typically build with GCC. However, a big chunk of that code also goes into the iOS app that I'm responsible for, and thus OpenMP support is very much something we desire.

Using macros that either use OpenMP or GCD, we addressed our biggest hot spots, but only for a very particular class of OpenMP constructs. I'd love to see it fully integrated into Xcode.

Hi Rick,

Keep in mind that LLVM integration doesn't imply or even indicate Xcode support, but you're right, pulling things together is a step in the right direction. Achieving a complete stack is likely to encourage the ongoing effort towards OpenMP language support in the clang frontend as well.

Not sure how much I can help, but if you want me to try to build our stuff with it as it moves forward, let me know. Mostly I just wanted to voice my support for the effort.

As for trying it out, I suspect the infrastructure isn't quite there yet to respond to bug reports if you hit issues (no bugs reports or fixes have been made yet since the runtime project launched) but James Cownie might be able to answer better.

It's easy to build and there haven't been success/failure reports in public yet so sure, if you have a minute to try the runtime with gcc or icc that could be useful!

Thanks
Alp.

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup as an LLVM subproject is sustainable going forward without some form of testing support, automated or otherwise.

...

Alp.

I work at a small start up that writes a lot of computationally-intensive code. The guys who work on that use OpenMP, but they're primarily targeting our Linux-based servers, and typically build with GCC. However, a big chunk of that code also goes into the iOS app that I'm responsible for, and thus OpenMP support is very much something we desire.

Using macros that either use OpenMP or GCD, we addressed our biggest hot spots, but only for a very particular class of OpenMP constructs. I'd love to see it fully integrated into Xcode.

Hi Rick,

Keep in mind that LLVM integration doesn't imply or even indicate Xcode support, but you're right, pulling things together is a step in the right direction. Achieving a complete stack is likely to encourage the ongoing effort towards OpenMP language support in the clang frontend as well.

Absolutely. I added that note to make it clear that the exercise is academic until I can use it in Xcode, although it is possible to point Xcode at another version of clang, which may or may not be sustainable on this team.

Not sure how much I can help, but if you want me to try to build our stuff with it as it moves forward, let me know. Mostly I just wanted to voice my support for the effort.

As for trying it out, I suspect the infrastructure isn't quite there yet to respond to bug reports if you hit issues (no bugs reports or fixes have been made yet since the runtime project launched) but James Cownie might be able to answer better.

It's easy to build and there haven't been success/failure reports in public yet so sure, if you have a minute to try the runtime with gcc or icc that could be useful!

I think you're saying to build the LLVM OpenMP runtime, but build our code with gcc and link against the LLVM OpenMP runtime instead of the default runtime from GCC/Intel? If you can point me at documentation on how to do this, I'll see what I can do.

I'd add that fairly high up the TODO list should probably be a build system that's a bit easier to modify. We're very interested in getting the runtime working on FreeBSD with a view to importing it into the base system at some point in the future (we've removed the GNU OpenMP runtime and would quite like a replacement).

Unfortunately, the current build system is a complex structure of Makefile driving Perl scripts and vice versa. After half an hour of trying to persuade it that FreeBSD was, in fact, a valid target, I gave up. The description of what things are valid targets is spread over multiple files with subtle interactions between them.

We have a number of people producing experimental manycore 64-bit MIPS systems running FreeBSD, so we'd also be interested in doing MIPS bring-up, but the build system is currently something of a show-stopper for us.

David

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal
to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup
as an LLVM subproject is sustainable going forward without some form of testing support, automated
or otherwise.

The motivation: It's difficult to make changes to the source without some way to see if a commit breaks
features or indeed works at all. This has a chilling effect on contributions -- without means to test
changes the only verification strategy is visual inspection and finger crossing which isn't really
a viable way to develop platform libraries.

I agree with all of that!

My understanding is that the proprietary OpenMP runtime test suite at Intel contains customer code so
there's little prospect of getting released for use within LLVM. So we can rule that out.

Correct. It *may* contain code snippets from customers for regression testing, and we have lawyers, so
even "may" is enough to seriously worry them. Attempting to audit it would be such a big job that
starting from somewhere else is likely to be more productive.

As such I'd like to propose that we set aside a few days, perhaps with developers from Nuanti
and Intel and anyone else who wants to lend a hand, to adapt a branch of the libgomp test suite to be
hosted externally, that can be used to regression test the LLVM OpenMP runtime.
This shouldn't be much work given that the libraries are ABI/API-compatible and will at least
show a way forward.

Another potential solution would be to take the validation suite from the OpenUH compiler
http://web.cs.uh.edu/~openuh/download/ (which is BSD licensed), and start to build a more
comprehensive set of tests from that. Since it is BSD I believe we could take it without asking,
however I'd like to get an ACK from the authors before doing that.

The other issues to consider here are
1) Which compiler is used to compile the tests?.
   Without the OpenMP support in clang we still don't have our own compiler that can do that, so
   we'd presumably have to use gcc.
2) If we build the runtime with clang and run tests built with gcc we need to be aware that there will be
   a potential incompatibility, since gcc supports a 128 bit float, and clang/llvm don't. Therefore
   it is impossible to compile the support for 128 bit reductions (which exists in the runtime source)
   using clang. (Though gcc may not be calling the runtime to do reductions anyway, in which case the issue
   is a more general one of test coverage).

-- Jim

James Cownie <james.h.cownie@intel.com>
SSG/DPD/TCAR (Technical Computing, Analyzers and Runtimes)
Tel: +44 117 9071438

James, I believe the UofHouston folks would be quite pleased that this project would utilize their test suite. Having used it for previous projects, I can say that its quite thorough once you get your arms wrapped around the infrastructure. We'll likely need to do a bit of work in order to get tests running for OpenMP 4.0 additions.

If anyone is interested, Sunita Chandrasekaran presented a paper at IWOMP'12 describing the infrastructure.

http://dl.acm.org/citation.cfm?id=2345831

cheers
john

Apologies to hi-jack the thread a bit

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup as an LLVM subproject is sustainable going forward without some form of testing support, automated or otherwise.

I'd add that fairly high up the TODO list should probably be a build system that's a bit easier to modify. We're very interested in getting the runtime working on FreeBSD with a view to importing it into the base system at some point in the future (we've removed the GNU OpenMP runtime and would quite like a replacement).

Some good news: We have a FreeBSD port covering both the build system and the OpenMP runtime itself. I'll see if we can contribute this upstream in the next few days.

This work generalises a few abstractions to ease porting to other platforms as well.

Unfortunately, the current build system is a complex structure of Makefile driving Perl scripts and vice versa. After half an hour of trying to persuade it that FreeBSD was, in fact, a valid target, I gave up. The description of what things are valid targets is spread over multiple files with subtle interactions between them.

Agree a CMake build system is desirable. Some of the perl configure and generation code is ingrained and might have to stay around longer.

We have a number of people producing experimental manycore 64-bit MIPS systems running FreeBSD, so we'd also be interested in doing MIPS bring-up, but the build system is currently something of a show-stopper for us.

The port is working on FreeBSD 10 x86_64 (modulo CPU affinity support) so MIPS shouldn't be a large leap from there, hopefully this time with some test coverage.
Alp.

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup as an LLVM subproject is sustainable going forward without some form of testing support, automated or otherwise.

I'd add that fairly high up the TODO list should probably be a build system that's a bit easier to modify. We're very interested in getting the runtime working on FreeBSD with a view to importing it into the base system at some point in the future (we've removed the GNU OpenMP runtime and would quite like a replacement).

Some good news: We have a FreeBSD port covering both the build system and the OpenMP runtime itself. I'll see if we can contribute this upstream in the next few days.

This work generalises a few abstractions to ease porting to other platforms as well.

That's awesome!

We have a number of people producing experimental manycore 64-bit MIPS systems running FreeBSD, so we'd also be interested in doing MIPS bring-up, but the build system is currently something of a show-stopper for us.

The port is working on FreeBSD 10 x86_64 (modulo CPU affinity support) so MIPS shouldn't be a large leap from there, hopefully this time with some test coverage.

Excellent news! We'll look at initially importing it and the Intel Clang fork into the ports tree and see if we can start using it for OpenMP-requiring ports.

What's needed for CPU affinity? We expose his via pthread_attr_setaffinity_np() in <pthread_np.h> - does the runtime need anything more from the interface, or was this support just not yet a high priority for you? I'd be happy to help with this support.

Since Linux and FreeBSD use the same calling conventions and data layouts on MIPS (and ARM), hopefully we can both benefit from improved architecture support in the library.

David

What's needed for CPU affinity? We expose his via pthread_attr_setaffinity_np() in <pthread_np.h> -
does the runtime need anything more from the interface, or was this support just not yet a high
priority for you? I'd be happy to help with this support.

The Linux code uses the sched_{get,set}affinity calls directly, rather than via the pthread veneer.
It can also parse /proc/cpuinfo, though on X86 machines we likely avoid that in most cases by using
info from the "cpuid" instruction to determine the hardware layout of the machine. I have no idea
whether there are equivalent interfaces on MIPS (Imagination) or ARM platforms that would allow you
to discover the hardware configuration of the machine in a portable manner (I.e. one that works no
matter what implementation of the architecture you happen to be running on).

-- Jim

James Cownie <james.h.cownie@intel.com>
SSG/DPD/TCAR (Technical Computing, Analyzers and Runtimes)
Tel: +44 117 9071438

Tangentially related, I have a question:

I don't know, well, anything about how OpenMP is implemented. But does it make sense to use Grand Central Dispatch on platforms that support it (instead of, I assume, pthreads)?

It would certainly be interesting for an OpenMP runtime and libdispatch to use the same kernel APIs for determining system load and the number of threads that it makes sense to create. For HPC applications, the number of threads that OpenMP should use is basically the number of cores (although not always, depending on the workload, many other disclaimers apply), but for a desktop or mobile there may be power management concerns or other higher-priority tasks that mean that you'll get better throughput using fewer threads.

David

We played with implementing OpenMP 3.x tasks on top of libdispatch and while we got it to work - there was some scalability of the scheduling across a lot of cores which would probably need to be addressed. For systems with 4-8 cores I doubt it would ever cause a significant problem though.

That's good news. Currently I need OpenMP support in Xcode because I'm incorporating code my colleagues wrote that uses it into the iOS app I write. We have yet to exceed four cores on an iOS device, so that should be okay. Perhaps the runtime can conditionally build to use GCD when targeting iOS or OS X (assuming there's a compelling reason, performance or energy, to do so).

Maybe you didn't understand 100% - Is the code your colleagues wrote using OpenMP tasks or some other part of the OMP standard? (I suspect it's not using tasks) In which case lowering down to pthreads is probably all that's available.

The good news - libdispatch is portable to almost all platforms (except Windows I think). We got the best scalability off OpenSolaris (RIP)

Tangentially related, I have a question:

I don't know, well, anything about how OpenMP is implemented. But does it make sense to use Grand Central Dispatch on platforms that support it (instead of, I assume, pthreads)?

We played with implementing OpenMP 3.x tasks on top of libdispatch and while we got it to work - there was some scalability of the scheduling across a lot of cores which would probably need to be addressed. For systems with 4-8 cores I doubt it would ever cause a significant problem though.

That's good news. Currently I need OpenMP support in Xcode because I'm incorporating code my colleagues wrote that uses it into the iOS app I write. We have yet to exceed four cores on an iOS device, so that should be okay. Perhaps the runtime can conditionally build to use GCD when targeting iOS or OS X (assuming there's a compelling reason, performance or energy, to do so).

Maybe you didn't understand 100% - Is the code your colleagues wrote using OpenMP tasks or some other part of the OMP standard? (I suspect it's not using tasks) In which case lowering down to pthreads is probably all that's available.

I'm 100% sure I don't understand 100%. :slight_smile:

I asked…I was told the one thing we use a lot is OpenMP parallel for, which we've worked around with a set of macros that uses OpenMP when building with GCC (for our Linux targets) and uses GCD when building for iOS. This mostly works, but every now and again they break the iOS build because a method return something other than void and the macros don't return anything

OpenMP thread-local storage was something they looked into using for some other operation, but haven't actually used yet.

The good news - libdispatch is portable to almost all platforms (except Windows I think). We got the best scalability off OpenSolaris (RIP)

Well, I wouldn't expect the world who wants to use OpenMP in LLVM also be willing to use libdispatch, so I imagine this would be a selectable implementation thing, and even then, only if there's a compelling reason to use GCD.

Now that we've kick-started the LLVM OpenMP runtime discussion, I want to make a concrete proposal to get a test suite up and running for the LLVM OpenMP runtime. I don't think the current setup as an LLVM subproject is sustainable going forward without some form of testing support, automated or otherwise.

I'd add that fairly high up the TODO list should probably be a build system that's a bit easier to modify. We're very interested in getting the runtime working on FreeBSD with a view to importing it into the base system at some point in the future (we've removed the GNU OpenMP runtime and would quite like a replacement).

Some good news: We have a FreeBSD port covering both the build system and the OpenMP runtime itself. I'll see if we can contribute this upstream in the next few days.

This work generalises a few abstractions to ease porting to other platforms as well.

That's awesome!

The FreeBSD port of the OpenMP runtime has landed: r202478

We have a number of people producing experimental manycore 64-bit MIPS systems running FreeBSD, so we'd also be interested in doing MIPS bring-up, but the build system is currently something of a show-stopper for us.

The port is working on FreeBSD 10 x86_64 (modulo CPU affinity support) so MIPS shouldn't be a large leap from there, hopefully this time with some test coverage.

Excellent news! We'll look at initially importing it and the Intel Clang fork into the ports tree and see if we can start using it for OpenMP-requiring ports.

What's needed for CPU affinity? We expose his via pthread_attr_setaffinity_np() in <pthread_np.h> - does the runtime need anything more from the interface, or was this support just not yet a high priority for you? I'd be happy to help with this support.

I've checked over and there are a couple of minor tasks remaining to get a full pass of the validation suite (currently 59/63) so let's prioritize those above affinity support.

Since Linux and FreeBSD use the same calling conventions and data layouts on MIPS (and ARM), hopefully we can both benefit from improved architecture support in the library.

Sounds good, I'll follow up on openmp-dev with details and a list of open tasks on the FreeBSD port. Let's take it from there?

Alp.

As an aside, there may be more interest than expected in porting the runtime to utilize threading/tasking backends other than pthreads. The work by Stephen Olivier, Kyle Wheeler and Dylan Stark to port portions of Qthreads under GOMP [for use with Rose] generated some interesting results. There are a number of other efforts out there working to generate/port fine-grained runtime mechanisms under standard programming models such as OpenMP. We may be in a good position now to begin permitting such things in the future.

cheers
john

John D. Leidel