Integration with LLVM CMake build when doing Universal Binary

Hello openmp-dev,

When trying a build with everything under the LLVM hierarchy (in this case openmp subversion trunk saved under projects/libiomp) I encountered this error :

[ 42%] Building C object projects/libiomp/runtime/CMakeFiles/iomp5.dir/src/z_Linux_asm.s.o
/tmp/llvm/projects/libiomp/runtime/src/z_Linux_asm.s:702:8: error: register %rbp is only available in 64-bit mode
pushq %rbp
           ^~~~

I usually do a "Universal build" (32-bit / 64-bit in the same bin/lib) something like :

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_OSX_ARCHITECTURES="x86_64;i386"

so you end up with the compilation done this way :

[ 42%] Building C object projects/libiomp/runtime/CMakeFiles/iomp5.dir/src/z_Linux_asm.s.o
cd /tmp/llvm/build/projects/libiomp/runtime && /usr/bin/cc -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Diomp5_EXPORTS -DKMP_ARCH_X86_64 -fPIC -Wall -W -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -DNDEBUG -arch x86_64 -arch i386 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk -fPIC -I/tmp/llvm/build/projects/libiomp/runtime -I/tmp/llvm/projects/libiomp/runtime -I/tmp/llvm/build/include -I/tmp/llvm/include -I/tmp/llvm/projects/libiomp/runtime/src -I/tmp/llvm/projects/libiomp/runtime/src/i18n -I/tmp/llvm/projects/libiomp/runtime/src/include/41 -I/tmp/llvm/projects/libiomp/runtime/src/thirdparty/ittnotify -D USE_ITT_BUILD -D KMP_ARCH_STR="\"Intel(R) 64\"" -D BUILD_I8 -D KMP_LIBRARY_FILE=\"libiomp5.dylib\" -D KMP_VERSION_MAJOR=5 -D KMP_NESTED_HOT_TEAMS -D CACHE_LINE=64 -D KMP_ADJUST_BLOCKTIME=1 -D BUILD_PARALLEL_ORDERED -D KMP_ASM_INTRINS -D USE_ITT_NOTIFY=1 -D INTEL_ITTNOTIFY_PREFIX=__kmp_itt_ -D _GNU_SOURCE -D _REENTRANT -D BUILD_TV -D USE_CBLKDATA -D KMP_GOMP_COMPAT -D USE_LOAD_BALANCE -D KMP_USE_ASSERT -D GUIDEDLL_EXPORTS -D KMP_STATS_ENABLED=0 -D OMPT_SUPPORT=0 -D OMPT_BLAME=1 -D OMPT_TRACE=1 -D OMP_50_ENABLED=0 -D OMP_41_ENABLED=1 -D OMP_40_ENABLED=1 -D OMP_30_ENABLED=1 -D KMP_USE_ADAPTIVE_LOCKS=1 -D KMP_DEBUG_ADAPTIVE_LOCKS=0 -D KMP_USE_INTERNODE_ALIGNMENT=0 -x assembler-with-cpp -o CMakeFiles/iomp5.dir/src/z_Linux_asm.s.o -c /tmp/llvm/projects/libiomp/runtime/src/z_Linux_asm.s

The flag -arch i386 conflicts with the -DKMP_ARCH_X86_64 and produce the result above (or at least I think so).

Would it be possible to support building with -DCMAKE_OSX_ARCHITECTURES="x86_64;i386" ?

Regards,

jonathan

Somebody else has asked for a fat build and I tried to wrangle this problem for a while. I may have a solution where the Linux assembly file (which is for mac as well) simply decodes which architecture it is using like the rest of the C and CXX files are doing implicitly. I'll look at this more closely tomorrow and see.

-- Johnny

Jonathan and Jack,

Can you guys test this patch and see if it allows the universal fat build?

Call cmake like Jonathan noted below with the -DCMAKE_OSX_ARCHITECTURES='...' flag set
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_OSX_ARCHITECTURES="x86_64;i386" ..

To apply:
patch -p1 < allow_fat_builds.patch

-- Johnny

allow_fat_builds.patch (8.87 KB)

Hello Johnny,

Thanks a lot. It works for me.

Johnathan,
     It works but has a rather nasty side-effect. Although passing
-DCMAKE_OSX_ARCHITECTURES="x86_64;i386" causes the libiomp5.dylib to
build as a universal binary, it also bloats the overall compiler build
by a factor of two because it also causes all of the executables in
the compiler built to become universal binaries as well...

bugpoint: Mach-O universal binary with 2 architectures
bugpoint (for architecture x86_64): Mach-O 64-bit executable x86_64
bugpoint (for architecture i386): Mach-O executable i386
clang: Mach-O universal binary with 2 architectures
clang (for architecture x86_64): Mach-O 64-bit executable x86_64
clang (for architecture i386): Mach-O executable i386

So it pretty much uses a sledgehammer to solve the libiomp5.dylib
problem. We really need to find a way to do thing without forcing the
use of -DCMAKE_OSX_ARCHITECTURES="x86_64;i386". You shouldn't have to
build a fat binary of the compiler itself to build 32-bit code.
          Jack

Jonathan,
     Apple has been building their system clang compilers as
x86_64-only executables since Xcode 5 (while still supporting
compilation of i386 code). The proposed patch regresses that behavior
by forcing fat builds of the compiler executables.
       Jack

These are all great points.

How about if I add -DLIBOMP_OSX_ARCHITECTURES (which defaults to CMAKE_OSX_ARCHITECTURES) which will set CMAKE_OSX_ARCHITECTURES when building libomp, but doesn't affect any other projects.

-- Johnny

In this patch, I have added LIBOMP_OSX_ARCHITECTURES which just defaults to CMAKE_OSX_ARCHITECTURES. So a user could either:

1) use nothing which will produce the default architecture of the compiler they are using.
2) use CMAKE_OSX_ARCHITECTURES which will force everything to be built as a universal binary.
3) use LIBOMP_OSX_ARCHITECTURES which will also produce the behavior of CMAKE_OSX_ARCHITECTURES, but any project "above" libomp would not be affected.

-- Johnny

allow_fat_builds_v2.patch (10.1 KB)

Jonathan,
      I can confirm on x86_64-apple-darwin14 that the
allow_fat_builds_v2.patch works as expected for all three
permutations. Thanks for fixing this issue.
             Jack
ps Do you plan on posting a patch to implement the library name change
from libiomp5 to libomp so the clang default for -fopenmp can be
restored back to the llvm openmp library? The only complication for
this change on darwin will be the requirement that the shared library
retains the compatibility version of 5.0.0 or greater under the new
libomp name. Otherwise dyld won't load the renamed libomp.dylib via a
libimp5.dylib symlink for legacy binaries.

ps Do you plan on posting a patch to implement the library name change from libiomp5 to libomp so the clang default for -fopenmp can be restored back to the llvm openmp library? The only >complication for this change on darwin will be the requirement that the shared library retains the compatibility version of 5.0.0 or greater under the new libomp name. Otherwise dyld won't load >the renamed libomp.dylib via a libimp5.dylib symlink for legacy binaries.

I don't see any problem with keeping the version at 5.0.0 for the new library name change.

I guess I have a question about this matter though. Would the plan be to create libomp.[so|dylib] and for "make install" to also create a libiomp5.[so|dylib] symlink as well? Or, are we going to just create libomp.[so|dylib] during the install phase?

Everyone can chime in here of course. I'm not positive what the plan was with this.

-- Johnny

ps Do you plan on posting a patch to implement the library name change from libiomp5 to libomp so the clang default for -fopenmp can be restored back to the llvm openmp library? The only >complication for this change on darwin will be the requirement that the shared library retains the compatibility version of 5.0.0 or greater under the new libomp name. Otherwise dyld won't load >the renamed libomp.dylib via a libimp5.dylib symlink for legacy binaries.

I don't see any problem with keeping the version at 5.0.0 for the new library name change.

I guess I have a question about this matter though. Would the plan be to create libomp.[so|dylib] and for "make install" to also create a libiomp5.[so|dylib] symlink as well? Or, are we going to just create libomp.[so|dylib] during the install phase?

My understanding is that the libomp library name was agreed upon…

http://lists.cs.uiuc.edu/pipermail/cfe-dev/2015-May/043035.html
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2015-May/043039.html
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2015-May/043042.html
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2015-May/043044.html
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2015-May/043040.html

so the legacy support should be done with a symlink as the old
libiomp5 shared library name pointing to the actual shared library
under the new name.

Everyone can chime in here of course. I'm not positive what the plan was with this.

I suspect that the purists will want the cfe and openmp sources
scrubbed of all references to libiomp5 or iomp5 outside of the legacy
symlink creation.
Mocking up a patch to implement this and posting it for comments would
be the best first step.
        Jack

so the legacy support should be done with a symlink as the old
libiomp5 shared library name pointing to the actual shared library under the new name.

This sounds good to me.

I have one patch that is pretty much finished up which moves the library building component of CMake into a new src/CMakeLists.txt file.
This structure is far more typical/logical for CMake than having one large CMakeLists.txt file.
After I commit the universal fat binary patch and get an ok on this recursive CMake patch, I will post a new patch which cleans up as much of the libiomp stuff as possible and uses the new name, libomp, and installs a symlink to libiomp.

-- Johnny

Jonathan,
      For now, please do check in the allow_fat_builds_v2.patch so we
get the fat openmp library issue on darwin out of the way.
             Thanks in advance.
                         Jack