[RFC] Improving the testing of exported LLVM CMake targets

Hi,

Following on from another thread (Long-Term Support for LLVM Projects
Extension to Build System) I'd like to discuss the testing of the
exported LLVM CMake targets which can be used by consumers of LLVM as
documented in [1].

Right now we don't test this feature **at all** and as a result it has
been or is

* broken in trunk
* broken by those packaging LLVM
* broken in the official release binaries

I'd like to propose a few ideas I have to improve this situation.

I've created a small toy project [2] that uses the exported LLVM CMake
targets. What I'd like to happen is for this project to become part of
the LLVM source tree (or as a sub project) so that it can be tested in
various scenarios. I do not want this toy project to be a maintenance
burden so it makes very minimal use of LLVM's C++ API.

## Testing trunk

We could teach some of the existing build bots to try building this
toy project after building LLVM. The scenarios we should test are
building the toy project

- Against the LLVM build tree when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with the
Autoconf/Makefile build system

## Testing packaged LLVM

For those producing LLVM packages for Linux/*BSD, OSX and Windows it
would be good if we could encourage them to test that the toy project
actually builds against their package. Building the toy projects only
takes a few seconds so this shouldn't be much of a burden but can
greatly increase a packagers confidence that LLVM has been properly
packaged.

## Testing official LLVM release binaries

This situation is currently broken. I've observed that

- The LLVM 3.6.* binary tarballs have the LLVM CMake files completely
missing, how did this happen?
- The LLVM3.5 binary tarballs have the CMake files but they're
completely broken.

When making official releases I think it would be a good idea to test
that building the toy project against an extracted binary tarball
actually works.

There are two problems currently AFAICT:

First, the current test-release.sh script does not package an install
properly it uses --prefix to set a temporary directory for the tarball
install when instead it should be using the DESTDIR makefile variable.
I have sent a patch to llvm-commits to try to fix this [3]. The result
of this mistake is that the CMake files end up containing absolute
paths to the temporary tarball directory which means they cannot be
imported by another project.

Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake)
are not relocatable (they contain absolute paths which are based on
the install prefix). So even if the first issue is fixed we still have
the problem that a user cannot simply extract the binary tarball to an
arbitrary location their system and build the toy project against it.

I think the LLVMExport.cmake file and LLVM-Config.cmake files need to
be loaded from the same directory containing the found
LLVMConfig.cmake file. The automatically generated LLVMExports.cmake
file would also need to contain relative paths to the libraries (it
currently has absolute paths). I'm not sure how to do this for both of
LLVM's build systems. Any ideas?

Alternatively we could explicitly state in our docs that the LLVM
release binaries can only be used from CMake when it is installed into
the right prefix. This simpler but not very useful because it means
those who want to use the official LLVM binary release builds on their
machine can't unless they have root access (so they can write to
/usr/local/).

Thoughts?

[1] http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project
[2] GitHub - delcypher/llvmCMakeImportDemo: A demo of importing LLVM into a CMake project via LLVM's exported targets
[3] http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150622/284436.html

Thanks,
Dan.

Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake) are not relocatable (they contain absolute paths which are based on the install prefix). So even if the first issue is fixed we still have the problem that a user cannot simply extract the binary tarball to an arbitrary location their system and build the toy project against it.

Really? My latest LLVMConfig (beginning of this month) is relative to "LLVM_INSTALL_PREFIX" which is set by

get_filename_component(LLVM_INSTALL_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)
get_filename_component(LLVM_INSTALL_PREFIX "${LLVM_INSTALL_PREFIX}" PATH)
get_filename_component(LLVM_INSTALL_PREFIX "${LLVM_INSTALL_PREFIX}" PATH)
get_filename_component(LLVM_INSTALL_PREFIX "${LLVM_INSTALL_PREFIX}" PATH)

and analogously in LLVMExports:

get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)
get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)

Did someone change this recently?

The files generated by the CMake build are completely relocatable
because they compute everything relative to the LLVMConfig.cmake
file location. The Autoconf/Makefile build system would have to
be taught to do the same thing in the files it generates. Take a
look at the CMake-generated files to see how they achieve it.

Thanks,
-Brad

Hi,

Yes, sorry I got my install directories confused (I should give them
better names!). I'll try to write a patch for the autoconf/Makefile
build system. The Debian/Ubuntu packages modify the install location
of the LLVM CMake files so (assuming my patch gets accepted) the
maintainers will need to do some more patching in the next release in
order to avoid breaking the installed CMake files. The toy project
that I mentioned in the original e-mail would be useful here because
it would let the packagers test their package with minimal effort.

The other points that I mention in the original e-mail (broken binary
builds, the need for more testing, etc.) still stand though.

Thanks,
Dan.

I'll try to write a patch for the autoconf/Makefile build system.

Great. Please Cc me on proposed patches.

order to avoid breaking the installed CMake files. The toy project
that I mentioned in the original e-mail would be useful here because
it would let the packagers test their package with minimal effort.

The other points that I mention in the original e-mail (broken binary
builds, the need for more testing, etc.) still stand though.

Yes, certainly the LLVM packaging and distribution process could use
infrastructure to test building applications against the (relocated)
packages.

One thing that may help a little bit is Debian may soon be packaging
my application that uses CMake to build against LLVM/Clang:

https://github.com/CastXML/CastXML
[CastXML] Release tarballs?

Packaging of the tool may fail if Debian's LLVM package does not
provide the CMake files correctly, so that will provide some coverage.

Thanks,
-Brad

I've just proposed a patch (
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150629/284641.html
)

Dan Liew wrote:

This situation is currently broken. I've observed that

- The LLVM 3.6.* binary tarballs have the LLVM CMake files completely
missing, how did this happen?
- The LLVM3.5 binary tarballs have the CMake files but they're
completely broken.

You could also consider changing the names of the llvm imported targets at
this point to llvm::foo (and similar for the clang imported targets). This
will give better diagnostics with current cmake releases, and may gain extra
benefit in the future.

See

CMP0028 — CMake 3.2.3 Documentation

As of the resolution of

https://llvm.org/bugs/show_bug.cgi?id=19462

clang installs appropriate files when built with CMake, but I don't know
about other llvm projects.

I introduced a policy to disallow doing what clang was doing there, so you
can use CMake 3.3-rc2 and CMP0062 to find any other possible instances of
that problem.

Thanks,

Steve.

Hi,

Dan Liew wrote:

This situation is currently broken. I've observed that

- The LLVM 3.6.* binary tarballs have the LLVM CMake files completely
missing, how did this happen?
- The LLVM3.5 binary tarballs have the CMake files but they're
completely broken.

You could also consider changing the names of the llvm imported targets at
this point to llvm::foo (and similar for the clang imported targets). This
will give better diagnostics with current cmake releases, and may gain extra
benefit in the future.

I'd like to do that but I worry about downstream projects that rely
directly on target names instead of using the
llvm_map_components_to_libnames() function. Would using aliases like
``add_library(LLVMSuport ALIAS llvm::Support)`` be sufficient to not
break downstream projects?

Thanks,
Dan.

Dan Liew wrote:

I'd like to do that but I worry about downstream projects that rely
directly on target names instead of using the
llvm_map_components_to_libnames() function. Would using aliases like
``add_library(LLVMSuport ALIAS llvm::Support)`` be sufficient to not
break downstream projects?

An IMPORTED target may not have an ALIAS. The solution might rather be to
generate and import two sets of targets. You would need to decide what your
deprecation strategy is for the old targets, and how much source
compatibility is needed. Making this change in the future is not going to be
easier than it is now.

Thanks,

Steve.

Hi,

I'd like to try and revive this dead thread. Another breakage of the
CMake exported targets has occurred so it's about time something was
done about this.

## Testing trunk

We could teach some of the existing build bots to try building this
toy project after building LLVM. The scenarios we should test are
building the toy project

- Against the LLVM build tree when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with the
Autoconf/Makefile build system

Unless there are any objections I'd like to go ahead with this. I need
to know where the toy project should live SVN tree. Could someone tell
me where this should be and I'll add it? I should also probably give
the files the usual LLVM copyright notice.

## Testing packaged LLVM

For those producing LLVM packages for Linux/*BSD, OSX and Windows it
would be good if we could encourage them to test that the toy project
actually builds against their package. Building the toy projects only
takes a few seconds so this shouldn't be much of a burden but can
greatly increase a packagers confidence that LLVM has been properly
packaged.

Our documentation needs to be updated to ask packagers to check that
the toy project builds.

## Testing official LLVM release binaries

The test-release.sh script probably needs to be taught to try to build
the toy project as part of its testing (probably make it optional when
building with autoconf because the user might not have CMake).

Thanks,
Dan.