Building SVN head with CMake - shared libraries?

That got accepted before I could review it.

I have left some comments with suggestions on how to improve what has
been committed to trunk.

http://reviews.llvm.org/D16208

Neither the LLVM nor clang libraries keep compatibility with previous
versions.

So having the version in the name (rather than using an soversion) is
typically more useful, as then llvm/clang 3.7 and 3.8 can be coinstallable
and you can decide at build time which one you want to build against
(instead of having only a single libclang.so symlink that points to one or
the other).

Hi again,

Hi,

I’m kinda scared that you’re using it. What are you trying to accomplish
that you are using it? Generally having LLVM split among that many
dynamically loaded libraries results in significant performance regressions.

When we first switched to cmake it was the only option to produce
shared libraries, hence we went with it. I am testing with
-DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON atm, and will let
you know if any problem arises.

I am trying to enable this on openSUSE but it seems to break
standalone lldb (note that we don't ship static libs):

cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
-DCMAKE_CXX_FLAGS=-stdlib=libc++ -DLLVM_BUILD_LLVM_DYLI
B=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_LIBDIR_SUFFIX=64 -DLLDB_PA
TH_TO_LLVM_BUILD=/usr -DLLDB_PATH_TO_CLANG_BUILD=/usr
-DLLVM_RUNTIME_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/buil
d/bin -DLLVM_LIBRARY_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/build/lib64
-DPYTHON_VERSION_MAJOR=2 -DPYTHON_VERSIO
N_MINOR=7 -G Ninja ..

[...]

CMake Error at /usr/share/llvm/cmake/LLVMExports.cmake:1011 (message):
  The imported target "LLVMSupport" references the file

      "/usr/lib64/libLLVMSupport.a"

  but this file does not exist. Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/usr/share/llvm/cmake/LLVMExports.cmake"

   but not all the files it references.

Call Stack (most recent call first):
  /usr/share/llvm/cmake/LLVMConfig.cmake:178 (include)
  cmake/modules/LLDBStandalone.cmake:76 (include)
  CMakeLists.txt:3 (include)

-- Configuring incomplete, errors occurred!

I am trying to enable this on openSUSE but it seems to break
standalone lldb (note that we don't ship static libs):

cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
-DCMAKE_CXX_FLAGS=-stdlib=libc++ -DLLVM_BUILD_LLVM_DYLI
B=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_LIBDIR_SUFFIX=64 -DLLDB_PA
TH_TO_LLVM_BUILD=/usr -DLLDB_PATH_TO_CLANG_BUILD=/usr
-DLLVM_RUNTIME_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/buil
d/bin -DLLVM_LIBRARY_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/build/lib64
-DPYTHON_VERSION_MAJOR=2 -DPYTHON_VERSIO
N_MINOR=7 -G Ninja ..

[...]

CMake Error at /usr/share/llvm/cmake/LLVMExports.cmake:1011 (message):
  The imported target "LLVMSupport" references the file

      "/usr/lib64/libLLVMSupport.a"

Could you try appending ``-DLLVM_LINK_LLVM_DYLIB=ON`` to your CMake
invocation? It looks like lldb is using the ``llvm_add_library()``
function in the ``cmake/modules/AddLLVM.cmake`` (LLVM source tree)
which uses ``LLVM_LINK_LLVM_DYLIB`` (and a few other variables) to
decide whether to link against LLVM's static component libraries or
the monolithic shared library. I could be wrong on this though but it
should
be pretty quick to check.

Out of curiosity though if you're building packages on openSUSE why do
you need a standalone build? I thought the normal way to do this was
to build everything and then generate split packages?

Thanks,
Dan.

I am trying to enable this on openSUSE but it seems to break
standalone lldb (note that we don't ship static libs):

cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
-DCMAKE_CXX_FLAGS=-stdlib=libc++ -DLLVM_BUILD_LLVM_DYLI
B=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_LIBDIR_SUFFIX=64 -DLLDB_PA
TH_TO_LLVM_BUILD=/usr -DLLDB_PATH_TO_CLANG_BUILD=/usr
-DLLVM_RUNTIME_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/buil
d/bin -DLLVM_LIBRARY_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/build/lib64
-DPYTHON_VERSION_MAJOR=2 -DPYTHON_VERSIO
N_MINOR=7 -G Ninja ..

[...]

CMake Error at /usr/share/llvm/cmake/LLVMExports.cmake:1011 (message):
  The imported target "LLVMSupport" references the file

      "/usr/lib64/libLLVMSupport.a"

Could you try appending ``-DLLVM_LINK_LLVM_DYLIB=ON`` to your CMake
invocation? It looks like lldb is using the ``llvm_add_library()``
function in the ``cmake/modules/AddLLVM.cmake`` (LLVM source tree)
which uses ``LLVM_LINK_LLVM_DYLIB`` (and a few other variables) to
decide whether to link against LLVM's static component libraries or
the monolithic shared library. I could be wrong on this though but it
should
be pretty quick to check.

I already use -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON

Out of curiosity though if you're building packages on openSUSE why do
you need a standalone build? I thought the normal way to do this was
to build everything and then generate split packages?

Nice question. Since Mesa depends on llvm and Mesa is very crucial to
a distro bootstrap, adding any new dependency to llvm creates a
circular dependency which we can't have. And lldb itself brings python
dependency which complicates the issue (python has a Mesa dependency).
I may try to pre-generate files with swig to remove the dependency but
it would be nice if this worked as expected.

Thanks!

ismail

Forgot this line, I thought swig was the problematic dep, its not.

I am trying to enable this on openSUSE but it seems to break
standalone lldb (note that we don't ship static libs):

cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
-DCMAKE_CXX_FLAGS=-stdlib=libc++ -DLLVM_BUILD_LLVM_DYLI
B=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_LIBDIR_SUFFIX=64 -DLLDB_PA
TH_TO_LLVM_BUILD=/usr -DLLDB_PATH_TO_CLANG_BUILD=/usr
-DLLVM_RUNTIME_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/buil
d/bin -DLLVM_LIBRARY_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/build/lib64
-DPYTHON_VERSION_MAJOR=2 -DPYTHON_VERSIO
N_MINOR=7 -G Ninja ..

[...]

CMake Error at /usr/share/llvm/cmake/LLVMExports.cmake:1011 (message):
  The imported target "LLVMSupport" references the file

      "/usr/lib64/libLLVMSupport.a"

Could you try appending ``-DLLVM_LINK_LLVM_DYLIB=ON`` to your CMake
invocation? It looks like lldb is using the ``llvm_add_library()``
function in the ``cmake/modules/AddLLVM.cmake`` (LLVM source tree)
which uses ``LLVM_LINK_LLVM_DYLIB`` (and a few other variables) to
decide whether to link against LLVM's static component libraries or
the monolithic shared library. I could be wrong on this though but it
should
be pretty quick to check.

I already use -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON

That'll teach me to suggest something without actually trying it. Then
I might have actually noticed you're already using that flag!

Let me see if I can reproduce this. Are you building on LLVM trunk on
the 3.8 release branch?

Out of curiosity though if you're building packages on openSUSE why do
you need a standalone build? I thought the normal way to do this was
to build everything and then generate split packages?

Nice question. Since Mesa depends on llvm and Mesa is very crucial to
a distro bootstrap, adding any new dependency to llvm creates a
circular dependency which we can't have. And lldb itself brings python
dependency which complicates the issue (python has a Mesa dependency).
I may try to pre-generate files with swig to remove the dependency but
it would be nice if this worked as expected.

It's not entirely clear to me why adding a package (build or runtime?)
dependency to LLVM creates a circular dependency.
If you build llvm, clang and lldb and make split packages from the
result you have a runtime dependency graph that is a tree with LLVM at
the root.
Maybe it's best not to digress. I agree that the standalone build of
lldb should work as expected.

Thanks,
Dan.

Hi,

I am trying to enable this on openSUSE but it seems to break
standalone lldb (note that we don't ship static libs):

cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
-DCMAKE_CXX_FLAGS=-stdlib=libc++ -DLLVM_BUILD_LLVM_DYLI
B=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_INSTALL_PREFIX=/usr
-DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_LIBDIR_SUFFIX=64 -DLLDB_PA
TH_TO_LLVM_BUILD=/usr -DLLDB_PATH_TO_CLANG_BUILD=/usr
-DLLVM_RUNTIME_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/buil
d/bin -DLLVM_LIBRARY_OUTPUT_INTDIR=/home/abuild/rpmbuild/BUILD/lldb-3.8.0/build/lib64
-DPYTHON_VERSION_MAJOR=2 -DPYTHON_VERSIO
N_MINOR=7 -G Ninja ..

[...]

CMake Error at /usr/share/llvm/cmake/LLVMExports.cmake:1011 (message):
  The imported target "LLVMSupport" references the file

      "/usr/lib64/libLLVMSupport.a"

Could you try appending ``-DLLVM_LINK_LLVM_DYLIB=ON`` to your CMake
invocation? It looks like lldb is using the ``llvm_add_library()``
function in the ``cmake/modules/AddLLVM.cmake`` (LLVM source tree)
which uses ``LLVM_LINK_LLVM_DYLIB`` (and a few other variables) to
decide whether to link against LLVM's static component libraries or
the monolithic shared library. I could be wrong on this though but it
should
be pretty quick to check.

I already use -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON

That'll teach me to suggest something without actually trying it. Then
I might have actually noticed you're already using that flag!

Let me see if I can reproduce this. Are you building on LLVM trunk on
the 3.8 release branch?

Trying the 3.8 branch atm.

Out of curiosity though if you're building packages on openSUSE why do
you need a standalone build? I thought the normal way to do this was
to build everything and then generate split packages?

Nice question. Since Mesa depends on llvm and Mesa is very crucial to
a distro bootstrap, adding any new dependency to llvm creates a
circular dependency which we can't have. And lldb itself brings python
dependency which complicates the issue (python has a Mesa dependency).
I may try to pre-generate files with swig to remove the dependency but
it would be nice if this worked as expected.

It's not entirely clear to me why adding a package (build or runtime?)
dependency to LLVM creates a circular dependency.
If you build llvm, clang and lldb and make split packages from the
result you have a runtime dependency graph that is a tree with LLVM at
the root.
Maybe it's best not to digress. I agree that the standalone build of
lldb should work as expected.

The cycle is

Mesa -> llvm -> python -> Mesa

@Brad: CC'ing you because I know you use
``llvm_map_components_to_libnames()`` so you will likely have
something to say about it breaking.

Okay Ismail, I took a much deeper look at this.

In LLDB the problem occurs when ``cmake/modules/LLDBStandalone.cmake``
tries to do

  # Import CMake library targets from LLVM and Clang.
  include("${LLDB_PATH_TO_LLVM_BUILD}/share/llvm/cmake/LLVMConfig.cmake")

``LLVMConfig.cmake`` will include ``LLVMExports.cmake`` which checks
that the exported targets actually exist. The problem here is that you
have removed those installed libraries from the install tree and the
build system has know way of knowing that you were going to do this.

The most obvious solution to me is to add an option to LLVM which
informs CMake that you do not want to install the individual component
libraries. If you don't need to install the component libraries that
means you don't need to export the targets to ``LLVMExports.cmake`` in
the install tree (note the ``LLVMExport.cmake`` in the build tree
should still contain the targets).

I tried implementing this by adding an extra option
(LLVM_INSTALL_COMPONENT_LIBRARIES) which can be set to OFF in the
attached patch (0001-Add-LLVM_INSTALL_COMPONENT_LIBRARIES-option-which-is.patch
, apply against LLVM). It is a hack and I do not recommend using this
in production. Instead it's more of a starting point for discussion. I
ran into lots of problems because if the component libraries are not
in the install tree then components that link against them and are
installed can't be so I had hack some of the cmake macros/functions to
link against libLLVM instead.

With this patch I built LLVM as so

cd /home/dsl11/dev/llvm-upstream/test_build/
mkdir install
cmake -DCMAKE_INSTALL_PREFIX=`realpath install`
-DLLVM_INSTALL_COMPONENT_LIBRARIES=OFF -DLLVM_BUILD_LLVM_DYLIB=ON
-DLLVM_LINK_LLVM_DYLIB=ON  -G Ninja ../src
ninja install

You will notice there are no ``install/lib/libLLVM*.a`` files
installed. At least I got that right :slight_smile:

Then I tried configuring LLD like so

cmake -DLLVM_LINK_LLVM_DYLIB=ON
-DLLDB_PATH_TO_LLVM_BUILD=/home/dsl11/dev/llvm-upstream/test_build/install
-DLLDB_PATH_TO_CLANG_BUILD=/home/dsl11/dev/llvm-upstream/test_build/install
../lldb-src

This didn't work out the box for me because it tries to pass
"${LLVM_MAIN_INCLUDE_DIR}" and "${CLANG_MAIN_INCLUDE_DIR}" to
``include_directories()`` which doesn't like being passed empty
strings. The attached patch for lldb ( lldb_no_need_src_tree.patch )
fixes that.

The configure succeed but with loads of warnings like this

CMake Warning (dev) in source/Plugins/ScriptInterpreter/Python/CMakeLists.txt:
  Policy CMP0022 is not set: INTERFACE_LINK_LIBRARIES defines the link
  interface.  Run "cmake --help-policy CMP0022" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "lldbPluginScriptInterpreterPython" has an INTERFACE_LINK_LIBRARIES
  property.  This should be preferred as the source of the link interface for
  this library but because CMP0022 is not set CMake is ignoring the property
  and using the link implementation as the link interface instead.

  INTERFACE_LINK_LIBRARIES:

    LLVM;LLVM

  Link implementation:

    LLVM

I think the reason for LLVM appearing twice in
INTERFACE_LINK_LIBRARIES is that ``add_lldb_library()`` uses
``llvm_add_library()`` which will add a dependency on the LLVM target.
A few lines later it then calls ``llvm_config()`` on it later which
will add the dependency again!

There's also warnings about mixing the old style
``target_link_libraries()`` with ``target_link_libraries(target
(PUBLIC|PRIVATE|INTERFACE))``.

The build doesn't succeed. For some reason when building liblldb.so
there are missing references to pthreads.

```
[ 81%] Linking CXX shared library ../../lib/liblldb.so
../../lib/liblldbHost.a(Host.cpp.o): In function
`lldb_private::Host::ThreadLocalStorageCreate(void (*)(void*))':
Host.cpp:(.text._ZN12lldb_private4Host24ThreadLocalStorageCreateEPFvPvE+0x1b):
undefined reference to `pthread_key_create'
...

This surprises me because ``libLLVM-3.9svn.so`` which is part of the
link line for ``liblldb.so`` (after all the static libraries) which is
linked against ``libpthread.so.0`` when I examine it with the ``ldd``
tool.

Anyway... I guess this is a reasonable starting point for discussing
how to do this properly. After looking at this I have some concerns:

* The way that LLVM components are linked against seems really messy
to me. We have this ``llvm_config()`` macro for linking against
components but also ``llvm_add_library()`` does this too. We don't
need two ways of doing the same thing. I think there should be one
clean way of doing this.

* If the static libraries are not installed then external projects
that use LLVM via [1] won't build because
``llvm_map_components_to_libnames()`` only works with the static
libraries. We need to provide something better that does the right
thing automatically but allows forcing to use the static components or
the dynamic library.

[1] http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project

HTH,
Dan.

0001-Add-LLVM_INSTALL_COMPONENT_LIBRARIES-option-which-is.patch (5.27 KB)

lldb_no_need_src_tree.patch (1.01 KB)

Hi,

@Brad: CC'ing you because I know you use
``llvm_map_components_to_libnames()`` so you will likely have
something to say about it breaking.

Okay Ismail, I took a much deeper look at this.

Thanks for the great analysis!

In LLDB the problem occurs when ``cmake/modules/LLDBStandalone.cmake``
tries to do

  # Import CMake library targets from LLVM and Clang.
  include("${LLDB_PATH_TO_LLVM_BUILD}/share/llvm/cmake/LLVMConfig.cmake")

``LLVMConfig.cmake`` will include ``LLVMExports.cmake`` which checks
that the exported targets actually exist. The problem here is that you
have removed those installed libraries from the install tree and the
build system has know way of knowing that you were going to do this.

The most obvious solution to me is to add an option to LLVM which
informs CMake that you do not want to install the individual component
libraries. If you don't need to install the component libraries that
means you don't need to export the targets to ``LLVMExports.cmake`` in
the install tree (note the ``LLVMExport.cmake`` in the build tree
should still contain the targets).

I tried implementing this by adding an extra option
(LLVM_INSTALL_COMPONENT_LIBRARIES) which can be set to OFF in the
attached patch (0001-Add-LLVM_INSTALL_COMPONENT_LIBRARIES-option-which-is.patch
, apply against LLVM). It is a hack and I do not recommend using this
in production. Instead it's more of a starting point for discussion. I
ran into lots of problems because if the component libraries are not
in the install tree then components that link against them and are
installed can't be so I had hack some of the cmake macros/functions to
link against libLLVM instead.

Indeed the correct solution would be such an option where we can skip
installing the components. The problem with installing the components
is that they are static libraries and in openSUSE we can't just ship
static libs to the end user. It would make updating libraries more
complicated and has security implications (even if it's less so for
llvm).

Regards,
ismail

@Brad: CC'ing you because I know you use
``llvm_map_components_to_libnames()`` so you will likely have
something to say about it breaking.

I'm using it in CastXML but that could easily be adapted to something
different if needed. The larger concern is that that API is the
documented way to use LLVM in an application with CMake so there could
be many such clients.

* The way that LLVM components are linked against seems really messy
to me. We have this ``llvm_config()`` macro for linking against
components but also ``llvm_add_library()`` does this too. We don't
need two ways of doing the same thing. I think there should be one
clean way of doing this.

* If the static libraries are not installed then external projects
that use LLVM via [1] won't build because
``llvm_map_components_to_libnames()`` only works with the static
libraries. We need to provide something better that does the right
thing automatically but allows forcing to use the static components or
the dynamic library.

IIUC the purpose of asking applications to map component names
instead of directly naming libraries is so that the set of actual
libraries can be changed out underneath while retaining the same
components. This abstraction means that if only the shared library
is installed then ``llvm_map_components_to_libnames()`` can simply
return it for all components built into it. Its implementation just
needs to be made conditional on the option for static libraries.

-Brad

Dan,

Thanks for looking so deeply at this. I think the fundamental problem is that we still have our infrastructure in an odd state as we’re moving off autoconf.

Brad’s explanation of the component mappings is correct, but with an added complication that we also have LLVMBuild which handles the dependency connections between static libraries. That allows a user to specify the specific components that their project depends on, and our build scripts will also add any components that those components depend on (see: `llvm_expand_dependencies`).

In our CMake `llvm_config` and `llvm_add_*` serve very different purposes. In general all the functions and macros that start with “${project}_*” are intended as internal to our CMake, and we really don’t encourage others to call them at all. We also have functions and macros that match the pattern "add_${project}_*" which are intended for use only within a given project, those are generally built on top of the “llvm_add_*” functions.

The intention is that out-of-tree clients (like CastXML) would use llvm_config, which would roughly map to the functionality of the llvm-config command line tool that we’ve used with autoconf. All of these code paths end up using “llvm_map_components_to_libnames", for the actual mapping. Part of the problem here is that we haven’t kept the functionality of that function in parity with the llvm-config tool.

Today the llvm-config tool knows which components get put into libLLVM.so, and knows how and when to map to the shared library. I think that is the missing bit of functionality here. I don’t even think adding that will be hard, we just need to make sure that LLVM_DYLIB_COMPONENTS, and LLVM_LINK_LLVM_DYLIB are both saved of in LLVMConfig.cmake.

I may have time to take a stab at this later today.

-Chris

Dan,

Thanks for looking so deeply at this. I think the fundamental problem is that we still have our infrastructure in an odd state as we’re moving off autoconf.

Brad’s explanation of the component mappings is correct, but with an added complication that we also have LLVMBuild which handles the dependency connections between static libraries. That allows a user to specify the specific components that their project depends on, and our build scripts will also add any components that those components depend on (see: `llvm_expand_dependencies`).

Eh? ``llvm_expand_dependencies()`` shouldn't be used. The only use in
LLVM's tree is being called indirectly via
``llvm_map_components_to_libraries()`` which is marked deprecated (has
been since LLVM 3.5 - we ought to remove it soon). I think
``llvm_map_components_to_libnames()`` to supposed to resolve all the
dependencies.

In our CMake `llvm_config` and `llvm_add_*` serve very different purposes. In general all the functions and macros that start with “${project}_*” are intended as internal to our CMake, and we really don’t encourage others to call them at all. We also have functions and macros that match the pattern "add_${project}_*" which are intended for use only within a given project, those are generally built on top of the “llvm_add_*” functions.

Our docs say differently. See [1] , we suggest using
``add_llvm_loadable_module()``.

The intention is that out-of-tree clients (like CastXML) would use llvm_config, which would roughly map to the functionality of the llvm-config command line tool that we’ve used with autoconf. All of these code paths end up using “llvm_map_components_to_libnames", for the actual mapping. Part of the problem here is that we haven’t kept the functionality of that function in parity with the llvm-config tool.

Today the llvm-config tool knows which components get put into libLLVM.so, and knows how and when to map to the shared library. I think that is the missing bit of functionality here. I don’t even think adding that will be hard, we just need to make sure that LLVM_DYLIB_COMPONENTS, and LLVM_LINK_LLVM_DYLIB are both saved of in LLVMConfig.cmake.

The ``LLVM_INSTALL_COMPONENT_LIBRARIES`` option I proposed (and gave a
rough patch for) would also need to be exported but I think
implementing that would be a lot easier after a refactor.

As Brad mentioned we do not want to tell people to move off
``llvm_map_components_to_libnames()`` just because it makes our
refactoring easier. ``llvm_map_components_to_libnames()`` is the
documented public interface since LLVM 3.5 and it would silly to
change it when we can easily avoid it. What I think we need to do is

- Remove ``llvm_map_components_to_libraries()`` and all the dead code
that results from that (i.e. ``expand_topologically`` and
``llvm_expand_dependencies``).
- Rename the current implementation of
``llvm_map_components_to_libnames()`` to
``llvm_map_components_to_module_libs()`` (need a better name for this,
we can't call it ``..._to_static_libnames`` because of the
``BUILD_SHARED_LIBS`` option)
- Implement ``llvm_map_components_to_libnames()`` to have a little
more functionality than before. Internally it may call
``llvm_map_components_to_module_libs()``.

The signature would be something like this. Sorry if I screw up
terminology (i.e. modules, components... I'm not sure what I'm
supposed to use here.)

llvm_map_component_to_libnames(<OUT_VAR> [ DYLIB_ONLY | MODULES_ONLY |
DYLIB_FALLBACK_MODULES ] <component1> <component2> ...)

The optional ``DYLIB_ONLY``, ``MODULES_ONLY`` and
``DYLIB_FALLBACK_MODULES`` arguments do the following

- ``DYLIB_ONLY`` - If specified will use the monolithic
``libLLVM.so``. If we know that this library was
not built then we throw a fatal error. If one of the requested
components was not built in the library then
throw a fatal error.

- ``MODULES_ONLY`` - If specified will use the modules (e.g.
LLVMSupport.a, etc...). When ``LLVM_INSTALL_COMPONENT_LIBRARIES``
(perhaps rename to LLVM_INSTALL_MODULE_LIBRARIES?)
  is implemented in the future and set to off (i.e. the module libs
won't get installed) throw a fatal error. We may need to be careful
here. The modules libraries are always available from the build tree
but not from the install tree.

- ``DYLIB_FALLBACK_MODULES`` - Try to use the monolithic
``libLLVM.so``. If the library wasn't built use the modules instead.
If some of the components aren't in the monolithic ``libLLVM.so`` then
use the modules for those
(i.e. the libraries returned is a mixture of the monolithic library
and the module libraries).

If neither of the above options are specified then the default in
``MODULES_ONLY`` (i.e. the existing behaviour). However when
``LLVM_INSTALL_COMPONENT_LIBRARIES`` is implemented and if it is set
to
off (i.e. the modules won't be installed) and then the default changes
to ``DYLIB_ONLY`` when being used from an install tree.

The point of the above is to give the client explicit control if they
want it (e.g. they might always want the monolithic libLLVM.so). The
default works in as many situations as possible.

Does this make sense? If so I'd happily have a go at this but I
wouldn't be able to take a look until next week.

[1] http://llvm.org/docs/CMake.html#developing-llvm-passes-out-of-source

I didn’t realize we had deprecated the dependency walking code, which actually makes me a little sad, but I’ve been thinking about this, and I think we’re making this all too hard on ourselves.

A lot of the work we’re doing here is actually just duplicating functionality already in llvm-config. Doing a normal in-tree build we can’t use llvm-config because we need the information at configuration time before llvm-config is built. For this use case, where you’ve already built LLVM and are essentially building LLDB as an out-of-tree project, we could use llvm-config because it should already be there.

Doing that would mean we don’t have to duplicate logic between the llvm-config tool and our CMake builds, and not duplicating things is generally good.

-Chris

I agree that duplicating functionality is usually not a good thing but
I see two reasons why we **should not** go down the route of invoking
the ``llvm-config`` executable for external projects

* As you said this some of this functionality basically needs be in
LLVM for its configure step. So we get the hard bit (mapping
components to module library names e.g. support -> libLLVMSupport.a)
for free. Supporting the dynamic lib case adds a small complication
but it's certainly possible

* Requiring clients to invoke (indirectly via whatever CMake interface
we provide) the ``llvm-config`` executable screws things up for those
doing cross compilation because ``llvm-config`` might not be compiled
to target the host machine.

I didn’t realize we had deprecated the dependency walking code, which actually makes me a little sad, but I’ve been thinking about this, and I think we’re making this all too hard on ourselves.

A lot of the work we’re doing here is actually just duplicating functionality already in llvm-config. Doing a normal in-tree build we can’t use llvm-config because we need the information at configuration time before llvm-config is built. For this use case, where you’ve already built LLVM and are essentially building LLDB as an out-of-tree project, we could use llvm-config because it should already be there.

Doing that would mean we don’t have to duplicate logic between the llvm-config tool and our CMake builds, and not duplicating things is generally good.

I agree that duplicating functionality is usually not a good thing but
I see two reasons why we **should not** go down the route of invoking
the ``llvm-config`` executable for external projects

* As you said this some of this functionality basically needs be in
LLVM for its configure step. So we get the hard bit (mapping
components to module library names e.g. support -> libLLVMSupport.a)
for free. Supporting the dynamic lib case adds a small complication
but it's certainly possible

I think that mapping Support -> libLLVMSupport is not the complicated part of what llvm-config does. The complicated part is what the deprecated support for dependency graph traversal did. Handling the dynamic lib case is just some extra functionality that would come along for the ride. llvm-config also has the benefit of working both with installed and non-installed build directories.

* Requiring clients to invoke (indirectly via whatever CMake interface
we provide) the ``llvm-config`` executable screws things up for those
doing cross compilation because ``llvm-config`` might not be compiled
to target the host machine.

I disagree. In cross-compilation CMake already generates host tablegen executables and a target to generate a host llvm-config. Adding the plumbing so that CMake does what autoconf used to do and installs the host llvm-config as llvm-config-host shouldn’t be very difficult.

-Chris