Improving deb packages

Hi LLVM,

I complained about the deb packages couple of times previously, even fixed some issues in packaging. I’m mostly interested in having reliable share cmake files available in llvm-dev packages. The version 3.7 was fine, but 3.8+ have regressions.

I’m not here to blame anybody. I would like to identify the issues and discuss long term solutions.

I started with building very simple test framework that checks different Ubuntu/Debian versions and currently supported LLVM versions. The first and only test just checks find_package(LLVM CONFIG) cmake function.
https://github.com/chfast/llvm-apt-tests/blob/master/CMakeLists.txt

I’ve tested {3.8, 3.9, 4.0} x {xenial, jessie} using docker images and Travis CI.
https://travis-ci.org/chfast/llvm-apt-tests/builds/146508275

As you can see, only the 3.8 on Jessie passed the test.

Issues I’ve identified:

  1. Packaging adds version prefix to binaries, directories, etc. – eg. llc is renamed to llc-3.8. I’m not sure how it is done, but maybe we should add support for such feature to LLVM’s cmake?

  2. Default install location for cmake shared files is /lib/llvm/share/llvm/cmake. find_package() is not able to find them there as it is non-standard path. find_package() needs a hint, like setting LLVM_DIR variable as in https://github.com/chfast/llvm-apt-tests/blob/master/configure.sh#L6
    If we moved the cmake shared files to /lib/llvm/cmake no hint would be needed.

  3. On Ubuntu, packaging script moves cmake shared files to /share/llvm-X.Y/cmake. I’m guessing to solve the issue (2). find_package() is able to find LLVMConfig.cmake file without any hint, but this file contains absolute paths referencing previous locations of other files. You usually get issues like this one:

    CMake Error at /usr/share/llvm-3.8/cmake/LLVMConfig.cmake:181 (include):
    include could not find load file:
    /usr/share/llvm/cmake/LLVM-Config.cmake

    Maybe it is good idea to include other cmake files assuming there are located next to the main file instead of relying on absolute paths.

  4. It’s a bit weird Debian and Ubuntu packages has different layout of installed shared files.

  5. Packages 3.9+ does not have any cmake’s shared files, just empty dirs where those files are supposed to be. That might be a bug in the latest packaging script.

  • Paweł

Sure, thanks for doing that! Nice work! I am not surprised that 3.8 is passing the test, this is the one I focused the most. Brad reported this bug for 3.9 and 4.0 I just applied them to the branches (I was in holidays before) Do you mind if I steal that and I add this to my CI to test that we don’t regress? or are you planning to maintain it? This is done in debian/rules, by creating symlink We do that to make sure that the different version of llvm/clang are co-installable. llc and others are provided by this llvm defaults ( ). Having this available upstream would be good but this hasn’t been a source of troubles in the past. This should be fixed This is probably related to llvm-defaults behavior. I don’t think it is the case. I am not making any changes in Ubuntu packages. In some cases, Ubuntu official packages are patched from Debian’s but patches are usually forwarded to me. Could you report a bug? Thanks for the feedback, this is appreciated! Sylvestre

Hello,

+Andrew and Brad who did most of the work on cmake.
To be honest, I am not a cmake expert, any help to improve the situation is appreciated.

Hi LLVM,

I complained about the deb packages couple of times previously, even fixed some issues in packaging. I’m mostly interested in having reliable share cmake files available in llvm-dev packages. The version 3.7 was fine, but 3.8+ have regressions.

I’m not here to blame anybody. I would like to identify the issues and discuss long term solutions.

Sure, thanks for doing that!

I started with building very simple test framework that checks different Ubuntu/Debian versions and currently supported LLVM versions. The first and only test just checks find_package(LLVM CONFIG) cmake function.
https://github.com/chfast/llvm-apt-tests/blob/master/CMakeLists.txt

I’ve tested {3.8, 3.9, 4.0} x {xenial, jessie} using docker images and Travis CI.
https://travis-ci.org/chfast/llvm-apt-tests/builds/146508275

As you can see, only the 3.8 on Jessie passed the test.

Nice work! I am not surprised that 3.8 is passing the test, this is the one I focused the most.
Brad reported this bug https://llvm.org/bugs/show_bug.cgi?id=28325 for 3.9 and 4.0
I just applied them to the branches (I was in holidays before)

Do you mind if I steal that and I add this to my CI to test that we don’t regress? or are you planning to maintain it?

No problem, you can steal whatever you want. What CI do you have in mind? I could join in and help maintaining it there.

Issues I’ve identified:

  1. Packaging adds version prefix to binaries, directories, etc. – eg. llc is renamed to llc-3.8. I’m not sure how it is done, but maybe we should add support for such feature to LLVM’s cmake?

This is done in debian/rules, by creating symlink http://anonscm.debian.org/viewvc/pkg-llvm/llvm-toolchain/branches/snapshot/debian/rules?revision=2020&view=markup#l319
We do that to make sure that the different version of llvm/clang are co-installable. llc and others are provided by this llvm defaults ( https://packages.qa.debian.org/l/llvm-defaults.html ).
Having this available upstream would be good but this hasn’t been a source of troubles in the past.

  1. Default install location for cmake shared files is /lib/llvm/share/llvm/cmake. find_package() is not able to find them there as it is non-standard path. find_package() needs a hint, like setting LLVM_DIR variable as in https://github.com/chfast/llvm-apt-tests/blob/master/configure.sh#L6
    If we moved the cmake shared files to /lib/llvm/cmake no hint would be needed.

This should be fixed

  1. On Ubuntu, packaging script moves cmake shared files to /share/llvm-X.Y/cmake. I’m guessing to solve the issue (2). find_package() is able to find LLVMConfig.cmake file without any hint, but this file contains absolute paths referencing previous locations of other files. You usually get issues like this one:

    CMake Error at /usr/share/llvm-3.8/cmake/LLVMConfig.cmake:181 (include):
    include could not find load file:
    /usr/share/llvm/cmake/LLVM-Config.cmake

    Maybe it is good idea to include other cmake files assuming there are located next to the main file instead of relying on absolute paths.

This is probably related to llvm-defaults behavior.

  1. It’s a bit weird Debian and Ubuntu packages has different layout of installed shared files.

I don’t think it is the case. I am not making any changes in Ubuntu packages. In some cases, Ubuntu official packages are patched from Debian’s but patches are usually forwarded to me.

I’ve rechecked. You are right, however Ubuntu packages it differently, but we should not care about that. I have to modify my test config to force installing llvm-3.8-dev from apt.llvm.org.

  1. Packages 3.9+ does not have any cmake’s shared files, just empty dirs where those files are supposed to be. That might be a bug in the latest packaging script.

Could you report a bug?

I think I was wrong about that too.

Thanks for the feedback, this is appreciated!

Sorry for late answer, was on holidays and busy with other projects. I will try proceed with the idea to move cmake shared files withing LLVM dir structure.

  • Paweł

    Do you mind if I steal that and I add this to my CI to test that we don't regress? or are you planning to maintain it?

No problem, you can steal whatever you want. What CI do you have in mind? I could join in and help maintaining it there.

Packages on apt.llvm.org are built using this jenkins instance:
http://llvm-jenkins.debian.net/

    Issues I've identified:

     1. Packaging adds version prefix to binaries, directories, etc. -- eg. llc is renamed to llc-3.8. I'm not sure how it is done, but maybe we should add support for such feature to LLVM's cmake?

    This is done in debian/rules, by creating symlink http://anonscm.debian.org/viewvc/pkg-llvm/llvm-toolchain/branches/snapshot/debian/rules?revision=2020&view=markup#l319
    We do that to make sure that the different version of llvm/clang are co-installable. llc and others are provided by this llvm defaults ( Debian Package Tracking System - llvm-defaults ).
    Having this available upstream would be good but this hasn't been a source of troubles in the past.

     1. Default install location for cmake shared files is <prefix>/lib/llvm/share/llvm/cmake. find_package() is not able to find them there as it is non-standard path. find_package() needs a hint, like setting LLVM_DIR variable as
        in llvm-apt-tests/configure.sh at master · chfast/llvm-apt-tests · GitHub
        If we moved the cmake shared files to <prefix>/lib/llvm/cmake no hint would be needed.

    This should be fixed

     1. On Ubuntu, packaging script moves cmake shared files to <prefix>/share/llvm-X.Y/cmake. I'm guessing to solve the issue (2). find_package() is able to find LLVMConfig.cmake file without any hint, but this file contains absolute paths referencing previous locations of other files. You
        usually get issues like this one:

        CMake Error at /usr/share/llvm-3.8/cmake/LLVMConfig.cmake:181 (include):
        include could not find load file:
        /usr/share/llvm/cmake/LLVM-Config.cmake

        Maybe it is good idea to include other cmake files assuming there are located next to the main file instead of relying on absolute paths.

    This is probably related to llvm-defaults behavior.

     1. It's a bit weird Debian and Ubuntu packages has different layout of installed shared files.

    I don't think it is the case. I am not making any changes in Ubuntu packages. In some cases, Ubuntu official packages are patched from Debian's but patches are usually forwarded to me.

I've rechecked. You are right, however Ubuntu packages it differently, but we should not care about that. I have to modify my test config to force installing llvm-3.8-dev from apt.llvm.org <http://apt.llvm.org>.

If you are talking about Ubuntu packages into the Ubuntu archives, Ubuntu devs are not making this kind of changes.
Minor changes are done: https://patches.ubuntu.com/l/llvm-toolchain-3.8/llvm-toolchain-3.8_1:3.8.1-8ubuntu2.patch
(and I don't agree with those, so, I haven't merged in Debian's packages).

Sorry for late answer, was on holidays and busy with other projects. I will try proceed with the idea to move cmake shared files withing LLVM dir structure.

No worries :wink:

S