Building bitcode modules

Hi,

I have updated llvm to the 113aa8612010434069fc5b5e09f6b2667e03e173 git
commit. I have a small llvm project that builds a bitcode library, so I
used MODULE_NAME=foo to build foo.bc. Now, I cannot build the library,
because a compatible compiler cannot be found (Makefile.rules:1052).

I have seen that configure.ac has been modified, so:

1) there is an update in progress?
2) I have to use cmake instead of autoconf?
3) The way to build bitcode modules has changed?

Thank you,
speziale.ettore@gmail.com

Hi,

I have updated llvm to the 113aa8612010434069fc5b5e09f6b2667e03e173 git
commit. I have a small llvm project that builds a bitcode library, so I
used MODULE_NAME=foo to build foo.bc. Now, I cannot build the library,
because a compatible compiler cannot be found (Makefile.rules:1052).

I have seen that configure.ac has been modified, so:

1) there is an update in progress?

No

2) I have to use cmake instead of autoconf?

Definitely not.

3) The way to build bitcode modules has changed?

Not really.

What compiler are you using to build with? I've made it default to clang with less looking around for llvm-gcc, so there may be an issue there. What is your configure line? What host are you trying to build on?

-eric

Hi,

What compiler are you using to build with? I've made it default to clang with less looking around for llvm-gcc, so there may be an issue there. What is your configure line? What host are you trying to build on?

First I have compiled llvm/clang compiled with "gcc (Gentoo 4.5.2 p1.1,
pie-0.4.5) 4.5.2". Then I have installed llvm/clang. They are in the
path:

$ clang --version
clang version 3.0 (http://llvm.org/git/clang.git
53ce37ff4b44c59e5ac682d4558df980a8a3fa33)
Target: x86_64-unknown-linux-gnu
Thread model: posix

My llvm configure line is:

../src/configure --prefix=/home/ettore/work/llvm/root

Host:

Linux ironman 2.6.35-gentoo-r10 #1 SMP Fri Oct 15 11:35:09 CEST 2010
x86_64 Intel(R) Xeon(R) CPU E5335 @ 2.00GHz GenuineIntel GNU/Linux
I have added a bitcode library to the sample project

I have modified the sample project to show my problem -- patch attached.

The warning comes from Makefile.rules:1192. The check is performed over
LLVMCC, set at Makefile.rules:428-443. Setting depends on
LLVMCC_OPTIONS, CLANGPATH, and ENABLE_BUILT_CLANG. Previously -- I have
tried with 3699261d3f49d65cee6a645c849c41cdca51a01f -- these vars was
set throught the configure script, now they are not set.

Thank you for the assistance,
speziale.ettore@gmail.com

bitcode-module.diff (1.14 KB)

Aha. I see.

I'd forgotten to remove that code when I moved the rest of the compiler support out of the top level configure for llvm. I hadn't realized anyone was using that functionality to build modules.

Out of curiosity what are you using this for? I.e. is it absolutely important to build bitcode modules within the llvm build system?

-eric

Hi,

What compiler are you using to build with? I've made it default to clang with less looking around for llvm-gcc, so there may be an issue there. What is your configure line? What host are you trying to build on?

First I have compiled llvm/clang compiled with "gcc (Gentoo 4.5.2 p1.1,
pie-0.4.5) 4.5.2". Then I have installed llvm/clang. They are in the
path:

$ clang --version
clang version 3.0 (http://llvm.org/git/clang.git
53ce37ff4b44c59e5ac682d4558df980a8a3fa33)
Target: x86_64-unknown-linux-gnu
Thread model: posix

My llvm configure line is:

../src/configure --prefix=/home/ettore/work/llvm/root

Host:

Linux ironman 2.6.35-gentoo-r10 #1 SMP Fri Oct 15 11:35:09 CEST 2010
x86_64 Intel(R) Xeon(R) CPU E5335 @ 2.00GHz GenuineIntel GNU/Linux
I have added a bitcode library to the sample project

I have modified the sample project to show my problem -- patch attached.

The warning comes from Makefile.rules:1192. The check is performed over
LLVMCC, set at Makefile.rules:428-443. Setting depends on
LLVMCC_OPTIONS, CLANGPATH, and ENABLE_BUILT_CLANG. Previously -- I have
tried with 3699261d3f49d65cee6a645c849c41cdca51a01f -- these vars was
set throught the configure script, now they are not set.

Aha. I see.

I'd forgotten to remove that code when I moved the rest of the compiler support out of the top level configure for llvm. I hadn't realized anyone was using that functionality to build modules.

Out of curiosity what are you using this for? I.e. is it absolutely important to build bitcode modules within the llvm build system?

(I'm jumping into the middle of this conversation as it looks like you're discussing something that might be relevant to my work. Sorry I'm not up to speed on the full context of the discussion...)

If you are asking whether anyone is using machinery in LLVM's build system to compile programs into LLVM bitcode files, the answer is yes. The LLVM Makefile machinery is used not only by LLVM but by LLVM sub-projects like SAFECode, Automatic Pool Allocation, and others. Practically every project I work on (either research or open-source) is organized as a sub-project of LLVM.

We compile our run-time libraries into LLVM bitcode files so that their functions can be inter-procedurally inlined and optimized by LLVM (either by llvm-ld or by libLTO). This is simple to do as defining a macro in the Makefile turns this feature on (BYTECODE_LIBRARY=1, IIRC).

If you remove this feature, I will most likely have to re-implement it in all of our projects' Makefiles. I may have to update our test suite Makefiles as well. I'm not sure how time consuming that would be, but I'd rather not have to change all of my projects if I don't have to, even if the change is trivial.

If you have already removed this feature, can you please revert the commits and add it back?

-- John T.

(I'm jumping into the middle of this conversation as it looks like you're discussing something that might be relevant to my work. Sorry I'm not up to speed on the full context of the discussion...)

If you are asking whether anyone is using machinery in LLVM's build system to compile programs into LLVM bitcode files, the answer is yes. The LLVM Makefile machinery is used not only by LLVM but by LLVM sub-projects like SAFECode, Automatic Pool Allocation, and others. Practically every project I work on (either research or open-source) is organized as a sub-project of LLVM.

We compile our run-time libraries into LLVM bitcode files so that their functions can be inter-procedurally inlined and optimized by LLVM (either by llvm-ld or by libLTO). This is simple to do as defining a macro in the Makefile turns this feature on (BYTECODE_LIBRARY=1, IIRC).

If you remove this feature, I will most likely have to re-implement it in all of our projects' Makefiles. I may have to update our test suite Makefiles as well. I'm not sure how time consuming that would be, but I'd rather not have to change all of my projects if I don't have to, even if the change is trivial.

If you have already removed this feature, can you please revert the commits and add it back?

It wouldn't be too bad. It was part of some larger commits, but I'll rework it a little cleaner and add in just the detection of a compiler that can emit bitcode.

Thanks for the note!

-eric

(I'm jumping into the middle of this conversation as it looks like you're discussing something that might be relevant to my work. Sorry I'm not up to speed on the full context of the discussion...)

If you are asking whether anyone is using machinery in LLVM's build system to compile programs into LLVM bitcode files, the answer is yes. The LLVM Makefile machinery is used not only by LLVM but by LLVM sub-projects like SAFECode, Automatic Pool Allocation, and others. Practically every project I work on (either research or open-source) is organized as a sub-project of LLVM.

We compile our run-time libraries into LLVM bitcode files so that their functions can be inter-procedurally inlined and optimized by LLVM (either by llvm-ld or by libLTO). This is simple to do as defining a macro in the Makefile turns this feature on (BYTECODE_LIBRARY=1, IIRC).

If you remove this feature, I will most likely have to re-implement it in all of our projects' Makefiles. I may have to update our test suite Makefiles as well. I'm not sure how time consuming that would be, but I'd rather not have to change all of my projects if I don't have to, even if the change is trivial.

If you have already removed this feature, can you please revert the commits and add it back?

It wouldn't be too bad. It was part of some larger commits, but I'll rework it a little cleaner and add in just the detection of a compiler that can emit bitcode.

I'm not exactly sure what you're saying. Are you saying that you will be maintaining this feature as it was in previous versions of LLVM, or are you saying that you're adding an alternate feature which will require that I adapt our Makefiles to use?

-- John T.

I'll be doing a) - there's no reason for you to need to update your makefiles. The current test-suite has the bits in it, I just moved it down to the test suite because nothing in the actual llvm tree depends on bit code being constructed.

On at least darwin and linux platforms -O4 will do a full interprocedural build. Does that not work for you?

-eric

-eric

I think you misunderstand. The runtime libraries for my projects build a bitcode library (i.e., when I do a top-level "make" in my projects, bitcode libraries are built); they rely on LLVM's build system to do this. If I have to change those Makefiles to use something like -emit-llvm or -O4, then not only will I spend my time updating the Makefiles that build my libraries, but I will also spend time updating the Makefiles that link in those libraries (because they'll no longer be suffixed with .bca).

Moving the feature to test-suite doesn't fix anything. The LLVM build system is designed so that it can be easily reused by projects (a feature I implemented in the build system in the early days). The test-suite build system is not quite so easily reused (or, at least, it's not designed to be, so I don't expect it to be). I'd probably be better off adding custom build rules to my projects' Makefiles than trying to reuse test-suite Makefiles.

I can certainly make the changes, and I don't think it'll take too much of my time. The question I have is why are you removing a feature that I and other people are using? If removing the feature makes LLVM significantly easier to maintain or paves the way for a new feature in the build system, then the inconvenience I'll face is worth it. On the other hand, if you're just removing the feature because you don't see a need for it, then from my perspective, I'm inconvenienced for nothing.

-- John T.

(I'm jumping into the middle of this conversation as it looks like you're discussing something that might be relevant to my work. Sorry I'm not up to speed on the full context of the discussion...)

If you are asking whether anyone is using machinery in LLVM's build system to compile programs into LLVM bitcode files, the answer is yes. The LLVM Makefile machinery is used not only by LLVM but by LLVM sub-projects like SAFECode, Automatic Pool Allocation, and others. Practically every project I work on (either research or open-source) is organized as a sub-project of LLVM.

We compile our run-time libraries into LLVM bitcode files so that their functions can be inter-procedurally inlined and optimized by LLVM (either by llvm-ld or by libLTO). This is simple to do as defining a macro in the Makefile turns this feature on (BYTECODE_LIBRARY=1, IIRC).

If you remove this feature, I will most likely have to re-implement it in all of our projects' Makefiles. I may have to update our test suite Makefiles as well. I'm not sure how time consuming that would be, but I'd rather not have to change all of my projects if I don't have to, even if the change is trivial.

If you have already removed this feature, can you please revert the commits and add it back?

It wouldn't be too bad. It was part of some larger commits, but I'll rework it a little cleaner and add in just the detection of a compiler that can emit bitcode.

I'm not exactly sure what you're saying. Are you saying that you will be maintaining this feature as it was in previous versions of LLVM, or are you saying that you're adding an alternate feature which will require that I adapt our Makefiles to use?

I'll be doing a) - there's no reason for you to need to update your makefiles. The current test-suite has the bits in it, I just moved it down to the test suite because nothing in the actual llvm tree depends on bit code being constructed.

I think you misunderstand.

No, I didn't.

The runtime libraries for my projects build a bitcode library (i.e., when I do a top-level "make" in my projects, bitcode libraries are built); they rely on LLVM's build system to do this. If I have to change those Makefiles to use something like -emit-llvm or -O4, then not only will I spend my time updating the Makefiles that build my libraries, but I will also spend time updating the Makefiles that link in those libraries (because they'll no longer be suffixed with .bca).

True, but that's not that bad IMO. It's how every other project that wants this sort of support works.

Moving the feature to test-suite doesn't fix anything.

Well, it fixes the need for llvm to bother trying to figure out something that it doesn't need to know.

The LLVM build system is designed so that it can be easily reused by projects (a feature I implemented in the build system in the early days). The test-suite build system is not quite so easily reused (or, at least, it's not designed to be, so I don't expect it to be). I'd probably be better off adding custom build rules to my projects' Makefiles than trying to reuse test-suite Makefiles.

I wasn't suggesting this, not even remotely.

I can certainly make the changes, and I don't think it'll take too much of my time. The question I have is why are you removing a feature that I and other people are using? If removing the feature makes LLVM significantly easier to maintain or paves the way for a new feature in the build system, then the inconvenience I'll face is worth it. On the other hand, if you're just removing the feature because you don't see a need for it, then from my perspective, I'm inconvenienced for nothing.

As far as I can tell you're one of the only people using it and none of the work is in the mainline llvm tree so by that definition it's an unused "feature" of the build system that wasn't necessary at the top level. llvm the library doesn't care about how to emit IR while being built. Subprojects may and the test-suite cared.

That said, I already said I was going to bring some level of support back so that the bitcode modules would continue to work. I don't see a reason for this feature, but also don't have any compelling reason to regress it on you.

-eric