LLVM bc converter from LLVM 3.9 to LLVM 3.1

Project Infrastructure LLVM Dev List Archives
1

Hi mailing list,

I has been working on a large project that is based on LLVM 3.1. Recently we are thinking to introduce an LLVM bc converter from LLVM 3.9 to LLVM 3.1, such that we can introduce some of the newest LLVM analyses and optimizations to our LLVM 3.1 based project.

Have you worked on similar things that converting LLVM bc downward and has anything to share?

Thanks
Hongbin

2

Hi Hongbin,

Android’s RenderScript uses a tool like that. You can find the sources here: https://android.googlesource.com/platform/frameworks/compile/libbcc/+/master/bcinfo/ .

Best,
Jakub

3

Thanks Jakub. Looks Like I didn’t rely all.

I also have a look at the code, looks like it directly parse the bitcode and build in memory representation in a different LLVM version than the bitcode. Is this correct?

According to your description, I guess the BitCodeWriter should be the one to do the bitcode version downgrade, right?

Does this took work on LLVM 3.9svn?

And could you give some hints about how you test the bitcode translator?

Thanks
Hongbin

4

I also have a look at the code, looks like it directly parse the bitcode and build in memory representation in a different LLVM version than the bitcode. Is this correct?

According to your description, I guess the BitCodeWriter should be the one to do the bitcode version downgrade, right?

I think so.

I don’t know much about it and don’t want to give you misleading information; I’m cc-ing Stephen.

5

Hi Hongbin,

6

Hi Steve,

Several people told me that LLVM TOT bcreader can read odder version of bitcode without any problem. Do you know the oldest version of bitcode that the TOT bitcode reader supports?

Also, why do you generate bitcode instead of the textural representation, i.e. the “ll” file, for older version of LLVM? I guess generating “ll” code is simpler.

Another approach I am thinking is linking to two different version of LLVM at the same time, and use the old version of IRBuilder to build bitcode from new version of LLVM IR. How do you think about this?

At last, we also do not support exception handling:)

Thanks.
Hongbin

7

Hi Hongbin,

8

No this is not the current situation: with the version numbering change, the policy changed as well. The new and current policy is: we support bitcode since 3.0 till we changed our mind.
Note also that LLVM 4.1 will be a minor “patch” release (like 3.8.1), you can expect LLVM 5.0 to be released ~6 months after LLVM 4.0.

9

Hi Hongbin,

Hi Steve,

Several people told me that LLVM TOT bcreader can read odder version of
bitcode without any problem. Do you know the oldest version of bitcode that
the TOT bitcode reader supports?

The current policy is that LLVM can read bitcode for any prior version
with the same major version number. Thus 3.9svn can read all the way back
to 3.0 (and also 3.1/3.2/...). It cannot read 2.9 or earlier bitcode. It
was also policy that the next major release (but only the .0 version) will
be able to read the prior version of bitcode too. Thus 4.0 is expected to
read 3.x bitcode as well. 4.1 is not expected to read 3.x bitcode, and it
will be an error to try to do so. With the new shift to version numbering
(i.e. major versions are coming out faster - check the recent archives
here), I am not sure how reliable bitcode will be as a storage format. This
is something that the RenderScript team is also planning for, since 4.0
will be the last supported release that can easily read the 3.2 IR that we
currently generate.

No this is not the current situation: with the version numbering change,
the policy changed as well. The new and current policy is: we support
bitcode since 3.0 till we changed our mind.
Note also that LLVM 4.1 will be a minor “patch” release (like 3.8.1), you
can expect LLVM 5.0 to be released ~6 months after LLVM 4.0.

Isn't that consistent with what I wrote (i.e. 4.0 will read 3.x, but
anything afterwards will not)? Is 4.0 not going to be able to read 3.x at
all? If so, that seems like a big surprise for quite a few out-of-tree
clients of LLVM. I didn't see anything altering the policy in that email
thread, and only assumed that the more rapid major release cycle was going
to make things worse (but still consistent with the old policy in theory).

Thanks,
Steve

10

Sorry if mis-explained, let me try again.

This is correct.

This isn’t. The policy was till 3.8:: "The bitcode format produced by a X.Y release will be readable by all following X.Z releases and the (X+1).0 release.” (Source http://llvm.org/releases/3.8.1/docs/DeveloperPolicy.html#ir-backwards-compatibility )
This policy is supporting your claim. However with the version numbering change, the policy is now: "The current LLVM version supports loading any bitcode since version 3.0.” (Source: http://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility )

Unless we change the policy (and I expect it would need a strong motivation), we will support the 3.0 bitcode in LLVM 5.0, LLVM 6.0, etc.

The intent is that the new version numbering does not lower the bar on the bitcode backward compatibility.

I hope it is more clear now.

11

Ah, I had missed that. That is great news to hear, actually. The RS team had already started planning how to adapt their translator to have a 4.0 (really 3.x) reader for when the dreaded 4.1/5.0 release would make things difficult. I guess they can reclaim some of their time now. :wink: Thank you so much for clarifying this.

Thanks,
Steve

12

Thanks Steve and Mehdi for the explanation.

Steve,

I am a little be confused by looking at the code in https://android.googlesource.com/platform/frameworks/compile/libbcc/+/master/bcinfo/BitReader_3_0/BitcodeReader.cpp. Looks like the BitcodeReader do some translation while reading the bitcode. If LLVM ToT can read the bitcode of LLVM 3.0, while can’t we just use the bitcode reader in LLVM ToT?

Also, the bitcode “downgrade” is done by :

// Use the LLVM 3.2 bitcode writer, instead of the top-of-tree version.
llvm_3_2::WriteBitcodeToFile(module, OS);

Is the corresponding source code also available? if so could you point me to the corresponding file?

Thanks again
Hongbin

13

Thanks Steve and Mehdi for the explanation.

Steve,

I am a little be confused by looking at the code in
bcinfo/BitReader_3_0/BitcodeReader.cpp - platform/frameworks/compile/libbcc - Git at Google.
Looks like the BitcodeReader do some translation while reading the bitcode.
If LLVM ToT can read the bitcode of LLVM 3.0, while can't we just use the
bitcode reader in LLVM ToT?

Ah, 3_0 is a bit of a misnomer. It is actually a 2.9 (pre-3.0) bitcode
format that this one reads. I would suggest that you (and most everyone
else on Earth, actually) ignore the readers that we have. We had an
unstable snapshot of LLVM when we first released, and nobody noticed.
Unfortunately, LLVM doesn't respect non-release bitcode formats, so we had
to check in our own readers to handle those cases.

Also, the bitcode "downgrade" is done by :
// Use the LLVM 3.2 bitcode writer, instead of the top-of-tree version.
llvm_3_2::WriteBitcodeToFile(module, OS);

Is the corresponding source code also available? if so could you point me
to the corresponding file?

https://android.googlesource.com/platform/frameworks/compile/slang/+/refs/heads/master/BitWriter_3_2/
is the location I mentioned before for the writer. There are 2 different
RenderScript projects that contain our bitcode libraries (one is readers -
libbcc, the other holds the writers - slang).

Thanks,
Steve

14

Thanks

15

Minor point: the patch release after 4.0 will be 4.0.1, to try to
avoid confusion about what kind of release it is.