ThinLTO status in trunk?

First, kudos on the ThinLTO results reported in your blog post — they’re impressive and the system sounds really well engineered.

I’m starting to try it out on a large piece of software and I’d like to make sure I know what to expect. The blog said it will be available in clang-3.9 but both clang-3.8 and trunk seem to have some degree of support for it. What is the status of ThinLTO in 3.8 (preferably) or trunk (otherwise)? Are either of these usable with multiple threads? What about for a distributed build?

Thanks!

-—Vikram

// Vikram S. Adve
// Professor, Department of Computer Science
// University of Illinois at Urbana-Champaign
// vadve@illinois.edu
// http://llvm.org

Hi Vikram,

Thanks!

I’m not sure what part got committed in the 3.8 timeframe - it looks like that was released back in March? A number of fixes have gone in since then so I would stick with a more recent version. The ThinLTO in trunk and (presumably 3.9 which seems to have not yet gone out?) is complete and working with multiple threads using gold.

The distributed build support is also in (which needs support in the build system however), although I am fixing a few bugs right now that popped up in testing our internal apps. Which distributed build system do you use? We have support coming out in Bazel, which is our open sourced distributed build system.

Let me know if you run into any issues or have any other questions!

Teresa

Hi Vikram,

Thanks!

I’m not sure what part got committed in the 3.8 timeframe - it looks like that was released back in March? A number of fixes have gone in since then so I would stick with a more recent version. The ThinLTO in trunk and (presumably 3.9 which seems to have not yet gone out?) is complete and working with multiple threads using gold.

Sounds good. I will give this a spin.

The distributed build support is also in (which needs support in the build system however), although I am fixing a few bugs right now that popped up in testing our internal apps.

OK it will be a little while before I get to try that. And the build system for the set of applications I’m working with is quite painful, so I don’t know how long that will take, sigh.

Which distributed build system do you use? We have support coming out in Bazel, which is our open sourced distributed build system.

I don’t know yet — I’ll find out when I get to that stage. But it’s almost certainly something very simple or home-brewed.

Let me know if you run into any issues or have any other questions!

Teresa

Thanks,

-—Vikram

// Vikram S. Adve
// Professor, Department of Computer Science
// University of Illinois at Urbana-Champaign
// vadve@illinois.edu
// http://llvm.org

3.8 was branched in early January though.
It has some of the work-in-progress for ThinLTO, it “could” work in simple cases I guess, but it hasn’t really been tested.
Also, in 3.8, the state was basically what was presented at EuroLLVM, while (as explained in the blog post), we ended up redesigning it significantly.
In 3.9 we changed the bitcode format and broke backward compatibility: 3.9 won’t be able to read a ThinLTO bitcode generated by 3.8 (this is possible only because “ThinLTO was not supported in 3.8”).

Hi Vikram,

Thanks!

I’m not sure what part got committed in the 3.8 timeframe - it looks like that was released back in March?

3.8 was branched in early January though.
It has some of the work-in-progress for ThinLTO, it “could” work in simple cases I guess, but it hasn’t really been tested.
Also, in 3.8, the state was basically what was presented at EuroLLVM, while (as explained in the blog post), we ended up redesigning it significantly.

OK. I’m not really trying to use 3.8.

In 3.9 we changed the bitcode format and broke backward compatibility: 3.9 won’t be able to read a ThinLTO bitcode generated by 3.8 (this is possible only because “ThinLTO was not supported in 3.8”).

Since 3.9 isn’t quite out yet, is this also true for trunk, then? Thanks,

—Vikram

3.9 has been branched last week, so you can consider trunk “feature complete” for ThinLTO.

Hi Teresa,

Thanks, Andrey.

The reason is that we enable a more aggressive pass pipeline in the ThinLTO backends, Mehdi added this (see https://reviews.llvm.org/D17115). For full LTO it is too expensive to have the more aggressive pass pipeline.

Teresa

No, is a static decision on the optimization pipeline: we run strictly more optimization passes with ThinLTO.

See: populateThinLTOPassManager vs populateLTOPassManager: https://github.com/llvm-mirror/llvm/blob/master/lib/Transforms/IPO/PassManagerBuilder.cpp#L773

OK, got it.