The new pass manager has been on by default since the 13 branch. Now that we’ve branched for 14, I’d like to start the process of deprecating and removing legacy pass manager support for the optimization pipeline. This includes clang, opt, and lld LTO support.
Note that this doesn’t apply to the codegen pipeline since there’s no new pass manager support for that yet.
Are there any objections?
"deprecating" and 'removing" are different.
For 14.0.0, do you plan that -DLLVM_ENABLE_NEW_PASS_MANAGER=off will
give a warning or will be completely unsupported?
I probably should have said “deprecating and removing”.
My intention was to remove it completely. -DLLVM_ENABLE_NEW_PASS_MANAGER wouldn’t do anything and we’d remove the -flegacy-pass-manaager/-fexperimental-new-pass-manager flags, as well as the corresponding lld flags.
Since deprecating something is giving warning that it’s going to be removed, I don’t think it’s fair to deprecate and remove in one step. If legacy pass manager isn’t currently formally deprecated (as in, loudly complains when you try to use it), then I’d personally like to see it deprecated for a release before any steps to completely remove it are taken.
Since deprecating something is giving warning that it’s going to be removed, I don’t think it’s fair to deprecate and remove in one step. If legacy pass manager isn’t currently formally deprecated (as in, loudly complains when you try to use it), then I’d personally like to see it deprecated for a release before any steps to completely remove it are taken.
I just wanted to say the same thing:)
I don't mind that we install a warning for -DLLVM_ENABLE_NEW_PASS_MANAGER=off
in the release/13.x branch so that the signal is clearer to downstream users.
If nobody is setting -DLLVM_ENABLE_NEW_PASS_MANAGER=off then I’m not sure that we need to go through the whole deprecation dance. Is there anybody doing that?
But if we do the deprecation dance, should it be a CMake warning or a clang warning? I’m not sure people will notice a CMake warning.And would putting that in the 13.x branch be good enough to start removing after 14.x?
Downstreams could be re-enabling the legacy pass manager in CMake. It’s still there, and still supported. If a downstream has made non-trivial changes to the pass pipeline for legacy pass manager, it’s going to be a ton of work to ensure performance parity when enabling new pass manager.
I think, if LLVM is built with LLVM_ENABLE_NEW_PASS_MANAGER set to OFF, then you should get a big scary warning at CMake configure time, but clang/opt should not complain. If LLVM is built with DLLVM_ENABLE_NEW_PASS_MANAGER set to ON, but the flag is passed to clang/opt to build using the legacy pass manager, you should get a warning from clang/opt.
In my opinion, if these deprecation warnings make it in for LLVM 13, then it’s fine to begin removing LPM in LLVM 14.
thanks,
Chris Tetreault
Let me clarify that when I say “it’s fine to begin removing LPM in LLVM 14”, I mean that it’s fine to begin removing LPM once LLVM 14 is released, and the version string in main is set to 15.
Let me clarify that when I say “it’s fine to begin removing LPM in LLVM 14”, I mean that it’s fine to begin removing LPM once LLVM 14 is released, and the version string in main is set to 15.
While I think such a deprecation and removal policy may be fine for many non-trivial things, asking
this may be too much for the pass manager. With a grain of salt, "downstreams are on their own." New
PM migration and legacy PM removal has been repeatedly forewarned. I think llvm-project has done
above and beyond what it was expected. The flip in git was done in February.
After 13.0.0 is released for, say 2 months, which additional attests its robustness, I think
removing non-codegen legacy PM pieces should be fine.
Legacy pass manager is a very large piece of infrastructure in LLVM, and for that reason I think it's especially important to provide a good long deprecation period. Up until now, there's been no deadline for its removal, so for us to say "in 2 months passes are going to start getting deleted" is unfair. As it stands, NPM is a many-years-long project, and it's always been the case that it's going to replace LPM "soon", but there's never been a date. (if I'm wrong, please correct me) Given the level of effort required to migrate to new pass manager, it no surprise that users may not have done it until "soon" became "on this specific date". I know that "downstreams are on their own" is our policy, but it's also the case that we value not making things difficult for downstreams. I know I've repeatedly been told to back out changes due to push back from downstreams.
I don't think a 6 month deprecation period is a big ask. This RFC is essentially proposing "let's remove some dead code", so I don't see how waiting is going to block anybody's work. And as Björn mentioned in his other email (that I assume he meant to CC the lists on, but I'll refrain from quoting him verbatim), there are things we can get started on in the meantime, such as cleaning up the lit tests and changing CMake defaults. But please, let's not remove any actual code until we've had a formal deprecation period.
If the concern is that people will keep pushing code that uses LPM infrastructure, then we can mark it deprecated so the build bots reject it. Preferably we'd do some sort of `#define LLVM_NPM_ATTRIBUTE_DEPRECATED(decl, msg) LLVM_ATTRIBUTE_DEPRECATED(decl, msg)` thing so that downstreams can just redefine one thing instead of deleting a bunch of deprecation flags.
Thanks,
Chris Tetreault
Now with llvm-dev as receiver (thanks to Chris T for noticing the mistake).
Why is that? Is there a significant cost to keep the pass manager in LLVM for the next 5 months?
Otherwise it would be nice to release 14 with a release notes that indicates what will be removed in 15 on this aspect.
There are likely things that can be removed quickly already (like any uses of the LPM in clang/lld/…) and keep only the support in opt for now.
I’d vote for immediate removal. I don’t have much sympathy for downstream consumers who haven’t moved. This effort has been underway for literal years. Many - though not by any means all - downstream projects moved before upstream finally switched. Let’s put a nail in this coffin, and remove code aggressively.
Supporting both has serious ongoing costs. As a real example, I have twice spent time in the last two weeks tracking down some odd quirk of the unrolling implementation to find it supports some quirk of the legacy pass. That slows down development, compromises quality, and is generally a “bad thing”.
We might want to wait a couple of weeks/months to ensure stability, but we should only consider the needs to the upstream project itself when doing so. Giving downstream projects time to react should be an explicit non-goal.
Philip
p.s. I don’t expect this to be the actual decision reached, but since I only see opinions down-thread arguing for migration windows, I wanted to make a point of sharing the opposite opinion. Fair warning, I probably won’t reply to this thread further. I don’t have sufficient interest in the topic to make it worthwhile.
Are https://reviews.llvm.org/D108789 and https://reviews.llvm.org/D108775 sufficient if we cherrypick them into 13?
I think that’s a sufficiently obnoxious warning. I still strongly prefer that no removals of functionality come until we branch for LLVM 14, but I think this will do for a notice of deprecation.
Thanks,
Chris Tetreault
These revisions are deprecating (IIUC):
These don’t seem like the most impactful for downstream users, are they?
On the other hand, what seems to me to be critical to clarify is if you also intend to deprecate other things, like the use of the legacy pass manager in opt (or as a pass manager in a downstream tool/compiler)? Are we gonna also remove all the legacy adapters that make the passes work in the legacy pass manager as well?
This looks more important to me because this affects directly how other compilers are built on top of LLVM and push them to migrate, while the deprecation revision you sent may just have no effect on them.
Thanks,
This is a good point. I’ve stated my preferences, but I think we can all agree that it would be good to settle on a concrete timeline of what’s getting removed, and when.
Thanks,
Chris Tetreault
Sounds to me like we need to have a short description of exactly what functionality is going to be deleted, and then put it at the top of the release notes for llvm 13.
These revisions are deprecating (IIUC):
1) Clang user flags to use the legacy pass manager.
2) The CMake build flag that defines the *default* pass manager.These don't seem like the most impactful for downstream users, are they?
The previous disabling mechanism
-DENABLE_EXPERIMENTAL_NEW_PASS_MANAGER=off is ignored in main and
release/13.x.
If a user switches (-DLLVM_ENABLE_NEW_PASS_MANAGER=off), they should
notice the warning.
So I think we don't need something like
-DLLVM_FORCE_USE_OLD_TOOLCHAIN=on which was done in 2019 to bump the
toolchain requirement.
What legacy PM stuff I’d like to remove soonish:
Support for the CMake flag DLLVM_ENABLE_NEW_PASS_MANAGER=off (which is related to defaults for the next 3 items)
LTO support in various linkers
Clang support for the optimization pipeline
opt support for translating -instcombine into -passes=instcombine, will require updating many tests
PassManagerBuilder for building an optimization pipeline with the legacy PM (requires all of the above to be done first)
I can add these to the 13.x release notes.
If there are any existing users right now who continuously test against LLVM trunk and would like some time to port legacy PM passes to the new PM I’m happy to wait a couple of weeks/a month (and happy to answer any questions), but otherwise we shouldn’t worry about downstream users who don’t speak up. Porting most passes is generally pretty quick and easy, but some passes can be tricky.
As mentioned earlier, having two ways to do something can result in confusion upstream when debugging issues. And code cleanup is nice. Also, a fair number of lit tests currently run against both pass managers and removing some RUN lines against the legacy PM can speed up tests.
What won’t be removed anytime soon:
opt -amdgpu-lower-ctor-dtor. But most passes like instcombine should only be invokable via the opt -passes=instcombine syntax so that we don’t have to worry about people testing the wrong pass manager by using the wrong syntax. Even if some passes like instcombine are added to the codegen pipeline in a legacy pass manager, I don’t think the confusion of opt -instcombine vs opt -passes=instcombine is worth the test coverage we may gain by being able to run instcombine with the legacy PM. Perhaps if we find that some passes do regress only under the legacy PM and not the new PM we can revisit this point.