[RFC] Python 2 / Python 3 status, final step(s)

Hi Folks,

Now that LLVM 11.0.0 has been released, it's time to prepare for the final step
envisionned in the previous RFC named *[RFC] Python 2 / Python 3 status* [0],
ie. requiring Python3.6 for LLVM 12.0.0, to be released in 2021.
At least Fedora already only ships Python3 and we didn't have much bugs reported
wrt. Python compatibility for the LLVM toolchain.

Indeed, all Python scripts should now be at least compatible with both python2 (py2)
and python3 (py3), some of them already are already py3 only.

The build system still depends on Python in a few place, but it explicitly
mentions Python3_EXECUTABLE, and the main dependency (llvm-build) is currently
being removed in D89142.

The shebangs have already been harmonized in D83857: some mention /usr/bin/env
python, some mention /usr/bin/env python3, and none mention python2 anymore. It
would be great to have all script use the same shebang, PEP394 [1] makes it
(relatively) clear that in our case, explicitly requiring python3 is the way to
go.

So basically, next step would be to update the documentation, remove the Python2
fallback in root CMakeLists.txt and update shebangs.

Anyone seeing an issue with that approach? I'm still a bit worried about
buildbots still running on py2 (?)

Serge

[0] http://lists.llvm.org/pipermail/llvm-dev/2020-January/138730.html
[1] https://www.python.org/dev/peps/pep-0394/

Yay, thanks for persevering with this migration!

thumbs up

Re: out of date buildbots: maybe stage the shebang changes in a time/way that’s easy to rollback (or even intended to rollback after a small time window (few hours) off-peak) to flush out the buildbots. Then after that rollout, go over the buildbot failures and follow-up with buildbot owners to get them cleaned up before a more intended-to-stick rollout is done.

Sounds good. In LLDB we have a slightly different timeline for the test suite (outlined in http://lists.llvm.org/pipermail/lldb-dev/2020-August/016388.html) but everything else aligns with the plan here. I already removed the fallback to Python 2 in preparation for the 12.0 release.

Hi Folks,

unless someone has strong arguments against it, the transition to Python 3 will happen this week, on Thursday morning, european time.
The associated review is https://reviews.llvm.org/D93097, feel free to drop in.
The impact should be low, Python3 is already picked by CMake as default, but a Python 2 fallback still exists, and it will be removed.

If you’re a bot owner please make sure Python3 is available on your system ! Now, is the time :slight_smile:

Can we delay this? I maintain two Windows buildbots, and I’m about to go on holiday break, and I’d really rather not scramble to get them up to date:
http://lab.llvm.org:8011/#/builders/clang-x64-windows-msvc

http://lab.llvm.org:8011/#/builders/sanitizer-windows

Looking at them now, it seems you broke them five hours ago.

There is risk involved in simply installing Python 3, since it might necessitate upgrading the buildbot worker version, which I have successfully avoided doing for almost five years now.

This has not been a good month for Windows bots. As a long suffering maintainer, I’m asking for a reprieve.

Nevermind. I went ahead and took the risk of installing Python 3 and putting it on PATH, and the buildbot service code seems to be working as is. Both bots have made it past cmake, so they’re probably on track to recover.

Thanks Reid for taking this risky step. As someone who just lost one day upgrading/downgrading/messing up its laptop, I fully appreciate the risk you took, and I’m glad everything looks fine.

The two slaves I had issues whith were:

http://lab.llvm.org:8011/#/builders/127 (now green)
http://lab.llvm.org:8011/#/workers/14 (now green)

Thanks a bunch!

As a long suffering maintainer

And extra thanks for the maintenance task too o/

Hi folks,

  • easy versioning/diffing/reverting of your changes
  • you can run two different images in production and staging
  • Easy to copy-and-past to run a different config
  • You can test your exact changes locally, before deploying them. This gives you much faster iteration times.
    Downsides:
  • Need to get familiar with Docker
  • Need to workaround infrastructure issues if you have some special usages (e.g. GPU access)
  • Bumpy experience on Windows
    My Dockerfiles are in llvm-zorg, feel free to reuse these.