The "you" in eucatastrophe - the story of building SciPy with Flang

Subprojects Flang
1

Dear Flang community

I’ve been following the development of Flang with bated breath since way before the upstreaming into LLVM happened, spurred initially by the desire / need to have a freely available Fortran compiler on windows for conda-forge, which is a cross-language packaging ecosystem (historical focus having been Python) that aims to serve all major platforms and architectures.

In particular, one of the key packages in python – SciPy – requires a Fortran compiler,
which has long been a source of problems (and the cause for my interest in Flang initially). This was exacerbated by recent changes in Python 3.12, which made it impossible to keep using the ducktape-and-string solution that had been in place before, and so we were in quite urgent need of an alternative – and indeed I had been trying to maneuver Flang into our infrastructure for almost a year already.

Though it turned out to be quite a non-trivial operation to get everything built & running with all components involved (not just our infra, but also the package itself and its build system), the final moment of this story was so unexpectedly positive that I was reminded of Tolkien’s term “eucatastrophe”.

After we had managed to get through all the build & installation woes, I was expecting the SciPy test suite to blow up with failures, hangs or crashes, and things… just passed 100% on the first time we got to the test suite… :exploding_head:

There are some factors that help explain this (e.g. the Fortran code in SciPy is quite old, we don’t use OpenMP[1], etc.), but the main reason was just that Flang was already at a high enough quality to do so, and so I want to take the opportunity to thank you all for your work on Flang, which managed to get us out of quite a pickle – I hope you can forgive the terrible pun in the title. :wink:

If you’re curious about the rest of this story, I wrote it up in blog post. It tries to paint a big picture and it’s very possible that I made mistakes in summarizing so much history – if you spot something that you think is wrong, please let me know.

  1. intentionally, it’s a long story ↩︎

15 Likes
2

My understanding of SciPy is that it’s fairly vanilla F’77 code with little I/O, but still, it’s pretty hard to believe that you didn’t fail a single test. If possible, please modify a local copy of the SciPy source to introduce an intentional syntax error or bug to the code, and see whether it still passes all of those tests.

1 Like
3

This is great, thanks for sharing and for contributing! You’ve spent quite some effort digging into some quite intricate issues - that’s been greatly appreciated and hugely beneficial to Flang :pray:t2: Fantastic to see this paying off :slight_smile:

-Andrzej

1 Like
4

The windows builds based on this have been public for over a month and downloaded ~100k times – so far no-one has screamed. :sweat_smile:

I actually trust the test suite in this regard, because it’s completely Fortran-agnostic. It will run a test from the Python side, and if the Fortran code (through its Python wrapper) does not perform correctly, the test would fail. That, and syntax errors would lead to compilation errors even before we get to the installation, much less the test suite.

1 Like
5

Ok. I still don’t fully believe this result, though. But thank you for the nice things that you wrote about this compiler in the blog post!

2 Likes
6

@h-vetinari If it’s not too much work, could you let me know roughly how many lines of Fortran code are compiled into SciPy? Thanks.

7

According to github-generated stats (see bottom of right-hand band of main repo), it’s about 17% of all lines of code in SciPy. By git ls-files | xargs cat | wc -l, current scipy main has about 1’172’028 lines across all files, so that would put it around 200k lines.

Running a more specific command yields

$ git ls-files '**/*.f' | xargs cat | wc -l
147793

That’s useful as a lower bound (perhaps github recognises some more files for its count; also, submodules might not be counted correctly or at all).

1 Like
8

Thank you for sharing the story, this is really motivating!

1 Like
9

I should add that MSYS2 was the first one to build scipy with Flang 16.

Of course with some workarounds because meson still doesn’t support LLVM-Flang.

1 Like
10

It’s now Impossible to build scipy with Flang-18.

Because someone decided to throw errors with unrecognized flags.

11

Indeed. Though warning on unrecognized options is a good thing long term, there’s still a bunch of relevant options that flang does not support. I have a very WIP meson PR that works around the unknown flags (by not emitting them), but the bigger problem is that flang 18 provides no way anymore to link Fortran_main.lib in the scenario that SciPy needs.

The code around this has been changing a lot, and I’ve been wanting to try the main branch (to help ensure that at least flang 19 can build SciPy again), but that runs into another unrelated issue currently.

12

They are errors not warnings.

13

We should ensure that llvm-19 works fine. llvm-19 branches off sometime next month.

Ideally if we can set up a CI that always tests main development branch with scipy on Windows then we can guard against regressions.

1 Like