The "you" in eucatastrophe - the story of building SciPy with Flang

Dear Flang community

I’ve been following the development of Flang with bated breath since way before the upstreaming into LLVM happened, spurred initially by the desire / need to have a freely available Fortran compiler on windows for conda-forge, which is a cross-language packaging ecosystem (historical focus having been Python) that aims to serve all major platforms and architectures.

In particular, one of the key packages in python – SciPy – requires a Fortran compiler,
which has long been a source of problems (and the cause for my interest in Flang initially). This was exacerbated by recent changes in Python 3.12, which made it impossible to keep using the ducktape-and-string solution that had been in place before, and so we were in quite urgent need of an alternative – and indeed I had been trying to maneuver Flang into our infrastructure for almost a year already.

Though it turned out to be quite a non-trivial operation to get everything built & running with all components involved (not just our infra, but also the package itself and its build system), the final moment of this story was so unexpectedly positive that I was reminded of Tolkien’s term “eucatastrophe”.

After we had managed to get through all the build & installation woes, I was expecting the SciPy test suite to blow up with failures, hangs or crashes, and things… just passed 100% on the first time we got to the test suite… :exploding_head:

There are some factors that help explain this (e.g. the Fortran code in SciPy is quite old, we don’t use OpenMP[1], etc.), but the main reason was just that Flang was already at a high enough quality to do so, and so I want to take the opportunity to thank you all for your work on Flang, which managed to get us out of quite a pickle – I hope you can forgive the terrible pun in the title. :wink:

If you’re curious about the rest of this story, I wrote it up in blog post. It tries to paint a big picture and it’s very possible that I made mistakes in summarizing so much history – if you spot something that you think is wrong, please let me know.

  1. intentionally, it’s a long story ↩︎


My understanding of SciPy is that it’s fairly vanilla F’77 code with little I/O, but still, it’s pretty hard to believe that you didn’t fail a single test. If possible, please modify a local copy of the SciPy source to introduce an intentional syntax error or bug to the code, and see whether it still passes all of those tests.

1 Like

This is great, thanks for sharing and for contributing! You’ve spent quite some effort digging into some quite intricate issues - that’s been greatly appreciated and hugely beneficial to Flang :pray:t2: Fantastic to see this paying off :slight_smile:


1 Like

The windows builds based on this have been public for over a month and downloaded ~100k times – so far no-one has screamed. :sweat_smile:

I actually trust the test suite in this regard, because it’s completely Fortran-agnostic. It will run a test from the Python side, and if the Fortran code (through its Python wrapper) does not perform correctly, the test would fail. That, and syntax errors would lead to compilation errors even before we get to the installation, much less the test suite.

1 Like

Ok. I still don’t fully believe this result, though. But thank you for the nice things that you wrote about this compiler in the blog post!


@h-vetinari If it’s not too much work, could you let me know roughly how many lines of Fortran code are compiled into SciPy? Thanks.

According to github-generated stats (see bottom of right-hand band of main repo), it’s about 17% of all lines of code in SciPy. By git ls-files | xargs cat | wc -l, current scipy main has about 1’172’028 lines across all files, so that would put it around 200k lines.

Running a more specific command yields

$ git ls-files '**/*.f' | xargs cat | wc -l

That’s useful as a lower bound (perhaps github recognises some more files for its count; also, submodules might not be counted correctly or at all).

1 Like

Thank you for sharing the story, this is really motivating!

1 Like

I should add that MSYS2 was the first one to build scipy with Flang 16.

Of course with some workarounds because meson still doesn’t support LLVM-Flang.

1 Like