Do we need to use the LLVM Decision Making Process for this proposal?
You just made my day. @rouson and I have funding to do just that. We will promise to monitor bug reports related to flang, reach out to those submitting them and to collaborate on generating simple reproducers as necessary. We absolutely do not wish inundate current active developers with distractions or busy work, just to start building a larger user community. If you want a more formal plan, we can put it together. Does that satisfy your concerns?
That sounds like a support strategy that still depends on the current active developer community to fix your bugs, and I told you how to convince me that they’re ready (edit: and willing) to take that on now.
Ok, I’ll modify my plan slightly.
and provide them with reasonable expectations that current active development may not have solving their issue as their current priority.
Also, @rouson, @ktras , myself, and some others working on our team at LBL, relatively new contributors (i.e. not current active developers), will work to provide fixes for the reported bugs.
I will also start reaching out to current developers reported by
git blame to ask if they are willing to accept feedback from new users and a growing community.
As I said
so we commit to avoiding that.
I’m leaning towards yes. I’d love to hear what other LLVM developers and community leaders think on the topic. It’s not exactly clear (to me), how to initiate it from here.
Let me take a stab at this…
I think what is really being discussed here is the determination of when Flang will have its first release candidate and what the feature set will be. IMHO, the name of the executable and command line arguments are symptomatic of this but alone won’t address them; a lot of the frustration appears to be issues with communication and a lack of information. Apologies, but I think these are the critical topics for us to discuss.
Regarding the first release: To test Flang’s latest status you need to use the main branch of llvm-project. This has the caveat of bringing along the active state of the entire project. Unless you are experience with the infrastructure, using the main branch for general/end-user testing is asking for trouble. If someone is enthusiastic and understands what they are testing from end-to-end, I’m not against such efforts but I think this testing would be most effective in the release candidate and point-release phases. YMMV but I would consider this much more concerning in the grand scheme of things than the name of the executable.
Understanding this means we need to agree on what it explicitly means for Flang to reach its first release. @sscalpone has suggested some criteria. Are there others? I would suggest we focus on this as I think it is more constructive for the community in both the short and long term. To kick this idea off, I’d like to propose we start a “status notes” document at the top-level of the Flang repo that highlights a target feature set and any other criteria for the first release candidate. We can then track these details aligned with the LLVM release cycle. If we can keep this on target for each release in the LLVM cycle it will help communicate to both developers and potential users a more concrete view of our efforts. Hopefully, this could also addresses a portion of @banach-space’s concerns re: documentation.
The second challenge is testing. The lack of an open, robust test suite for Flang is problematic. I would say more in the impact it has on project-wide communications than testing itself at this point in time. The closed testing infrastructure was done to expedite the testing that was desperately needed to harden and stand Flang up. Unfortunately, the nature of the situation means the test features are not visible to the broad community. The only obvious way I see to address this is to provide one or more alternative paths. E.g., the gfortran piece mentioned by @kiranchandramohan, the NIST test suite – although it has older Fortran, and perhaps Fortran-centric OpenMP tests would all be good starting places). This suite could also capture key features that the applications @everythingfunctional and @rouson mention so key features are more visible to the entire community and can be more easily discussed in a community setting (vs. entire applications that compiler folks are unlikely to focus on). Without an alternative set of tests I think we will continue to struggle with visibility, constructively grow the community, and setting project-wide priorities for the foreseeable future.
Maybe I’ve simply stared at this discussion for too many days in a row but this was where I ended up. Hopefully we can find a constructive set of next steps…
@pat thanks for the guidance and suggestions. To make sure I understand, when you mention placing " a ‘status notes’ document at the top-level of the Flang repo," I’m imagining a
STATUS.md file with checkboxes in the
flang/ subdirectory of llvm-project repository on GitHub. That would be helpful as long as it’s maintained and referenced prominently in
flang/README.md and if the items on the checklist are grouped under release milestones. What I think would be more maintainable and provide a much richer experience would be a GitHub project board similar to Semantic Tests for Parallel Features project board. In keeping with the theme of your message, the name of the new project could be “First Release Candidate.” The project board could group GitHub issues into columns with headings such as “To Do,” “Under Development,” “In Code Review,” and “On LLVM main.” This provides a lot more functionality than just a status list. For example, a GitHub project can be set up such that
- Every issue tagged with the project name automatically appears in the “To Do” column on the project board,
- Every issue assigned to a developer automatically moves to the “Under Development” column on the project board,
- Commit messages that contain text like “fixes #42” can automatically trigger the closing of the corresponding issue when the commit is pushed to the main branch, and
- Every issue closed moves to the “On LLVM main” column on the project board.
Moreover, developer discussions can be tracked as comments on the issues. Seeing such discussions in the open is really important. The lack of visibility into what’s happening has caused some in the Fortran community to express pessimism publicly about seeing anything useful from LLVM Flang anytime soon. See this recent comment in the Fortran-lang Discourse. Most Fortran programmers don’t fully appreciate the monumental amount of work that has happened and continues to happen in developing LLVM Flang.
The approach that I’m suggesting invites a broader group of people into the discussion. Users might offer comments on the relative importance of certain optimizations or features that can help prioritize development. Some users might even roll up their sleeves and contribute code. On the OpenCoarrays project, we’ve seen this happen several times over the years when we received unsolicited bug fixes and new features from people we didn’t know prior to their contributions.
Lastly, the project board gives a rough sense of the percent completion based on the number of issues in each of the various columns. It’s not perfect because the list is likely to grow as new work items are discovered and not every issue requires the same amount of development time, but it’s still useful information for the potential user community in an open-source project. I hope these ideas are helpful.
As @kiranchandramohan mentioned earlier, there is such a board but it was started recently and it is very incomplete. https://github.com/orgs/llvm/projects/12/views/1
The LLVM release 16.0.0 is just around the corner. The next release, 17.0.0 will be in 6 month.
Do you want to ship flang or flang-new with LLVM 17?
IMO this is completely orthogonal to what the name and usability of the executable is during development.
My point is that at present, there’s more to it than that. If you are interested in joining development of Flang, you’d think that (and in my experience with other projects) it would be straightforward to try out the latest version; i.e. you clone the official repo, follow the build instructions, and get something potentially usable by a non-developer (i.e. you don’t need to know special names and hidden options). I don’t mean it won’t have missing features or bugs, but it shouldn’t present itself differently just because it’s not “done”.
Yes, and some manual testing from potential new contributors and users can assist in that endeavor. If you’re intentionally excluding them from early testing and feedback, you’re completely missing the bugs and features they might care about (or at least you can’t be sure).
Even with an alternative set of tests, if you have barriers to entry for potential new contributors or users (i.e. if you are not in “the club” and know “the secret handshake”), you will continue to struggle with these things.
Is there any plans to have a
flang executable at all included with the 16 release? What are the chances of having one for the 17 release?
My main point has always been that these artificial barriers to early use are exclusionary, are preventing you from getting useful early feedback, and are turning potential contributors away. The sooner the official/upstream repository can stop this the better (IMO).
You can download flang packages for the current rc:
It is unlikely that 16 ships with flang instead of flang-new. It is up to you which name flang will have in 17.
As the FreeBSD port maintainer for LLVM I’m giving serious consideration to applying ⚙ D143592 [flang][driver] Rename `flang-new -flang-experimental-exec` to `flang` to our release. The other realistic option is to disable flang in the default config which means no packages.
@rouson: My goal was to try and find a simple way to reach the community so there is a quick way to understand what is happening and where things stand in terms of a release candidate. Nothing less, nothing more. I’m hesitant to take on anything more complex until we can find a simple way to communicate these aspects so we can come to agreement on what the feature set for the first RC will be and do the work to bring it to reality.
Classic Flang might be a better choice for distribution until LLVM flang is further along.
@sscalpone Classic Flang has been completely useless across every project on which I’ve worked over the past ~5 years and that’s a long list of projects, some open-source and some not. The stagnation of Classic Flang is understandable because presumably it will be displaced by LLVM Flang eventually. Presenting a stagnant compiler that doesn’t support large parts of Fortran 2008 and 2018 to the community is not a good look for Fortran – particularly when there are already four compilers that are either 2018 compliant or or at least support the most significant and widely used parts of 2018. What would be the point of building interest in LLVM Flang by presenting the predecessor that led to a complete rewrite?
@pat the goal of the automations that I described are to save time. Teams I’ve led have used the described approach across several projects for years and I don’t recall a single team member suggesting that these tools were costing time. But the openness of the described approach is equally important. One of the main points of having GitHub issues is to foster discussion – preferably open discussion when the project is an open-source project. A lack of openness about the process is one cause of the impressions expressed in the Discourse thread that I cited earlier. A spreadsheet or checklist doesn’t open the dialogue and lacks a lot of functionality that would make life simpler.
@clementval thanks for the link. That project board is a great start. A board with the level of detail that is currently in spreadsheets would be really helpful – partly because it makes it more likely that users who visit more than once will see change over time. Also, if the cards listed on that project are converted to issues, then the issue comment threads can capture dialogue, including comments from potential users or potential contributors.
Stagnant aside, LLVM Flang doesn’t support large parts of Fortran 2008 and 2018 yet.
Classic Flang may not meet the needs of @rouson, however, it has a healthy development community, several commercial instantiations, and is available for FreeBSD.
How is this not deliberately misleading your future users? People will start using
flang (as far as they know), and then when behavior drastically changes be completely confused. This is (IMO) the worst of both worlds. You’ll start getting bug reports and feature requests based on experiences with the thing you’re not actually working on. If the main argument against this renaming is not to inundate the developers with superfluous feedback, why on earth would you make this suggestion, which practically guarantees it?
I don’t think it’s misleading at all! I expect the transition from Classic Flang to LLVM Flang to be a fairly transparent and pleasant experience.
Classic Flang has been out there for a while. Both communities have thus far been very adept at redirecting users to the proper channels.
It’s really up to FreeBSD, not me, what they choose to do.