I think what is really being discussed here is the determination of when Flang will have its first release candidate and what the feature set will be. IMHO, the name of the executable and command line arguments are symptomatic of this but alone won’t address them; a lot of the frustration appears to be issues with communication and a lack of information. Apologies, but I think these are the critical topics for us to discuss.
Regarding the first release: To test Flang’s latest status you need to use the main branch of llvm-project. This has the caveat of bringing along the active state of the entire project. Unless you are experience with the infrastructure, using the main branch for general/end-user testing is asking for trouble. If someone is enthusiastic and understands what they are testing from end-to-end, I’m not against such efforts but I think this testing would be most effective in the release candidate and point-release phases. YMMV but I would consider this much more concerning in the grand scheme of things than the name of the executable.
Understanding this means we need to agree on what it explicitly means for Flang to reach its first release. @sscalpone has suggested some criteria. Are there others? I would suggest we focus on this as I think it is more constructive for the community in both the short and long term. To kick this idea off, I’d like to propose we start a “status notes” document at the top-level of the Flang repo that highlights a target feature set and any other criteria for the first release candidate. We can then track these details aligned with the LLVM release cycle. If we can keep this on target for each release in the LLVM cycle it will help communicate to both developers and potential users a more concrete view of our efforts. Hopefully, this could also addresses a portion of @banach-space’s concerns re: documentation.
The second challenge is testing. The lack of an open, robust test suite for Flang is problematic. I would say more in the impact it has on project-wide communications than testing itself at this point in time. The closed testing infrastructure was done to expedite the testing that was desperately needed to harden and stand Flang up. Unfortunately, the nature of the situation means the test features are not visible to the broad community. The only obvious way I see to address this is to provide one or more alternative paths. E.g., the gfortran piece mentioned by @kiranchandramohan, the NIST test suite – although it has older Fortran, and perhaps Fortran-centric OpenMP tests would all be good starting places). This suite could also capture key features that the applications @everythingfunctional and @rouson mention so key features are more visible to the entire community and can be more easily discussed in a community setting (vs. entire applications that compiler folks are unlikely to focus on). Without an alternative set of tests I think we will continue to struggle with visibility, constructively grow the community, and setting project-wide priorities for the foreseeable future.
Maybe I’ve simply stared at this discussion for too many days in a row but this was where I ended up. Hopefully we can find a constructive set of next steps…
@pat thanks for the guidance and suggestions. To make sure I understand, when you mention placing " a ‘status notes’ document at the top-level of the Flang repo," I’m imagining a STATUS.md file with checkboxes in the flang/ subdirectory of llvm-project repository on GitHub. That would be helpful as long as it’s maintained and referenced prominently in flang/README.md and if the items on the checklist are grouped under release milestones. What I think would be more maintainable and provide a much richer experience would be a GitHub project board similar to Semantic Tests for Parallel Features project board. In keeping with the theme of your message, the name of the new project could be “First Release Candidate.” The project board could group GitHub issues into columns with headings such as “To Do,” “Under Development,” “In Code Review,” and “On LLVM main.” This provides a lot more functionality than just a status list. For example, a GitHub project can be set up such that
Every issue tagged with the project name automatically appears in the “To Do” column on the project board,
Every issue assigned to a developer automatically moves to the “Under Development” column on the project board,
Commit messages that contain text like “fixes #42” can automatically trigger the closing of the corresponding issue when the commit is pushed to the main branch, and
Every issue closed moves to the “On LLVM main” column on the project board.
Moreover, developer discussions can be tracked as comments on the issues. Seeing such discussions in the open is really important. The lack of visibility into what’s happening has caused some in the Fortran community to express pessimism publicly about seeing anything useful from LLVM Flang anytime soon. See this recent comment in the Fortran-lang Discourse. Most Fortran programmers don’t fully appreciate the monumental amount of work that has happened and continues to happen in developing LLVM Flang.
The approach that I’m suggesting invites a broader group of people into the discussion. Users might offer comments on the relative importance of certain optimizations or features that can help prioritize development. Some users might even roll up their sleeves and contribute code. On the OpenCoarrays project, we’ve seen this happen several times over the years when we received unsolicited bug fixes and new features from people we didn’t know prior to their contributions.
Lastly, the project board gives a rough sense of the percent completion based on the number of issues in each of the various columns. It’s not perfect because the list is likely to grow as new work items are discovered and not every issue requires the same amount of development time, but it’s still useful information for the potential user community in an open-source project. I hope these ideas are helpful.
IMO this is completely orthogonal to what the name and usability of the executable is during development.
My point is that at present, there’s more to it than that. If you are interested in joining development of Flang, you’d think that (and in my experience with other projects) it would be straightforward to try out the latest version; i.e. you clone the official repo, follow the build instructions, and get something potentially usable by a non-developer (i.e. you don’t need to know special names and hidden options). I don’t mean it won’t have missing features or bugs, but it shouldn’t present itself differently just because it’s not “done”.
Yes, and some manual testing from potential new contributors and users can assist in that endeavor. If you’re intentionally excluding them from early testing and feedback, you’re completely missing the bugs and features they might care about (or at least you can’t be sure).
Even with an alternative set of tests, if you have barriers to entry for potential new contributors or users (i.e. if you are not in “the club” and know “the secret handshake”), you will continue to struggle with these things.
Is there any plans to have a flang executable at all included with the 16 release? What are the chances of having one for the 17 release?
My main point has always been that these artificial barriers to early use are exclusionary, are preventing you from getting useful early feedback, and are turning potential contributors away. The sooner the official/upstream repository can stop this the better (IMO).
@rouson: My goal was to try and find a simple way to reach the community so there is a quick way to understand what is happening and where things stand in terms of a release candidate. Nothing less, nothing more. I’m hesitant to take on anything more complex until we can find a simple way to communicate these aspects so we can come to agreement on what the feature set for the first RC will be and do the work to bring it to reality.
@sscalpone Classic Flang has been completely useless across every project on which I’ve worked over the past ~5 years and that’s a long list of projects, some open-source and some not. The stagnation of Classic Flang is understandable because presumably it will be displaced by LLVM Flang eventually. Presenting a stagnant compiler that doesn’t support large parts of Fortran 2008 and 2018 to the community is not a good look for Fortran – particularly when there are already four compilers that are either 2018 compliant or or at least support the most significant and widely used parts of 2018. What would be the point of building interest in LLVM Flang by presenting the predecessor that led to a complete rewrite?
@pat the goal of the automations that I described are to save time. Teams I’ve led have used the described approach across several projects for years and I don’t recall a single team member suggesting that these tools were costing time. But the openness of the described approach is equally important. One of the main points of having GitHub issues is to foster discussion – preferably open discussion when the project is an open-source project. A lack of openness about the process is one cause of the impressions expressed in the Discourse thread that I cited earlier. A spreadsheet or checklist doesn’t open the dialogue and lacks a lot of functionality that would make life simpler.
@clementval thanks for the link. That project board is a great start. A board with the level of detail that is currently in spreadsheets would be really helpful – partly because it makes it more likely that users who visit more than once will see change over time. Also, if the cards listed on that project are converted to issues, then the issue comment threads can capture dialogue, including comments from potential users or potential contributors.
How is this not deliberately misleading your future users? People will start using flang (as far as they know), and then when behavior drastically changes be completely confused. This is (IMO) the worst of both worlds. You’ll start getting bug reports and feature requests based on experiences with the thing you’re not actually working on. If the main argument against this renaming is not to inundate the developers with superfluous feedback, why on earth would you make this suggestion, which practically guarantees it?
We should listen to and support our community members who provide and maintain binary packages for LLVM Flang - ultimately they make sure that LLVM Flang reaches end users, which is vital for the success of the project. We should really avoid them having to maintain any downstream patches:
Also, can we please avoid including Classic Flang in this discussion? That’s a completely separate project.
This thread is an opportunity for the LLVM Flang community to make the life of binary package managers (and many other people) easier.