Response to the Move to Discourse Retrospective

We got 8 responses to the anonymous form, and 9 responses with feedback on the move to Discourse Retrospective thread.

We’d like to summarize at a high level the feedback that we have received, before we go deeper and discuss next steps. The 3 questions we asked with a summary of the responses are listed below:

  1. What went well?
  • Technical transition: Archives transferred cleanly (mostly), minimal downtime.
  • Documentation about getting set up in Discourse was well done.
  • After the transition, people were able to use email just like before.
  1. What did not go well?
  • Communication about the decision to move and how community consensus was determined.
    • This transition did not follow the usual RFC process by the community.
    • Who is supposed to make this decision? Why not the IWG?
  • Short communication about the move.
  • Switching off mailing list mode during the transition.
  • List traffic appears to be down.
  • Criticism was taken as personal attacks and dismissed.
  • Categories should have been discussed beforehand.
  • Technical concerns:
    • There is no way to respond via email to any post that was transitioned from the mailing list until someone on Discourse replies to it.
    • Discourse emails go directly to spam for those with GMail accounts and sends mail only in HTML format
    • Not all of the archives were transitioned successfully to Discourse.
  1. What can be improved?
  • The decision-making process needs some serious improvement.
    • The community needs a transparent way to judge community consensus before any further infrastructure projects are undertaken.
  • Communication about such a crucial decision and infrastructure change must be improved.
    • There should have been communication between the proposal to move and the actual plan to move.
    • Give early communication of the roadmap and regular updates on the progress.
  • Make the foundation’s decisions more transparent.
  • The LLVM Foundation should not put the business entity’s needs above those of code owners within the community on community infrastructure topics.

What happened?

In June 2021, Tom Stellard posted on the mailing lists about the proposed migration plan to Discourse. In this post, he asked folks to engage with the Infrastructure working group to discuss their workflow and concerns over the migration. From the discussion on the mailing list, the feedback on the proposed migration plan was a mixture of concern and questions, some against the plan, and some approving of the plan. There were approximately 6 individuals that expressed that they did support the move to Discourse, with some having a stronger standpoint than others. There were 6 individuals who were supporting the move to Discourse by answering questions and correcting assumptions, or just commenting they support the plan. When the mailing lists were moved to Discourse, we had 4,856 subscribers to llvm-dev and 2,827 to cfe-dev. This is very low community engagement in a public discussion about the migration. Additionally, there was minimal engagement with the Infrastructure Working Group during subsequent meetings.

How does one interpret community consensus when the community is mostly silent on the topic? We feel that silence can be interpreted in two main ways - that many were in favor of the change or had no opinion at all. It could also signal a bigger problem with communication as it stands through the mailing lists - How many actually read and respond on our mailing lists?

The LLVM Foundation also received many private messages from individuals that expressed support over the migration plan. Some did not want to voice them publicly because of fear of the response they would receive. Discussions were also held at the Community.o summit, LLVM Dev Mtg round tables and workshops that supported the move to Discourse. Taking in all of these data points, we concluded that the majority of the community was not opposed to moving to Discourse.

The LLVM Foundation holds the legal and financial responsibility for most of the project infrastructure which makes the decision slightly different than the traditional RFC process. Legal responsibilities include - GDPR removal requests, account removal, removal of inappropriate or confidential information that was accidentally posted by community members, and more. From a technical perspective, moving to Discourse gives us the tools to support these sorts of things much easier than mailman lists.

Some community members have spent 18+ years supporting and maintaining the Mailman lists and have a wealth of expertise in the specific problems and concerns that have come up over the years with the Mailman lists. These administrators have a huge investment in volunteer hours and expressed a strong desire to investigate alternative options to lower the workload.

Most importantly, the LLVM Foundation’s mission is to support the LLVM community through infrastructure that supports a healthy, growing, and thriving community. While we can poll the current community and get their opinions, we feel that the LLVM Foundation should also be the voice of newcomers who have yet to join the project. We are always trying to lower the barriers to entry and bring more people into the community, and Discourse is widely used in other communities, including Rust and Swift.

The LLVM Foundation aims to make a decision that the majority of the community is in favor of, takes into consideration future growth of the project, and also supports legal and fiscal responsibilities on behalf of the community. The IWG was created to help gather information for the LLVM Foundation and increase community involvement. Ultimately, the final decision has to be made by the LLVM Foundation.

What could be improved?

Communication and decision making

With all this said, we realize that there was a communication failure from us in that too much time had lapsed without an update and we did not communicate clearly how we had made the decision. Therefore we will aim to do the following for future infrastructure changes:

  • Investigate additional tools/methods to gather feedback from the community about infrastructure changes.
  • Document the feedback in a single location that is accessible to all.
  • Communicate clearly who is ultimately making the final decision and why.
  • Communicate to the community more frequently about the status of an infrastructure proposal.

Short communication about the move

The second most frequent feedback was that the time between when the move was announced and the actual move was too short. The move was announced on January 7th in a blog post with the actual move to occur on Feb 1st (25 days later). However, the blog post highly recommended that LLVM Developers’ transition over by January 10th (3 days later).

The suggestion that people move over as quickly as possible was a last minute addition to the blog post as we thought it would reduce developer downtime. While it was a good faith effort to reduce downtime for users, it caused unnecessary stress to the situation and ultimately did not impact overall downtime for users. We regret making this recommendation.

The LLVM Foundation often receives feedback from community members and sponsors that we move too slowly on our initiatives and community wide efforts. This effort was another item that we heard was taking too long, was becoming an even bigger barrier for newcomers (with the requirement to email to subscribe to the lists), and we needed to move forward. However, we should have communicated more directly (not just through our minutes and the IWG) and frequently about the status of the decision. If we had communicated more effectively, the actual time frame of 25 days should have been sufficient from a technical perspective.

Switching off mailing list mode during the transition

While we can understand the frustration over this development, it was not something that we anticipated, nor something we could do anything about. We understand that communication about the change was not sent to the mailing lists which was something that we should have done in addition to our other steps to notify impacted individuals.

The move has caused list traffic and engagement to decrease

There have been some statements that list traffic is down. When we used Mailman, we did not have any tools to measure the level of engagement on the lists. Discourse provides us with a wealth of data about users activity on the lists. Since we imported our archives, we also have some data from the past to compare it to.

Discourse considered a new post to a category, to be a topic. This is equivalent to the starting email to the mailing lists. Subsequent responses to topics are called posts which is the equivalent to replies to the initial email on the mailing list.

This graph shows the number of monthly topics from October 1, 2019 to April 30, 2022.

As you can see from the graph, topics were on the downward trend on the mailing lists for several months (since March 2021). We can see a huge small increase towards November (possibly related to the LLVM Developers’ Meeting) and a big jump in January which we could possibly attribute to all the communications about the move to Discourse. Note that these are speculations about the abnormal data, we don’t know for sure. However, you can see that since the move on Feb 3rd, the traffic is pretty similar to the months before the move. There is no drastic downturn in topics.

This graph shows the number of posts per month that were created from October 2019 to April 30, 2022.

The data jumps around a lot more, but the last 2 months of posts to Discourse seem on par with what was occurring on the mailing lists.

Given this information, we do not think that list traffic is down because of the move.

Criticism was taken as personal attacks and dismissed

How the discussion proceeded after the mailing list mode was turned off, was very disappointing. Before that, there could have been questions asked about the decision and process. We do believe that there were many attacks thrown at the LLVM Foundation out of anger about the decision that was made. The initial email was filled with threats and demands and not an approach to work together. This set the tone for the following responses which included attacks about how the LLVM Foundation board operates.

Categories should have been discussed beforehand

We discussed categories within the IWG. We should have posted this as part of our status communication about the move to Discourse. One benefit of Discourse, is that nothing is set in stone, we can refine the categories as more people get used to Discourse and have first hand experience with what works or does not work for them with the current setup.

Technical concerns

We attempted several test archive migrations and had several volunteers attempting to verify content was accurate. We identified several issues, but unfortunately there are still a few outstanding issues. We are actively working with the Discourse support to identify these situations where the archives did not import properly. No data has been lost and an administrator can modify posts. We still would like to find and fix these issues proactively, so it is an open issue.

To help with these known issues, we are putting into each category description a link to the old mailman archives.

Regarding the inability to respond to posts created before the migration, while this is annoying, you can always reply through the web interface. Lastly, we do not know about widespread issues with Discourse emails going to spam folders.

Make the foundation’s decisions more transparent

The LLVM Foundation has published meeting minutes on our website. The meeting minutes are posted 1 month after the meeting has occurred as they are voted on in the subsequent meeting. To increase visibility of our meeting minutes, we will post them on Discourse under the Community - LLVM Foundation category. If anyone has questions, they can respond there or contact the board privately.

The LLVM Foundation should not put it’s business needs above the community

The LLVM Foundation is a 501(c)(3) nonprofit organization that is the legal entity that is behind most of the infrastructure services. There was an exchange regarding the Terms of Service for Discourse and during the discussion, we said that “we have a right to protect ourselves” in regards to what was in the Terms of Service. We only had 2 individuals express a problem with the Discourse Terms of Service. The LLVM Foundation feels that by establishing clear terms regarding participation, conduct, and privacy it is beneficial to the community as a whole. In addition, protecting the LLVM Foundation is a benefit to the community so we can continue to provide the programs and services we do. Therefore, we do not believe we have put business needs above community needs.

Last Comments

We also want to take the time to point out that we got a couple very inappropriate responses through the anonymous form. While we would typically ignore these sorts of comments, one in particular was something that we do want to call out. The comment was as follows:

“The board imposes the Code of Conduct, imposes change to discourse, using the foundation funds to sponsor the outreachy program that doesn’t seem to be useful at all (just a waste of money in a couple of girls that don’t become contributors),”

These sorts of comments are in line with what many of us involved in the Women in Compilers and Tools Community.o group receive frequently. These sorts of comments and attitudes are not welcome in our community and we highly encourage everyone in our community to speak up when you hear things that are sexist, racist, or otherwise against the core values of the LLVM community.

We hope that this retrospective was useful to the LLVM Community. If there are any questions or comments, please feel free to contact the board directory (board@llvm.org) or respond on Discourse.

Thanks,

The LLVM Foundation Board of Directors

9 Likes

Thank you for the report. I know there are differing opinions on this - but for me the move to Discourse have been a good move:

  • I find the discussions more structured and easier to follow code examples
  • The ability to view all discussions over all project under LLVM is way better than having to subscribe to many different mailing-lists.
  • I haven’t seen any spam on the forum
  • Searching the archives is much easier and have found me many old threads that I probably would have problem finding before since I haven’t been subscribed for that long.
  • Notifications from the iOS app is great

Some things that I think can improve:

  • Seems like some stuff is still missing categories or tags. Actually the whole category / tag distinction is not easy to understand. Most of that is just how Discourse works though.
  • Better integration and linking between discourse and phabricator would be nice - like being able to say DXXYYZZ and it would link to the right diff.

In general - for me that is pretty new in the community the move has been a beneficial one with better access to the discussions and all things that makes me spend less time with Outlook the better :wink:

Thanks,
Tobias

7 Likes

Thanks for the very comprehensive report!
The “Communication and decision making” steps you’re aiming to improve in the future look good to me!

This is extremely shocking to me, I’m sorry you (and others at the foundation) have to go through this. Thanks for for sharing: I don’t think people (like me) are aware of this and really appreciate what you’re going through and the mental load it has to be to continue the effort to move everything forward: thanks for that!

6 Likes

Thank you to Tanya and the rest of the LLVM Foundation Board for writing up this retrospective.

It’s definitely important to highlight these restrictions, though my observation is that LLVM RFCs or the (not yet used much) contentious decision making process are compatible with having various constraints as their input. After all, we often have various concrete technical constraints that must be met and account for these when producing RFCs. Couldn’t the legal responsibilities identified by the LLVM Foundation be similarly listed and addressed while using the RFC or contentious decision making process?

Anyway, independent of that particular point I think the suggested actions under “Communication and decision making” should address the majority of the concerns I’ve seen.

2 Likes

Thanks for posting this, especially the graphs about the mail/discourse traffic. I’m looking forward to seeing the changes to communication & decision-making in practice.

2 Likes