The LLVM Foundation is asking for feedback about the process or the communication methods we used to move to Discourse.
For reference, the plan was posted to the LLVM Blog on January 7th:
Given this is a retrospective and we would like people to reflect on all aspects of the move. Please make sure to include these 3 things:
What went well?
What did not go well?
What can be improved?
Things to consider in your response:
Communication about the move (ie. blog post, mailing lists, etc)
Timeframe for the move
The actual migration and communication during
Post migration communication
To keep things focused, I ask that you limit your feedback to the actual process and communication versus any technical points about Discourse.
Ideally we would like to keep the discussion here, but if you prefer to post your feedback anonymously (or just not on Discourse), you may use this google form: https://forms.gle/QhdZE5gkPyNx2XqE9
We would like to have all feedback by March 7th. The LLVM Foundation board does not plan to reply to comments posted here. Please know that every comment is being read, and that a collective reply will be posted with any action plan after March 7th. We will aim to exclude any identifying or personal information in any published reports or responses.
Some of this has been posted on a couple of the mailing list threads before the migration, but here are my key comments.
First and foremost, the decision making process that led to moving to Discourse was inadequate at best. The discussion was raised a couple of times on the llvm-dev mailing list prior to the move, yet there was no clear consensus on that thread as far as multiple individuals seemed to be concerned. There wasnāt even a clear majority in favour of the move, with other suggestions regarding things like mailman 3 beign raised but never adderssed on the thread. Addressing these concerns publcily on the mailing lists before any final decision had been made would have helped reduce a significant amount of upset by individual contributors who were opposed to the move. Itās possible that these concerns were addressed directly with the individual, or in Foundation or IWG meetings, but there are many interested parties in these discussions who donāt have the time to commit to such meetings. When a discussion is started somewhere, it needs to be finished there, unless clear signposts have been posted on the relevant thread indicating where the discussion continues.
Announcing the move on the mailing list was good, but it was done at way too short notice for the timeline, meaning it was impossible for individuals to raise further concerns, clarify the process and so on. Contrast this with the recent bugzilla ā Github Issues migration, where the plan was communicated well in advance. In that case, a number of concerns were raised, allowing changes to the plan and so on. The timeline was posted late Friday after the end of work in European timezones and even some US ones, with the start of the timeline the following Monday. Many people wonāt have seen the blog post until the timeline had already started. Thatās before you even consider the fact that it was a holiday weekend in some parts of the world. Additionally, even a week or two would be insufficient gap between announcement and start of timeline for these sort of big changes, because that sort of time period is a common length of time to be on leave. At a minimum, big changes should be announced at least a month in advance, and preferably longer. Yes, there was an overlap period, but the suggestion was to start using Dsicourse straight away, meaning there would have been some people who did so, leaving those who didnāt out-of-the-loop on potentially important discussions.
Mailing list mode: the limitation here should have been discovered beforehand, one way or another. However, accepting the fact that there are always snags in these big moves, when a problem was discovered, and it had to be turned off, the announcement of it being disabled should have been clearly communicated on the old mailing lists at the time it happened, so that people could be aware of the problem. Again, contrast this with how the bugzilla migration went: Anton clearly updated whenever an
ther snag had been hit, delaying the timeline, rather than leaving everyone out-of-the-loop.
There should have been an opportunity before the migration to discuss the different categories. As mentioned elsewhere, there are a number of glaring categories missing, such as Debug Information and Testing that have no clear home. Even now, a week into the final migration, this hasnāt been solved. Without clear categories, some communications may never happen, or interested parties may miss out, when topics are posted elsewhere.
The getting set up documentation that Iām sure I read somewhere was reasonably useful. Such documentation should be prepared again, and ideally reviewed by those with knowledge of new infrastructure, to aid in future migrations.
This feedback thread is a great idea. I hope it isnāt just tokenism though and that the feedback is acted upon in future infrastructure moves!
Sorry, the message is supposed to have several paragraphs sent via a response in email. Apparently, multi-paragraph reply emails donāt work with Discourse?
(Trying again with plaintext, letās see if this fixes the issue.)
Several points of feedback to provide here, in addition to what James Henderson has already said:
Communication about the migration has been, and continues to be, exceptionally poor. To explain the italicized part: I was under the impression from discussions about mailing-list mode that it would be turned on after the month changed. It has not been turned on, nor have any of my attempts to ask questions about plans to turn it on been acknowledged, much less answered.
James Henderson already mentioned the problems with the decision to undertake the move in the first place, but to expand on that: based on what Iāve seen in the IWG minutes, there was no recognition within the iWG of a lack of consensus for the move within the community, nor any discussion of the points brought up in that most recent thread.
List traffic appears to be substantially down since the move compared with before the move, I think all of Discourse sans MLIR (which was most of Discourse pre-move) receives less messages per day than llvm-dev by itself used to. It seems something about the move may have caused people to communicate less, and I think itās worth trying to collect statistics to understand a) if this was the case and b) if so, what may have driven that.
My overall sense of the migration process is that speed of transition has been considered the single most important quality, and communication and consensus-building has been considered the least important quality of the process. To the point that it has been left out altogether in some cases, with only a weak justification that thereās not enough resources to devote to communication.
Some of the design goals of the migration process appear to have been at complete odds with what the community actually desires in its mailing lists. Iād highlight the issue of cross-posting in particularāit appears that disabling any ability to cross-post, both between LLVM mailing lists and external mailing lists such as libc ABI discussions, and within LLVM mailing lists (e.g., cross-posting llvm-dev and cfe-dev), was considered an anti-feature and therefore would have no migration path without actually elucidating why people would do that.
Looking back at the other migration processes, I can give my personal estimates of the quality of IWG and board with respect to other transitions:
ā¢ Relicensing: A+. Honestly, I am surprised and astounded at the lengths you have gone to actually get agreement for relicensing. Itās a superb effort, and I must applaud them for their ability to track down some of the contributors.
ā¢ Move to github: A. The transition took a long time, but I canāt recall any complaints about the process (other than the length of time). And note that the discussion had to navigate a pretty contentious issue: whether to use monorepos or to rely on submodules.
ā¢ Moving bugs to github issues: B. The complaint I have here is that some of the timelines were a bit short, those for associating github ids with Bugzilla email addresses, and also the preview merge (which was hampered by my inability to access the test repo prior to migration). So work could be done on it. But the actual migration process, when it hit the inevitable snags.
ā¢ Moving to discourse: D, or maybe F (if the apparent drop-off in discussion is true, as that would mean that itās failed as a replacement for discussion and any passing grade would be too high).
While Iām happy to see that feedback is being solicited, I am still deeply concerned that the feedback will not actually be addressed. Most of this feedback, after all, should be unsurprising: there is already one thread expressing displeasure with the move to Discourse with lots of unsolicited feedback, and there largely remains no acknowledgement of any of that feedback, instead dismissing most of it as an attack against the board for its poor communication regarding the āmailing list modeā feature.
Iām in general agreement with most of the above.
In addition, there is at least one prominent member of the community who has some kind of issue with the Terms of Service and chooses not to participate on Discourse unless/until that is resolved. With the shutdown of the mailing list, there does not appear to be any way to discuss the specifics in public. I hope Aaron is in direct contact with someone about this.
Finally, I feel compelled to expand on @jcranmerās last remark. Criticism can be taken as directed at the work, or directed at the person(s) doing the work. As technical folks, we might not always be careful about that difference. But I had the good fortune, early in my career, to encounter someone who deliberately handed out red pens with his patches (reviews were on paper back then) and was disappointed if the review packets didnāt come back ācovered in blood.ā I hope the board, IWG, and others receiving these criticisms will take them in the spirit offered. After all, the Code of Conduct requires that people offer criticism constructively, and whether they succeed or fail in this, it equally requires people to take criticism constructively. This is never more important than when the speaker is saying something we donāt like.
Just a quick clarification and to be very clear about the process. The LLVM Foundation board does not plan to reply to comments posted here. Please know that every comment is being read, and that a collective reply will be posted with any action plan after March 7th.
I agree with all the feedback so far, but expanding on thisā¦
During some discussions on IRC and the mailing list, people from the board/IWG have taken the criticism as personal and have responded strongly, condemning the whole response as personal attacks and dismissing all the content with it.
This is not just wrong and offensive (to a group or people everyone knows and who were only trying to communicate), but it piles up on the problems already exposed in this thread (and the list, and IRC).
The IWG move was sudden (Friday night, expiring Monday morning), filled with problems (mailing list mode, sub-category watching, email reply, TOS) and not based on consensus (and after 6 months of silence).
Most importantly, it didnāt adopt the previous agreed method of progress, which was to use specifically the llvm-dev list as consensus forming. Once the move was complete, on a time-based deadline, ignoring all the feedback, the new method of progress was changed to use Discourse, again, without consensus on the previous method.
To me, this looks like the Board and any of its working groups has the ability, or even the right, to change our agreed past consensus to shape its own vision.
For over a decade this community, and especially Chris Lattner, have steered away from strong mandates: benevolent dictators, voting process, steering committees. Again and again that was outlined by Chris and agreed by all that it would stiff the community, alienate developers, reduce contributions.
When the board was founded, fears that a strong mandate came through the back-door (no one knew about the foundation until after it was done) have surfaced and the foundationās quick response was that this was not true: they would serve the community, not push it towards places it was comfortable with.
In that process, the foundationās board and Chris have repeated that the decision making and consensus forming would still happen on the llvm-dev list, and thatās what triggered us to design the process and commit to the docs. However, this move has failed every single point in that document.
If this recent migration is any indication, it only proves Chris and the community were right all along: Strong handed opaque decisions pushed without the communityās consent will fracture it. Each new one breaking a bit more, pushing more and more people away.
I will certainly contribute less in Discourse than the mailing list. Not out of spite, I hope you know Iām not a petty person, but out of the incapacity to follow what goes in here. Itās too confusing for me.
Iāve tried Discord for a few weeks and I cannot cope with that either, so, at the very least, youāve already lost one soldier in this crusade. I hope the promise of new people bears fruit.
Itās a little hard to tell, but it does look like the archives transferred cleanly. Itās too hard to navigate for me to really tell, though.
I had no idea how much nicer LLVM Weekly would look in html, that was very pleasant.
Beyond what people have already saidā¦
Discourse has a really poor UX, as far as Iām concerned. I find it hard to navigate, it uses truly impressive amounts of unnecessary whitespace, I canāt be sure Iām seeing everything I want to see, it has demoted essentially everyone (outside of MLIR, who were already using this instance) to n00b status. (And it calls them ātrust levelsā which is really grating for this former security guy; it has nothing to do with trust.)
My reading of the replies so far is that the decision-making process needs some serious improvement. You might consider this more āmetaā than being directly about the migration, but itās hard to determine whether you consider that topic to be off the table. If it is off the table, it needs to be brought up properly elsewhere so we can fix it before the next big decision/migration/transformation happens.
Moving to some other platform with a distinctly better UI would be an improvement, but I assume that is also off the table, since weāve havenāt even finished this transition.
Almost. There is a glitch in the system which causes it to cut off parts of an email, I suspect it thinks itās a signature. This happened recently with emails send but is also true for archive emails. Or at least the symptoms are the same. Here is an example: Memory barrier problem
Itās too early for me to determine if anything has gone well or not. The only measure that really tells me this is whether people use the Discourse service more or less than the mailman service, and it will take time to get meaningful data. To me, if the volume drops on Discourse, I believe it demonstrates a clear failure of the experiment and if the volume increases it may signal a benefit (if itās more questions but no answers, itās a failure; if itās more questions and theyāre getting answers, then itās a benefit).
For the folks who like Discourse and dislike email, Iām guessing this went well for them.
What did not go well?
The process for this transition.
Itās worth noting that my complaints are directed towards the board as I still donāt know honestly how this decision got made. The only identifiable IWG member Iāve seen say anything on the topic said they quit the IWG over this ([llvm-dev] LLVM Discourse migration: goals justify means?), but every impression I have is that this was ultimately a board-driven decision.
This transition did not follow the usual RFC process by the community nor the one that was proposed by Chris in https://github.com/llvm/llvm-www/blob/main/proposals/LP0001-LLVMDecisionMaking.md and is marked as accepted (though is not documented anywhere in public and so is questionable just how āacceptedā this really is when it seems weāre not following it). Specifically, it missed critical feedback loops with the community (unanswered questions, no idea who the 2-4 āreview managersā were, no clear decision reached, lack of communication around the decision having been made, etc) and came across as a decision by fiat.
Our RFC process, even with the proposal from Chris, does not attempt to measure consensus of the community at all. Instead, it delegates this responsibility to 2-4 people. When those 2-4 people are in an echo chamber, this leads to decisions that are in the best interests of those few people and not the community as a whole. Ownership over community workflows needs to rest with the community and not with 2-4 self-selected people with a vested interest in what theyāre proposing.
The communication around this transition was inappropriately short (both in terms of timeline and in terms of tone). We heard no conclusion on the email RFC until numerous months later when weāre told the transition is starting in two days (over a weekend no less) or risk losing information. As a code owner and representative of the community on two different ISO standards bodies, I pointed out I wasnāt able to work within the very short timeframe given (for the full transition), but the transition happened anyway. This disrupted my ability to do my job to the point where I had to explore whether I needed to give up those responsibilities entirely. When I approached the board about the difficulties I faced, the board eventually told me to hire a lawyer because they needed to protect the LLVM Foundation. Almost every interaction I had on this topic made me feel as though my input was a frustrating bother, often due to defensive or dismissive responses.
This is the second time an RFC has severely damaged our ability to communicate with one another (the first was the bifurcation of our real-time communications with the IRC/Discord split).
On the technical front:
Not all of the archives were transitioned successfully to Discourse.
There is no way to respond via email to any post that was transitioned from the mailing list until someone on Discourse replies to it (thereās no way to discover what email address to reply to from the web UI).
Discourse emails go directly to spam for those with GMail accounts and sends mail only in HTML format (which reduces the accessibility for those of us who force plain text email). This can be worked around locally by setting up filters in GMail.
The workflow became much harder and more confusing for people who use an email workflow and itās clear it will never have the same ergonomics that email had.
I donāt see how Discourse will be an acceptable replacement for the commits mailing lists, so weāre still going to have the mailman instance sticking around for some time. Also, because the transition lost data in email messages, I think the archives need to stay around (almost) perpetually.
What can be improved?
My understanding for why the LLVM Foundation came into existence was largely for it to serve community needs for hosting developers meetings and has been highly successful at that. But when the foundation puts the business entityās needs above those of code owners within the community on community infrastructure topics, I start to lose faith that the LLVM Foundation should have decision-making authority over the infrastructure used by the entire community. I looked for the LLVM Foundationās mission statement to see how the community factors in; there is no mission statement. I looked at the bylaws to see if the word community ever appears; it does not. When the board replied to my concerns with āIt is within our rights to protect ourselvesā they were absolutely correct, but this also demonstrates clearly why the board does not always have the best interests of the community at heart ā the board members have a fiduciary duty to the organization, but they have no such duty to the community. I think the LLVM Foundation needs to make a strong commitment to factor in the communityās needs and when the community needs something the US-based LLVM Foundation 501(c)(3) corporation finds problematic, the onus is on the board to convince the community of why the boardās needs should come first rather than the communityās. This commitment could come in many forms (adding a mission statement, making changes to bylaws or policy documents, etc); the key point is to be clear that the LLVM community drives the LLVM Foundation decisions, not vice versa.
Relatedly, I think the community needs a transparent way to judge community consensus before any further infrastructure projects are undertaken.
I like that we decided to move away from the mailing lists. I had used a Google groups readonly shadow of the LLVM developer email which was nice, but I had no such option for clang or other projects. Having it all in Discourse has potential.
At this point, however, I feel less connected to the community. I think I need to check my Discourse settings on which area Iād like to follow. Is there a Discourse page where I can see all of the posts regardless of which area? (the way I used to skim over the newsreader group). I would often see a topic that would spark my interest. Using Discourse for me means I have to already know which things might be of interest to me.
https://discourse.llvm.org/latest shows all recent posts in any topic. (Thereās a link to it on the homepage, but I guess itās not entirely obvious.)
Despite it being redundant feedback, Iād like to throw my +1 in here, in part because I feel like part of the ādecision of consensusā on this was based on discounting the feedback as āa vocal minorityā rather than as representatives with a sizable group nodding our heads heads in agreement.
I agree with just about everything said here:
The āwhat went wellā to me seems that the technical transition seemed about appropriate. Some of the history seems to have been lost, but otherwise I thought the data transition was reasonably painless.
āWhat did not go wellā is pretty much everything having to do with community consensus building and communication. From my perspective reading the llvm-dev threads, there was at least as many folks against as for, which is, IMO, no where near consensus.
The announcement of the transition happening on a Friday Evening and taking place Monday Morning was absurd as well. It seems this was (at least unintentionally) designed to give dissenters as little time to question the decision as possible.
All communication since the transition seems to have been received poorly/personally (rather than as an objection to the process), and frankly, otherwise ignored. I really expected better from a board who is intended to help the community, rather than itself.
What could improve:
Decisions need to be made with consensus. It is obvious that our previous way of measuring consensus is completely broken if the board can so differently interpret the communityās feedback than everyone in this thread. I suggest some sort of voting mechanism (how we choose to limit the voting set is up for suggestion, but I could think either a poll of the code-owners, a poll of āseriousā contributors, or just a poll of contributors), and follow some level of ISO consensus determination. I fear another determination of consensus like this will severely alienate the community at large.
Look at that, seems I had a lot to say for a silent +1er:D
Sorry for very delayed response here. I hadnāt seen this thread. Which, well, says it all right there.
Iām not going to go into any detail because a) I have before in both public and private and b) the major topics are already well covered above. I am only adding a couple of points which seem most relevant.
What went well?
The actual migration. I was expecting chaos similar to the github issues import issues, and that did not happen.
What did not go well?
The decision making and communication leading up to the transition. This has been covered in detail by others. This has been an utter and complete failure, the result of which is I have much less trust in the board going forward.
The actual experience of using discourse is, for me, strictly worse than email. Iāve managed to make it mostly work, but a) responses take longer to prepare, and b) I reply to less. Given the mentions of decreased traffic*, I seem to not be the only one.
I have not seen anyone present actual statistics on this. My impression is that traffic has dropped sharply, but without actual statistics it is hard to say if this is a correct impression.
What can be improved?
Honestly, nothing the board seems willing to entertain. Iāve largely given up on seeing honest discussion of the merits on this topic.
), but very happy about the move to discourse.