RFC: Improving the experience of first-time contributors

Hi all,

Some discussions the night before prompted me to do a short lightning
talk on 'improving the experience of first-time contributors' at the
LLVM Cauldron back in September. I intended to write it up as an RFC,
but have only just got round to doing so. I've tried to make this
email self-contained, but you may still want to look at the slides or
recording of the lightning talk.

Slides: Can we improve the experience of first-time LLVM contributors? - Speaker Deck
Video: Can we improve the experience of first-time LLVM contributors? - LLVM Cauldron 2016 - YouTube

# The problem
As a first-time contributor, it's easy to be put off during the patch
submission process. e.g. the patch may sit unreviewed for weeks.
Typically people new to the project haven't yet reviewed 'review
currency', and so their patches are less likely to be picked out by
others scanning through llvm-commits. Obviously nobody is doing
anything wrong here, it's just likely to be off-putting to newcomers.
Even if feedback is negative, it's valuable to know someone has at
least looked at your code.

# How do other projects solve this?
I've always been impressed at how Rust has been able to attract a
steady stream of new contributors. They have a simple but seemingly
effective tool for helping people land their first patch: the highfive
bot. Upon a user making their first pull request, it posts a welcome
message and assigns someone to review it.

# A proposal
I propose that we form a group of people who specifically volunteer to
help review first-time patches. A simple phabricator bot could assign
someone randomly much like Rust's highfive.

One potential pitfall I'd like to highlight is that many patches
really require the sign-off of the code owner. It could be equally
frustrating for newcomers if people helpfully tell them to tweak their
variable names, fix their formatting, and make other cosmetic changes
only to find that time was wasted when the code owner step in a week
or two later to say a completely different approach needs to be used.

# Next steps

Does anybody have any views here? My 'raise your hand' poll at the
LLVM Cauldron did seem to think many felt things could be improved.
I'll be at the LLVM Dev Meeting this week and would love to discuss
further. Perhaps the BoF on raising the next generation of LLVM
developers would be an appropriate venue?
http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html

Best,

Alex

From: llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org] On Behalf Of Alex
Bradbury via llvm-dev
Sent: Tuesday, November 01, 2016 6:17 AM
To: llvm-dev
Subject: [llvm-dev] RFC: Improving the experience of first-time
contributors

Hi all,

Some discussions the night before prompted me to do a short lightning
talk on 'improving the experience of first-time contributors' at the
LLVM Cauldron back in September. I intended to write it up as an RFC,
but have only just got round to doing so. I've tried to make this
email self-contained, but you may still want to look at the slides or
recording of the lightning talk.

Slides: https://speakerdeck.com/asb/can-we-improve-the-experience-of-
first-time-llvm-contributors
Video: Can we improve the experience of first-time LLVM contributors? - LLVM Cauldron 2016 - YouTube

# The problem
As a first-time contributor, it's easy to be put off during the patch
submission process. e.g. the patch may sit unreviewed for weeks.
Typically people new to the project haven't yet reviewed 'review
currency', and so their patches are less likely to be picked out by
others scanning through llvm-commits. Obviously nobody is doing
anything wrong here, it's just likely to be off-putting to newcomers.
Even if feedback is negative, it's valuable to know someone has at
least looked at your code.

It's not just first-time contributors who can have this problem...
...but it can be especially off-putting to first-timers.

# How do other projects solve this?
I've always been impressed at how Rust has been able to attract a
steady stream of new contributors. They have a simple but seemingly
effective tool for helping people land their first patch: the highfive
bot. Upon a user making their first pull request, it posts a welcome
message and assigns someone to review it.

Sweet.

# A proposal
I propose that we form a group of people who specifically volunteer to
help review first-time patches. A simple phabricator bot could assign
someone randomly much like Rust's highfive.

That's an awesome idea for people who do actually post a patch to
phabricator. I'll just point out that phabricator isn't officially
required, and is in fact an extra barrier to entry, so we should
have some tactic for handling emailed patches if we can manage it.

I know people will react with "what barrier?" but it's YADA
(Yet Another Damn Account) and I know *I* hate doing that for
something I view as a one-off.

One potential pitfall I'd like to highlight is that many patches
really require the sign-off of the code owner. It could be equally
frustrating for newcomers if people helpfully tell them to tweak their
variable names, fix their formatting, and make other cosmetic changes
only to find that time was wasted when the code owner step in a week
or two later to say a completely different approach needs to be used.

People who sign up to be first-responders :slight_smile: should probably have
some sense of when patches are of that nature, and say so in their
initial response. If nothing else, cosmetic feedback will get new
contributors in tune with our style guide and make subsequent rounds
feel more substantive/less picky.

# Next steps

Does anybody have any views here? My 'raise your hand' poll at the
LLVM Cauldron did seem to think many felt things could be improved.

Agreed. Partly it's the raw size of the project; when I was on a
two-person team it was never a problem to get reviews done, but
with a dozen or more and looking for someone to volunteer, there's
a significant "somebody else will get to it" factor. And then nobody
does. With volunteer first-responders it's something they've signed
up to do, however, and that makes a big difference.
--paulr

Hi all,

Some discussions the night before prompted me to do a short lightning
talk on ‘improving the experience of first-time contributors’ at the
LLVM Cauldron back in September. I intended to write it up as an RFC,
but have only just got round to doing so. I’ve tried to make this
email self-contained, but you may still want to look at the slides or
recording of the lightning talk.

Slides: https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-time-llvm-contributors
Video: https://www.youtube.com/watch?v=32fmbEI9WrM

Great initiative! Thanks.

Have you seen this past discussion: http://lists.llvm.org/pipermail/llvm-dev/2016-March/097027.html
I think the main idea has been to set-up a regular IRC session dedicated to “new contributors”.

The problem

As a first-time contributor, it’s easy to be put off during the patch
submission process. e.g. the patch may sit unreviewed for weeks.
Typically people new to the project haven’t yet reviewed ‘review
currency’, and so their patches are less likely to be picked out by
others scanning through llvm-commits. Obviously nobody is doing
anything wrong here, it’s just likely to be off-putting to newcomers.
Even if feedback is negative, it’s valuable to know someone has at
least looked at your code.

How do other projects solve this?

I’ve always been impressed at how Rust has been able to attract a
steady stream of new contributors. They have a simple but seemingly
effective tool for helping people land their first patch: the highfive
bot. Upon a user making their first pull request, it posts a welcome
message and assigns someone to review it.

A proposal

I propose that we form a group of people who specifically volunteer to
help review first-time patches. A simple phabricator bot could assign
someone randomly much like Rust’s highfive.

One potential pitfall I’d like to highlight is that many patches
really require the sign-off of the code owner.

I don’t believe it is exact, unless you used “code owner” in a broader sense that what we mean usually with CODE_OWNERS.txt ; as you can see here: http://llvm.org/docs/DeveloperPolicy.html#code-owners “The sole responsibility of a code owner is to ensure that a commit to their area of the code is appropriately reviewed, either by themself or by someone else.”
You may need a sign-off from a “core contributor” well aware of the particular area you’re touching. And even that depends on the level of the changes: a small bug fix is less involved than something that requires large change or involves some design.

It could be equally
frustrating for newcomers if people helpfully tell them to tweak their
variable names, fix their formatting, and make other cosmetic changes
only to find that time was wasted when the code owner step in a week
or two later to say a completely different approach needs to be used.

I hope the “group of volunteers” will be well aware of LLVM and be able to spot when they need to pull-in someone else for a particular review (e.g. an important change in the new pass manager is likely to require Chandler to sign-off).

Best,

Hi all,

Some discussions the night before prompted me to do a short lightning
talk on ‘improving the experience of first-time contributors’ at the
LLVM Cauldron back in September. I intended to write it up as an RFC,
but have only just got round to doing so. I’ve tried to make this
email self-contained, but you may still want to look at the slides or
recording of the lightning talk.

Slides: https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-time-llvm-contributors
Video: https://www.youtube.com/watch?v=32fmbEI9WrM

The problem

As a first-time contributor, it’s easy to be put off during the patch
submission process. e.g. the patch may sit unreviewed for weeks.
Typically people new to the project haven’t yet reviewed ‘review
currency’, and so their patches are less likely to be picked out by
others scanning through llvm-commits. Obviously nobody is doing
anything wrong here, it’s just likely to be off-putting to newcomers.
Even if feedback is negative, it’s valuable to know someone has at
least looked at your code.

How do other projects solve this?

I’ve always been impressed at how Rust has been able to attract a
steady stream of new contributors. They have a simple but seemingly
effective tool for helping people land their first patch: the highfive
bot. Upon a user making their first pull request, it posts a welcome
message and assigns someone to review it.

On GitHub there is a project called “mention bot” [1] by Facebook that assigns reviewers by analysing the history of the files touched by the patch. Recently I was welcomed by a message: “thanks for your PR! By analyzing the history of the files in this pull request, we identified @A, @B and @C to be potential reviewers.”

[1] https://github.com/facebook/mention-bot

Have you seen this past discussion:
http://lists.llvm.org/pipermail/llvm-dev/2016-March/097027.html
I think the main idea has been to set-up a regular IRC session dedicated to
“new contributors”.

Thanks. I don't think I had looked at that discussion in detail.

I don’t believe it is exact, unless you used “code owner” in a broader sense
that what we mean usually with CODE_OWNERS.txt ; as you can see here:
http://llvm.org/docs/DeveloperPolicy.html#code-owners "The sole
responsibility of a code owner is to ensure that a commit to their area of
the code is appropriately reviewed, either by themself or by someone else."
You may need a sign-off from a “core contributor” well aware of the
particular area you’re touching. And even that depends on the level of the
changes: a small bug fix is less involved than something that requires large
change or involves some design.

Yes, I simply meant in a broader sense - just like the example you
give below of getting sign-off from Chandler for the pass manager.

I hope the “group of volunteers” will be well aware of LLVM and be able to
spot when they need to pull-in someone else for a particular review (e.g. an
important change in the new pass manager is likely to require Chandler to
sign-off).

I agree, it is an easily solved problem. Just something I thought I'd
highlight. Another reason for mentioning it is that if the fundamental
issue is that code owners who are needed to provide sign-off are
fundamentally too busy to review, then having this 'first responder'
team won't help in those cases.

Best,

Alex

From: "Paweł Bylica via llvm-dev" <llvm-dev@lists.llvm.org>
To: "Alex Bradbury" <asb@asbradbury.org>, "llvm-dev"
<llvm-dev@lists.llvm.org>
Sent: Tuesday, November 1, 2016 12:02:12 PM
Subject: Re: [llvm-dev] RFC: Improving the experience of first-time
contributors

> Hi all,

> Some discussions the night before prompted me to do a short
> lightning

> talk on 'improving the experience of first-time contributors' at
> the

> LLVM Cauldron back in September. I intended to write it up as an
> RFC,

> but have only just got round to doing so. I've tried to make this

> email self-contained, but you may still want to look at the slides
> or

> recording of the lightning talk.

> Slides:
> Can we improve the experience of first-time LLVM contributors? - Speaker Deck

> Video: Can we improve the experience of first-time LLVM contributors? - LLVM Cauldron 2016 - YouTube

> # The problem

> As a first-time contributor, it's easy to be put off during the
> patch

> submission process. e.g. the patch may sit unreviewed for weeks.

> Typically people new to the project haven't yet reviewed 'review

> currency', and so their patches are less likely to be picked out by

> others scanning through llvm-commits. Obviously nobody is doing

> anything wrong here, it's just likely to be off-putting to
> newcomers.

> Even if feedback is negative, it's valuable to know someone has at

> least looked at your code.

> # How do other projects solve this?

> I've always been impressed at how Rust has been able to attract a

> steady stream of new contributors. They have a simple but seemingly

> effective tool for helping people land their first patch: the
> highfive

> bot. Upon a user making their first pull request, it posts a
> welcome

> message and assigns someone to review it.

On GitHub there is a project called "mention bot" [1] by Facebook
that assigns reviewers by analysing the history of the files touched
by the patch. Recently I was welcomed by a message: "thanks for your
PR! By analyzing the history of the files in this pull request, we
identified @A, @B and @C to be potential reviewers."

[1] https://github.com/facebook/mention-bot

That sounds really nice!

-Hal

+Chandler, who has apparently been thinking along similar lines.

One point I'd like to reiterate is that this proposal is not the same
as just making it easier to select a reviewer. It also doesn't try to
solve the review problem in general, simply to pick one area where
slow reviews are likely to be most painful (first time contributors
being put off) and to try to improve it through a concerted group
effort. I like the 'first responders' name from Paul.

Best,

Alex

Hi,
The process of finding out something to work on that will be welcome upstream is also somewhat tricky.
I would appreciate an effort towards streamlining that.
Manasij

There was some discussion about that in the BoF on 'raising the next
generation of LLVM developers' today. Hopefully someone took notes and
will share.

Best,

Alex

I think that addressing slow reviews would be the one thing that would have the most impact. A timely response of any kind is better than no response at all.

-Krzysztof