Proposal for an alternative bugtracking workflow

Project Infrastructure LLVM Dev List Archives
1

Hi LLVM community,

As discussed earlier, we in the clangd land feel that buganizer does not address the clangd’s needs as a bug-tracking system.

In our previous attempt to raise this on llvm-dev 1 we shared our idea to put the clangd issue tracker on GitHub. The participants raised multiple concerns, including the migration costs, whether GitHub is the right choice as an issue tracker, whether moving issues for clangd only will result in unwanted community fragmentation and others

These are all valid concerns, but a big portion of the thread was focused on migration of the existing bugs, community fragmentation issues, etc.
We feel it would also be useful to ask a more focused question on whether using the proposed workflow of tracking issues in multiple GitHub projects is a good fit for LLVM, ignoring the migration costs and such. Please note we’re not proposing to migrate from Bugzilla right away or saying that’s the only way to go forward with LLVM issue tracking, we merely want to understand what the community thinks about the proposed workflow and potential advantages and disadvantages of using it.

To reiterate, our proposal was to create a repository for each of the LLVM subprojects under the official LLVM GitHub account, e.g. github.com/llvm/clangd.
This repository would be run by the part of the community working on that project and would host the issue tracker for the project. The existing ‘github.com/llvm/llvm-project’ repository will be used to solely host the code, it will not have an issue tracker associated with it.

Do you think this workflow would be a good fit for tracking bugs in LLVM?

2

I don’t think that’d be a good way to implement it issue tracking with github.

If we want to switch to github issues, we can do so within a single repository and provide a tailored user experience for different sub-products, by using issue templates. Issue templates can automatically add issues to an appropriate label, assign to a default assignee, and provide default text for users to fill in.

E.g., reusing “llvm-git-prototype” repository for testing, I’ve created an test issue template for clangd. Click “New Issue” here, <https://github.com/llvm-git-prototype/llvm/issues>. Documentation for clangd could link directly to the clangd bug template, avoiding even the “select a template” screen.

3

+llvm-dev

To reiterate, our proposal was to create a repository for each of the
LLVM subprojects under the official LLVM GitHub account, e.g.
github.com/llvm/clangd <http://github.com/llvm/clangd>.
This repository would be run by the part of the community working on
that project and would host the issue tracker for the project. The
existing ‘github.com/llvm/llvm-project
<http://github.com/llvm/llvm-project>’ repository will be used to solely
host the code, it will not have an issue tracker associated with it.

Do you think this workflow would be a good fit for tracking bugs in LLVM?

It’s quite annoying to have a bug tracker on GitHub that isn’t in the
same place as the code. GitHub lets you close bugs by putting things
like ‘Fixes #123’ in the commit message. If we add a bug tracker to
llvm/llvm-project in the future ever, then messages like this in clangd
commits will close the wrong bugs and we can’t use this directly to
close bugs.

It’s true and unfortunate that ‘Fixes #123’ comments don’t work with this approach.
We can avoid closing random issues (which would be most annoying), by removing
an issue tracker from llvm-project repo that hosts the code, otherwise people will
constantly get confused anyway.
Point taken, certainly a bad thing that code and issues are in different (albeit) close
locations.

The big advantage of using GitHub’s issue tracker is its integration
with the rest of the workflow. For example, if you send a pull request
with ‘Fixes #123’ in the commit message, then this shows up
automatically in the bug’s stream with a message saying that the issue
will be closed automatically if the pull request is merged. I don’t
believe that this works across repos, even if you do manage the full
invocation (llvm/clangd#123, I think, but I end up having to look it up
every time, because you can’t test it without pushing). You can’t send
a pull request to the llvm repo that will close a clangd bug and have it
show up in the issue feed.

I’ve worked with a downstream pair of clang and llvm repos hosted on
GitHub, with associated issue trackers. People often filed bugs on the
wrong component (e.g. an LLVM bug causes clang to crash, people filed a
clang bug) and for other people (not just me!) to then close an

unrelated bug by putting ‘Fixes #12’ in their commit message and forget
that it’s clang bug 12, not llvm bug 12.

Now that the bugs can be moved across different repos, one can at least move
the bugs into the proper component.

This may be fine when we have only clangd using the GitHub issue
tracker, but when we get a second or third project doing this then it
quickly becomes a problem.

It is very easy with the GitHub issue tracker to filter bugs by tag. It
might be a better idea to have a clangd tag and use the
llvm/llvm-project’s issue tracker. This would need someone to go and
correctly tag things (only project members are allowed to modify tags,
because they’re used for things like ‘this person has signed a CLA’),
but it would give a workflow that would scale to more components using
the issue tracker and would allow better integration with the rest of
the GitHub tooling.

David

Would be interested in other GitHub tooling things that break on multi-project setups,
I would expect things like CI bots (i.e. Travis and such) to expect both code
and issues being hosted at the same project, but have no hands-on experience there.
Do you have anything in mind that might also break (other than ‘Fixes #123’ comments discussed above)?

The major downside of tag-based workflow is the fact that one can’t subscribe to notifications
only for specific tags. Having a single issue tracker for all of LLVM would mean, e.g. that there’s no
way for clangd developers to subscribe only for clangd issues and getting notified on all LLVM issues
would make the bug triaging process much more complicated on our side (we expect the number of clangd
bugs to be significantly less than the number of bugs for all backend+clang+lld+…)

4

Has anyone reached out to GitHub about potentially enabling this?

Or is it possible to implement via their API?

5

Issue templates certainly work in single-issue-tracker approach and filing a bug seems to be just fine with them.

See comments in my previous email about bug triaging, the fact that one can’t subscribe to be notified on certain tags only seems like a significant flaw for a smaller-scale project like clangd.

6

There’s another downside that’s IMO bigger (and not addressed by an e.g. API-based solution, or even Github offering this feature).

When multiple projects share a repository, partitioned by tags, then the “file bug”, “browse bugs”, “search bugs” actions are too far apart in the UI.
Instead of X home → browse bugs → file bug, the path becomes LLVM home → browse bugs → browse X bugs → file bug → file X bug.
It takes longer, there are more chances to get lost and confused.

(This is assumes there’s little overlap between projects in terms of bugs, which may not be true for e.g. llvm-backend <-> clang, certainly is true for clangd, other projects are probably debatable)

7

For the lack of a better place, filed a ticker against GitHub support. Will update the thread when they come back to me.

8

BTW, Google search found this tool:
https://github.com/bcongdon/github-label-notify

9

The script queries all issues with specified labels and “notifies” on each of those.

While this approach works, I doubt it would scale well if everyone is doing this by hand. OTOH, building a service that does this for everyone might be feasible, albeit non-trivial and it’s not clear who should own this.

So overall having this supported by GitHub would mean a much better UX…

10

(briefly focused on notifications / subscriptions to tags)

Somewhat regardless of whether we use a single repo of issue tracking or multiple repos of issue tracking, we will need this. As a concrete example, I suspect maintainers of a particular target will have a strong desire to follow issues with that target, but not all other targets, and yet here we will have enough overlap to make these necessarily go into a single repo.

As to who should own this, I think that the community as a whole has to invest in infrastructure to support excellent issue tracking. The current situation is actively hurting the entire project, and IMO is even more urgent to fix than moving the code to github. So I think there are lots of resources we can and should point at addressing basic missing functionality like notifications.

Ilya, can you forward whatever ticket or tracking entry there is here? I’d love to try and surface the critical nature of this for the LLVM project to GitHub folks.

11

Well, it’s not really critical for us now, because we have not switched to github issues. And I can’t really see any possibility of that happening in the short-term, either.

Even once we do decide we want to move that way – which we haven’t yet even done – it’ll be a long road to making it happen, and I suspect there’s many more critical missing features that we’ll discover along the way.

IMO, the best thing to focus on for fixing issue tracking in the short term, would be to resolve the most pressing issue in our bugzilla installation: that users cannot file bugs, because they’ll get discouraged at step #1: creating an account.

The best thing there is to figure out how to support ‘login with github’ (https://bugs.llvm.org/show_bug.cgi?id=39430)

A next-best-thing, since that seems a lot more difficult than it should be, would be to figure out how to re-enable account creation.

Possibly the spam overload was a temporary thing, and we can just re-enable it.
Or, if spam remains a serious issue, quite possibly it’s untargeted, and we can fix it just by slightly tweaking the account creation form (e.g. adding another field: “Type LLVM into the following box: ”, and verifying that that was done on the server).
Or, if it’s targeted spam, maybe we have to figure out how to add a captcha to the account create page.

12

See responses inline.

Well, it’s not really critical for us now, because we have not switched to github issues. And I can’t really see any possibility of that happening in the short-term, either.

Sure, the purpose of this thread was to explore what kind of issue we might run into if that happens, not trying to build consensus/sell the move to GitHub.

Even once we do decide we want to move that way – which we haven’t yet even done – it’ll be a long road to making it happen, and I suspect there’s many more critical missing features that we’ll discover along the way.

IMO, the best thing to focus on for fixing issue tracking in the short term, would be to resolve the most pressing issue in our bugzilla installation: that users cannot file bugs, because they’ll get discouraged at step #1: creating an account.

The best thing there is to figure out how to support ‘login with github’ (https://bugs.llvm.org/show_bug.cgi?id=39430)

A next-best-thing, since that seems a lot more difficult than it should be, would be to figure out how to re-enable account creation.

Possibly the spam overload was a temporary thing, and we can just re-enable it.
Or, if spam remains a serious issue, quite possibly it’s untargeted, and we can fix it just by slightly tweaking the account creation form (e.g. adding another field: “Type LLVM into the following box: ”, and verifying that that was done on the server).
Or, if it’s targeted spam, maybe we have to figure out how to add a captcha to the account create page.

(briefly focused on notifications / subscriptions to tags)

Somewhat regardless of whether we use a single repo of issue tracking or multiple repos of issue tracking, we will need this. As a concrete example, I suspect maintainers of a particular target will have a strong desire to follow issues with that target, but not all other targets, and yet here we will have enough overlap to make these necessarily go into a single repo.

As to who should own this, I think that the community as a whole has to invest in infrastructure to support excellent issue tracking. The current situation is actively hurting the entire project, and IMO is even more urgent to fix than moving the code to github. So I think there are lots of resources we can and should point at addressing basic missing functionality like notifications.

Ilya, can you forward whatever ticket or tracking entry there is here? I’d love to try and surface the critical nature of this for the LLVM project to GitHub folks.

I have filed a support ticket, but didn’t get any public link I could share (or any kind of response) yet.
I’ll update the thread as soon as they’ll come back. It might be worth posting the same bug in one of the public repositories under github.com/github, I just couldn’t figure out which one should I use…

13

(cross-posting from llvm-dev in hopes to potentially get more responses, see 2 for the original thread and sorry if you’re not interested and getting this twice)

Hi LLVM community,

As discussed earlier, we in the clangd land feel that buganizer does not address the clangd’s needs as a bug-tracking system.

In our previous attempt to raise this on llvm-dev 1 we shared our idea to put the clangd issue tracker on GitHub. The participants raised multiple concerns, including the migration costs, whether GitHub is the right choice as an issue tracker, whether moving issues for clangd only will result in unwanted community fragmentation and others

These are all valid concerns, but a big portion of the thread was focused on migration of the existing bugs, community fragmentation issues, etc.
We feel it would also be useful to ask a more focused question on whether using the proposed workflow of tracking issues in multiple GitHub projects is a good fit for LLVM, ignoring the migration costs and such. Please note we’re not proposing to migrate from Bugzilla right away or saying that’s the only way to go forward with LLVM issue tracking, we merely want to understand what the community thinks about the proposed workflow and potential advantages and disadvantages of using it.

To reiterate, our proposal was to create a repository for each of the LLVM subprojects under the official LLVM GitHub account, e.g. github.com/llvm/clangd.
This repository would be run by the part of the community working on that project and would host the issue tracker for the project. The existing ‘github.com/llvm/llvm-project’ repository will be used to solely host the code, it will not have an issue tracker associated with it.

Do you think this workflow would be a good fit for tracking bugs in LLVM?