I see a number of people jumping onto this topic recently, so I’d like to share my experience. I also hope we can agree on a common approach towards the problem, so that bugs are handled consistently.
Compiler Explorer is of great help, as it provides old Clang releases all the way to 3.0. There are couple of common pitfalls using it for triaging:
a) Clang 3.1 through 3.8.1 are set to use headers from GCC 6.3.0 toolchain (--gcc-toolchain), including stdlibc++, which is not period-correct, and have caused some troubles for me Update: this is now fixed by my PR.
b) -cc1 is not supported. The only option I found to work is to open “Detailed compiler flags” (via the down arrow to the right of compiler flags), and use -Xclang prefix on everything. But even that wasn’t sufficient in some cases.
c) CE uses release builds of Clang that lack assertions, whereas most of the opened crashes require them to reproduce. Later on I explain how to deal with that Update: Clang builds with assertions are now live on CE (issue1, issue2, PR1, PR2, PR3).
d) trunk and assertions trunk are moving targets. Use release versions if you can.
e) Clang 3.0 lacks any headers. You can enable them using the following flags: -I/opt/compiler-explorer/gcc-4.4.7/lib/gcc/x86_64-linux-gnu/4.4.7/include -I/opt/compiler-explorer/gcc-4.4.7/include/c++/4.4.7 -I/opt/compiler-explorer/gcc-4.4.7/include/c++/4.4.7/x86_64-linux-gnu/ Update: this is now fixed by my PR.
f) Assertion builds support compiling and executing with -stdlib=libc++ starting with 3.5.
To work around CE limitations about disabled assertions and Clang 3.0 being the oldest one, me and my friend deployed our private instance of CE available at https://cexplorer.testlabs.pro . Update: as mentioned above, assertion builds are now live on official CE. I’ll outline the plan to phase-out my CE instance in the thread. The following points now are focusing on CE assertion builds: a) Use official CE if you can reproduce the issue there. Sometimes official CE and assertions builds gives you different answer when the issue was fixed (official CE is more optimistic). Feel free to use my CE in those cases.
b) Release builds of Clang 2.6+ with assertions available. c) Clang 3.1+ match official CE compilers.
d) Clang 2.6 through 3.0 use latest period-correct GCC toolchain Update: all Clang version on CE (release and assertions included) are now using period-correct GCC toolchains (PR).
e) Clang 2.6 is patched to enable its own C++ frontend, because it was considered of preproduction quality. Don’t get your hopes high.
f) -stdlib=libc++ doesn’t work, because I totally missed it while compiling. I’ll fix that in the near future. In the meantime you can supply -I /opt/compiler-explorer/clang-4.0.0/... options if you really need libc++ today Update: -stdlib=libc++ is compiling since 3.5, and is able to execute binaries since 3.5 (assertions) and since 3.8 (release) (my builds are beating official ones from releases.llvm.org, huh).
g) I’ll publish detailed info about my builds after I fix missing libc++. Update: here they are.
h) We try to upstream this effort (CE issue), so stay tuned for updates. I promise to keep it available as long as I reasonably could, but the plan is to make assertion builds available upstream. Update: now live on CE!
I prepared a table that lists Clang, GCC, and Xcode versions chronologically, so that it’s easy to understand what the landscape was at the time the bug was filed. It uses official release dates, so keep in mind that respective release branches were created weeks or even months prior to that.
Common options that make Clang crash include -std=, -O, --gcc-toolchain with period-correct GCC toolchain, -triple with the right triple. Don’t forget to pass -x c for C sources!
I’m personally quite reluctant to close old crashed as non-reproducible. In my experience, 80-90% of old crashed can be reproduced with the right tools (and, to be honest, a trump card of SSH to my instance of CE). If you’re about to close a crash as non-reproducible, send it my way, please. CC’ing me (@endilll) in the issue is the best way.
I always mark issues with the labels I wish they had. Two most common ones are clang:frontend (or another component) that replaces catch-all clang, and some kind of crash label: crash, crash-on-valid, crash-on-invalid, llvm:crash (I use it for crashes below CodeGen, e.g. in an optimization pass). worksforme is a label for issues closed as non-reproducible. confirmed is a label for issues that are reproducible to this day which we leave open. I also tend to mark llc and opt crashes with their respective tools: label, so that I can filter them out. The same goes for objective-c and objective-c++.
Note that GitHub UI provides two reasons that you can close an issue with: “close as completed” and “close as not planned”. Non-reproducible bugs fall into the latter. Don’t forget to pick the right one.
I consider it sufficient to close the bug as completed if we can reproduce it, and find the first Clang release when it was fixed (or find out it was fixed on trunk after Clang 16). Sometimes things get broken and fixed fast enough that they are not visible through any of official releases. I don’t consider bisecting practical for old bugs in general, so I close such bugs as non-reproducible.
The highest number of open bugs I remember was 20 440 from mid-May. One month later we have 130 less, despite all the new opened bugs. I’d like to thank everybody who helps to bring that number down!
Again, send non-reproducible crashes my way if you’re about to close them as non-reproducible.
Thank you for all your effort, I think we are making a solid impact although it will definitely take time to work through them all.
One thing I would add is that often you can find duplicates of crash bugs by looking for the assertion string or the top item on backtrace that is still llvm or clang that is not crash handling. So often if you find one crash bug that was fixed you can find duplicates that are also fixed in the same clang version.
Thank you (to everyone!) for all the help triaging issues in the bug database, this is not the most glamorous of work, but it’s very important for us to have a health understanding of quality.
I hope that Compiler Explorer will accept those changes! I’d like to avoid using links to your internal server whenever possible because we’d like those links to remain valid for as long as we possibly can.
Another one to keep in mind is that -fsyntax-only hides any codegen related bug
Thank you for the offer!
FWIW, for truly old issues (ones filed against 3.x), I am comfortable erring on the side of closing the issue when in doubt, but be sure to look for duplicates before closing because another issue may have more details on how to reproduce. I’d think we’d rather lose a few ancient bugs that have been reported once and never seen again and are hard to reproduce than have a bug database filled with stale issues.
Two more to keep in mind: needs-reduction for issues that have a large reproducer that need a reduced testcase to be actionable, and incomplete for issues that do not have any reproducer. If the issue is "ancient’ (more than a few years old and the reporter is a “mannequin” from the bugzilla migration), it is reasonable to close an issue that’s incomplete, but be sure to add the label and leave a comment telling the reporter that the issue can be reopened with more details on how to reproduce.
Agreed. When it comes to issues, we want to err on the side of practicality. If it’s unlikely anyone can act on the issue, it can be closed, but we should leave comments letting the reporter know it can be reopened easily enough if it’s still an issue.
I’d also recommend adding the enhancement tag to anything you triage that turns out to be a feature request rather than a bug report.
I totally forgot to bring that up. Now the first post is updated. Thank you!
I understand how important longevity of those links are. I don’t have a clear plan discussed with CE staff, but I think we should be able to upload all submissions to official CE, and configure a bunch of redirects. If we get lucky, we can go farther than that, and provide a clear way for people to manually convert links in case redirection goes down.
I also plan to update all the links in our bug tracker (hopefully there won’t be too much) to make it fail-proof, but I’m sure those links have already spread wider than that.
I understand where you come from, but the whole idea of rebuilding old Clang versions with assertions and deploying my own instance of CE was born out of disagreement with this approach
There are also missing feature and feature request labels that have similar meaning, which I find a bit less abstract. I wonder which ones should be used.
I also would like to note that we should think twice before going beyond 5 labels in total (keep in mind that bugzilla is always there for anything 2+ years old), keeping only the most important ones.
Makes sense, but I’d hold that off until we hear from CE on assertion builds (or conclude that they are not going to answer). Private instance of CE is something even I wouldn’t recommend in official documentation, unless all other options are exhausted.
I’ve been using “enhancement” as that was got applied to the bugzilla-migrated entries. At some point, we really need to get a handle on our use of labels so we can remove duplicate or low-value labels without a lot of spam.
To hopefully be more clear, I am talking about the case where you cannot reproduce on either Compiler Explorer or your server. e.g., we need to put in the effort to really try to reproduce the issue as best we can, but at some point you don’t have anything else you can try. Once you feel you’ve exhausted your testing options, it’s better to close as not reproducible than it is to leave the issue open with no additional information on it.
You could invent a pedantic difference between missing feature and feature request e.g., “missing” implies it’s part of a C++ standard but isn’t (yet) supported, vs “request” is something unrelated to the standards.
But, I’d have to question whether that degree of distinction is really fair to someone asking for a Thing and then (most likely) randomly picking one of the two labels because it’s not at all clear from the label name that they actually mean different things.
Agree. It even makes me think that we should convert them to enhancement, and call it a day, because it’s hard to express such subtleties succinctly enough for labels.
It’s also worth noting that such distinction makes sense when you have a standard, but I guess about a half of LLVM codebase doesn’t have one to refer to.
Personally, I would prefer that. However, as I understand things, there is no way to add or remove a label without causing an email to be generated for everyone subscribed to the issue, so converting labels can generate quite a bit of noise.
It could, but one of the aspects I consider important is use counts here. That’s our best way to express things like “this label is used often and it’s OK to use it” and “this label is rarely used, are you sure?” without writing docs that are not always read carefully.
What is the best way to deal with old issues that are long fixed.
I started going through old issues with no comments.
Finding issues that I could reproduce on compiler explorer, finding the version that show the bug/ error message the same as in the bug report, then finding the version where the bug seems to have been fixed, and commenting on the old posts with the link to compiler explorer page with these two versions.
Is there anything else that is worth including, or that it should be tagged with?
In my experience, there are 2 kinds of non-reproducible issues. One provides the reproducer that I can’t compile/run actually. Another one provides the reproducer that I can compile and run but I just don’t meet the problem that the opinion man described.
For the first kind, it may be not good to close it. But I don’t know what is the better way to solve it for long-existed issues.
For the second kind, I’ll wait for the respond from the opinion man if this is a new issue. And for older issues, I feel good to close the issue as fixed-in-trunk.
You correctly described the core part of the workflow.
It would be nice if you can search for duplicates (e.g. using assertions message or diagnostics).
Labeling is important as well. I describe that in item 6.
If for some reason you have to use trunk version (and 16 doesn’t fit), make sure to copy assertion message and backtrace over to issue thread. People revisitng your CE link in 5 years from now would appreciate that.
At least for old bugs, this is quite rare. If we can’t reproduce it after exhausting numerous options we have today, I don’t see a good reason to keep it open. I’d like to believe we’re pretty consistent with worksforme, so this can be revisited if necessary. For the new bugs, we’d reach out to author to help us with reproduction, but it’s rarely possible for old bugs.
Agree. We also mention something along the lines “fixed on trunk after Clang 16”.
