[RFC] Debug info for coroutine suspension locations - take 2

[avogelsgesang]

TLDR: Reviving the discussion [RFC] Debug info for coroutine suspension locations from 2022; proposing to emit artificial DW_TAG_labels from the CoroSplit pass to map suspension point ids to the corresponding source code lines / code addresses. Draft implementation in #141937.

Motivation and use case

When inspecting a generator

generator<int> foo() {
    co_yield 1;
    co_yield 2;
    co_yield 1;
}

gen = generator();

it is not currently possible to write a debugger script to figure out at which line the generator gen is currently suspended.

A similar problem also arises when using coroutines for asynchronous programming. It is possible to get an asynchronous stack trace using debugger scripts (see, e.g., Debugging C++ Coroutines — Clang 21.0.0git documentation for an example). However, those scripts currently only show the function names, but not the exact line numbers where a coroutine was suspended.

E.g., when having a coroutine

task foo() {
    co_await bar();
    co_await baz();
    co_await bar();
}

the debugger script can only give us the backtrace "bar" was called from "foo", but it can’t tell us whether we are suspended at the 1st or the 2nd call to bar.

Both issues can be solved if we have a way to get the exact source location where a coroutine is suspended by inspecting a std::coroutine_handle in the debugger.

Current State of debugging coroutines

With the DWARF debug info generated by LLVM, one can already get the current suspension id __coro_index by inspecting the coroutine frame (currently requires manual reinterpret-casting in the debugger; with #141516, the pretty printer for coroutine_handle will also show the __coro_index out-of-the-box).

However, the suspension point id __coro_index is an opaque integer. There is currently no good way to map this compiler-generated, internal id back to a source location.

Proposed solution

Create artificial DW_TAG_label debug information. The labels would have well-known names, following the pattern __coro_resume_<N>.

The labels can then be looked up in gdb using either info line -function my_coroutine -label __coro_resume_2 or via gdb’s Python API gdb.lookup_symbol with domain=gdb.SYMBOL_LABEL_DOMAIN.

All the necessary infrastructure to emit debuginfo for labels is already in place, such that the corresponding code change is rather straightforward.

The generated DWARF looks like

0x00000c53:   DW_TAG_subprogram
                DW_AT_low_pc    (0x00000000000018f0)
                DW_AT_high_pc   (0x0000000000001de1)
                DW_AT_frame_base        (DW_OP_reg6 RBP)
                DW_AT_linkage_name      ("_ZL9coro_taski")
                DW_AT_name      ("coro_task")
                DW_AT_decl_file ("/home/avogelsgesang/Documents/corotest/llvm-example.cpp")
                DW_AT_decl_line (36)
                DW_AT_type      (0x0000005c "task")

[...]

0x00000c8e:     DW_TAG_label
                  DW_AT_name    ("__coro_resume_0")
                  DW_AT_decl_file       ("/home/avogelsgesang/Documents/corotest/llvm-example.cpp")
                  DW_AT_decl_line       (36)
                  DW_AT_low_pc  (0x0000000000001952)

0x00000c94:     DW_TAG_label
                  DW_AT_name    ("__coro_resume_1")
                  DW_AT_decl_file       ("/home/avogelsgesang/Documents/corotest/llvm-example.cpp")
                  DW_AT_decl_line       (38)
                  DW_AT_low_pc  (0x00000000000019db)

0x00000c9a:     DW_TAG_label
                  DW_AT_name    ("__coro_resume_2")
                  [...]

GCC’s behavior

Notably, gcc-14 also emits labels for the individual suspend points. However, gcc does not associate line number / low_pc with its labels, which currently makes them a bit pointless.

0x000027ec:   DW_TAG_subprogram
                DW_AT_name      ("coro_task")
                DW_AT_artificial        (true)
                DW_AT_low_pc    (0x0000000000001391)
                DW_AT_high_pc   (0x000000000000180b)
                DW_AT_frame_base        (DW_OP_call_frame_cfa)
                DW_AT_call_all_tail_calls       (true)
                DW_AT_sibling   (0x0000294c)

0x0000297c:     DW_TAG_label
                  DW_AT_name    ("resume.8")

0x00002981:     DW_TAG_label
                  DW_AT_name    ("destroy.8")

0x00002986:     DW_TAG_label
                  DW_AT_name    ("resume.6")

0x0000298b:     DW_TAG_label
                  DW_AT_name    ("destroy.6")

0x00002990:     DW_TAG_label
                  DW_AT_name    ("resume.4")

0x00002995:     DW_TAG_label
                  DW_AT_name    ("destroy.4")

0x0000299a:     DW_TAG_label
                  DW_AT_name    ("resume.2")

0x0000299f:     DW_TAG_label
                  DW_AT_name    ("destroy.2")

0x000029a4:     DW_TAG_label
                  DW_AT_name    ("coro.delete.frame")

0x000029a9:     DW_TAG_label
                  DW_AT_name    ("coro.delete.promise")

0x000029ae:     DW_TAG_label
                  DW_AT_name    ("actor.continue.ret")

0x000029b3:     DW_TAG_label
                  DW_AT_name    ("actor.suspend.ret")

0x000029b8:     DW_TAG_label
                  DW_AT_name    ("actor.begin")

0x000029bd:     DW_TAG_label
                  DW_AT_name    ("final.suspend")

Considered alternatives

It is possible to adapt the C++ code to keep track of the suspension point explicitly. However, this approach requires changes to the coroutine types and will be cumbersome for library-defined coroutine types like std::generator. Also it adds runtime overhead.

In [RFC] Debug info for coroutine suspension locations I previously proposed to represent __coro_index as an enum instead of an integer. The various enum entries would be represented as enumerators, where the name would reveal the line / column numbers. @dwblaikie expanded this proposal to attach DW_AT_low_pc annotations to the individual enumerators. One downside with this is that we would need to introduce some way to track a position of an enum-enumerator within the IR code. Not sure how to do so. For DW_TAG_label, all the required position tracking is already in place.

Open questions

• Do we think that using DW_TAG_label for this purpose makes sense?
• Should we coordinate with gcc, such that we write the same / similar debug information for coroutines between gcc and clang? If so, who would be good people from gcc to involve here? (I don’t know any gcc contributors)
• LLDB does currently simply ignores DW_TAG_label. As such, this information is currently only useful for gcc. I wonder how much work it would be to handle DW_TAG_label also in lldb. I will be writing a separate RFC on that shortly
• The DW_TAG_label for coroutines are compiler-generated, artificial labels. Intuitively, I would have attached DW_AT_artificial to it. However, the DWARFv5 standard, only mentions that types can be annotated as artificial, but not labels (see, e.g., “Appendix A. Attributes by Tag”).
  • should we propose a change to the DWARF standard to also allow DW_AT_artifical on DW_TAG_LABEL?
  • should we emit DW_AT_artifical, as a custom extension to the standard, already before this was officially blessed by the Dwarf standard? (Not sure how debuggers deal with unexpected attributes. Do they ignore them? Do they choke on them?)

CC @jyknight @adrian.prantl @dblaikie @dmitryduka @ChuanqiXu since you participated in the earlier thread on this topic. Also CC @hokein, since you were also looking into debuggability of coroutines in lldb

[dblaikie]

@avogelsgesang

The various enum entries would be represented as enumerators, where the name would reveal the line / column numbers. @dwblaikie expanded this proposal to attach DW_AT_low_pc annotations to the individual enumerators. One downside with this is that we would need to introduce some way to track a position of an enum-enumerator within the IR code. Not sure how to do so.

@tromey has been doing some Ada work that I think involved types that are dependent on runtime values (arrays with runtime bounds, perhaps) - maybe some of that work points to a direction for this work, if we did use an enum with addresses in it? (though code addresses would be another/different thing from the data values I think Tom’s looked at)

But if labels are just easier/good enough, I wouldn’t stand in the way.

Can’t hurt to ask GCC (perhaps Tom can speak to that too) about tehir thoughts/directions.

Using artificial seems fine/benign - I probably wouldn’t even bother asking the DWARF committee, but wouldn’t object either.

[ChuanqiXu]

The higher level sounds good. But since I don’t have a lot knowledge for DWARF info, I can’t make better comments in detail : )

[rnk]

This makes sense to me, but I think it’s important to clarify that the label is not going to be exactly equal to the PC point of the resume. The label is going to be good enough for producing a PC which can then be looked up in the line table to produce a source location for the resume point. It’s OK if register allocation inserts reloads and spills into the resume block before the label.

[tromey]

Can’t hurt to ask GCC (perhaps Tom can speak to that too) about tehir thoughts/directions.

It probably isn’t super hard to change DIEnumerator to take metadata rather than an APInt; or equivalently to add a different subclass of DINode that would be accepted by enumerations.

To me the label approach seems more “natural”. Though rather than have special names that mark the label as a resume point, perhaps using a new DWARF tag or attribute would be more appropriate. For example a new DW_AT_LLVM_coro_index, whose value is a constant that indicates the value of __coro_index (assuming IIUC), and whose presence marks the label as a resume point.

(Not sure how debuggers deal with unexpected attributes. Do they ignore them? Do they choke on them?)

Normally DWARF readers are expected to ignore tags or attributes that they do not understand. I would not worry about DW_AT_artificial at all.

[avogelsgesang]

Thanks for all the feedback!

Based on the input so far, I would go with DW_AT_label (instead of enumerator). It also feels more natural to me.

I will be adding the DW_AT_artificial flag (and, while at it, probably also support the column number in addition to the line number).

@rnk

This makes sense to me, but I think it’s important to clarify that the label is not going to be exactly equal to the PC point of the resume. […] It’s OK if register allocation inserts reloads and spills into the resume block before the label.

Noted. Afaik, this is also true for traditional C labels, where the compiler might also insert register spills before/after the label.

@rnk

The label is going to be good enough for producing a PC which can then be looked up in the line table to produce a source location for the resume point.

In addition to source code lookups, I am also condering to use them to set breakpoints at specific resumption points. Afaict, that should also still be fine, despite the additional register spills.

@tromey

rather than have special names that mark the label as a resume point, perhaps using a new DWARF tag or attribute would be more appropriate. For example a new DW_AT_LLVM_coro_index, whose value is a constant that indicates the value of __coro_index (assuming IIUC), and whose presence marks the label as a resume point.

What would be the underlying motivation here? Debug info size?

Do I understand you correctly that you would write DW_AT_LLVM_coro_index instead of DW_AT_name? Or are you proposing to write it in addition?

Removing DW_AT_name would make this debug information unusable from gdb. With gdb, I am already able to use debug scripts to lookup my __coro_resume_<N> labels today.

If I would write it in addition, we would not actually reduce debug info size

@dblaikie

Using artificial seems fine/benign - I probably wouldn’t even bother asking the DWARF committee, but wouldn’t object either.

Based on this thread, I think I wouldn’t “ask” but rather “inform”, so they can officially bless this usage if they see fit. But I wouldn’t block progress here on their approval.

How does the DWARF committee work? How do I submit a proposal?

[ArsenArsen]

the labels you see emitted by GCC are a byproduct of how it does coroutine implementation. ideally, they should be removed at some point (but they can be useful to debug the implementation itself). this is why they lack a location: they’re fictitious

it seems reasonable to me to associate suspension point values to labels whose locations correspond to the suspension points as you suggested, but I do agree with tromey WRT having a new tag or attribute (in general, IMO, more structured data is better) - the names as you suggested them would be okay since they’re in the reserved namespace, but it’d still mean creating some magic strings. no strong feelings if you prefer that approach (I’d not mind adjusting either way)

[avogelsgesang]

Thank you all, for all your input!

I just finished updating the Pull Request and think this should be ready for review.

With this PR, the generated DWARF is:

0x00000f71:     DW_TAG_label
                  DW_AT_name    ("__coro_resume_1")
                  DW_AT_decl_file       ("generator-example.cpp")
                  DW_AT_decl_line       (5)
                  DW_AT_decl_column     (3)
                  DW_AT_artificial      (true)
                  DW_AT_LLVM_coro_suspend_idx   (0x01)
                  DW_AT_low_pc  (0x00000000000019be)

I added the following additional attributes compared to the original proposal:

• DW_AT_decl_column - not strictly necessary, but introducing this later would have led to a weird layout for the binary LLVM IR representation. If I would have now introduced additional attributes without first introducing DW_AT_decl_column, line and column would not be next to each other in the binary representation, and we could never fix that in the future because of backward compatibility guarantees
• DW_AT_artificial - added as discussed
• DW_AT_LLVM_coro_suspend_idx - added as discussed

Note that DW_AT_decl_column is now also added for normal C / C++ labels

[ArsenArsen]

@avogelsgesang

How does the DWARF committee work? How do I submit a proposal?

AFAIK, change proposals for DWARF are to be submitted as described here: DWARF Public Comments

[pogo59]

Offhand it looks like the numeric suffix on the name and the LLVM_coro_suspend_idx duplicate information. If that’s the case, probably better to omit it from the name (fewer unique strings).

Regarding the DWARF committee, so far what you’re describing is a private agreement between producers and consumers. If the value of DW_AT_LLVM_coro_suspend_idx was somehow tied to the C++ standard for coroutines (e.g., if the index is specified in the standard) then adding a new standard attribute would be a welcome proposal. If you want to stick with it being a vendor attribute, we have floated the idea of a central informational repository of vendor extensions and it could be added to that list (which TBF does not yet exist).

[avogelsgesang]

Offhand it looks like the numeric suffix on the name and the LLVM_coro_suspend_idx duplicate information. If that’s the case, probably better to omit it from the name (fewer unique strings).

I am not sure that I understand the concerns about size correctly here.

Note that the labels __coro_resume_1, __coro_resume_2, …, __coro_resume_N will be the same across different coroutines. There is a natural upper bound on how may suspension points a single coroutine function has. Let’s assume 100. To store 100 of those strings, we need 17*100 = 1.7KB.

To my understanding, those strings will be reused across the functions inside a .o / .dwo files and will be deduplicated between those files by the linker. As such, we pay the 1.7KB once for the whole program, and not per coroutine.

Did I miss anything?

If that understanding is actually correct, I think we can live with those 2KB of additional memory consumption, as it gives us the benefit that we can gain value from it immediately, via debugger scripts, without first having to wait for debuggers to implement support for DW_AT_LLVM_coro_suspend_idx.

[avogelsgesang]

@pogo59

If the value of DW_AT_LLVM_coro_suspend_idx was somehow tied to the C++ standard for coroutines (e.g., if the index is specified in the standard) then adding a new standard attribute would be a welcome proposal.

I would be definitely interested in writing such a proposal.

The C++ standard does not dictate the use of a switch-resume model for lowering coroutines. Even the ABI which was agreed between vendors would theoretically allow other implementations (e.g., updating the resume function pointer inside the coroutine frame). As such, this is an implementation detail from the point of view of the C++ standard. However, all vendors (gcc, MS, clang) decided to go with the switch-based coroutine lowering, afaik. (Also see Debugging coroutine handles: The Microsoft Visual C++ compiler, clang, and gcc - The Old New Thing)

Is that sufficient to get it into the DWARF standard?

Also, where would this leave the PR currently in review? I would prefer to still get this merged in time for clang-21 (branch-cut is 2 weeks from now). Assuming I would now ship this with DW_AT_LLVM_coro_suspend_idx, do we have a way to elevate this custom attribute to a standard attribute without changes to the produced DWARF? E.g., could the DWARF standard simply standardize the 0x3e0d tag which I am now using in my PR? Or would adding a tag id from a vendor go against the basic tenets of the DWARF committee?

Would it be better to remove DW_AT_LLVM_coro_suspend_idx for clang-21 (and only use DW_AT_name + DW_AT_artificial for the time being) and re-add it after the new attribute is properly standardized?

[pogo59]

@avogelsgesang

Note that the labels __coro_resume_1, __coro_resume_2, …, __coro_resume_N will be the same across different coroutines. There is a natural upper bound on how may suspension points a single coroutine function has. Let’s assume 100. To store 100 of those strings, we need 17*100 = 1.7KB.

There’s a little more to it than that, but not enough to make a difference. If the names are scoped by each coroutine then as you say the number of unique label names will be small, and not a concern.

[pogo59]

@avogelsgesang

it [the label names] gives us the benefit that we can gain value from it immediately, via debugger scripts, without first having to wait for debuggers to implement support for DW_AT_LLVM_coro_suspend_idx.

I’ve tried to reread the previous discussion. It sounds like gcc/gdb currently leverage the private agreement about label names, and lldb does nothing. The idea is to add an attribute that lldb can use, and that gcc/gdb could move toward. Am I understanding that correctly?

If there are multiple vendors either using or moving toward matching implementations, not depending on the name scheme, then it is something that the DWARF committee is likely to look at favorably for standardizing.

As far as implementation goes, you should go ahead with DW_AT_LLVM_coro_suspend_idx because even if the DWARF committee does bless it, the new attribute would not be in the standard until DWARF v6 is published, which I would swag as being at least a year away. The standardized attribute would not have the same numeric value that you are using, because you are using an attribute number in the vendor-defined space, and the DWARF standard must not encroach upon that space. Vendors will just cope with having both attribute names/numbers; this would be far from the first time it has happened.

[avogelsgesang]

@pogo59

I’ve tried to reread the previous discussion. It sounds like gcc/gdb currently leverage the private agreement about label names, and lldb does nothing. The idea is to add an attribute that lldb can use, and that gcc/gdb could move toward. Am I understanding that correctly?

Almost

• gdb can currently read DW_TAG_label debug info, usually used for C/C++ goto labels. gdb itself has no support for C++ coroutines. It provides access to labels via its scripting APIs, though, and hence users can use debugger scripts to interact with those coroutine-specific labels. Those gdb debugger scripts would rely on the DW_AT_name of the labels and the naming convention used in my PR.
• gcc currently emits labels for coroutines, but they are not useful for the problem at hand as they are missing location information. I hope gcc would align on the same direction as proposed in this thread
• lldb does not currently support labels at all. As such, the debugging information written as part of my PR will currently still be worthless to lldb.

You can find example debugger scripts as well as a lot more “best practices” and background on debugging in coroutines in Debugging C++ Coroutines — Clang 21.0.0git documentation (which I recently overhauled)

@pogo59

If there are multiple vendors either using or moving toward matching implementations, not depending on the name scheme, then it is something that the DWARF committee is likely to look at favorably for standardizing.

Sounds good. @ArsenArsen would you be able to indicate whether gcc would be willing to adopt the same approach as proposed up-thread? Or could you point me to the right point of contact in the gcc / gdb community re C++ coroutine support?

@pogo59

As far as implementation goes, you should go ahead with DW_AT_LLVM_coro_suspend_idx because even if the DWARF committee does bless it

Sounds good. So my PR #141937 is already doing the right thing

I already got a LGTM from @ChuanqiXu, but would appreciate a short review from a DWARF / debug-info expert here. It’s the first time I am modifying anything related to MDNode / DILabel / DWARF. Could you, @pogo59, provide such a code review or point me in the right direction to find such a reviewer?

[pogo59]

See my comments on the PR.

[ArsenArsen]

@avogelsgesang

@ArsenArsen would you be able to indicate whether gcc would be willing to adopt the same approach as proposed up-thread?

I think the idea is fine, I’d also like @iains and, preferably, Jason Merrill to weigh, but, as is always the case, to discuss proposals you can simply post an RFC to gcc@gcc.gnu.org and gdb@sourceware.org (also throw in a link to this discussion if you do)

there’s no strict definition for what an RFC is: it is enough to simply start a thread named RFC: ... or similar to indicate that it is an RFC.

[iains]

Actually, the intention is that the labels in the coroutine expansion in GCC should have the location of the await expression to which they belong (so, if folks are observing that this is missing in the output, that is something to investigate). Likewise the suspension point index is tracked and updated regardless of whether awaiter.ready() is true or false. Other text that relates to the user’s original function body should retain its debug locations.

There are parts of the expressions that are hard to annotate tho (e.g. the ramp function and the ‘wrapper’ around the original function body).

I don’t have any significant state on GCC’s debug output processes. Jason Merrill or Jakub Jelinek would be much better to comment on any details of proposals to add to the DWARF std.

+1 for @ArsenArsen 's suggestion to start an RFC thread on gcc@gcc.gnu.org.