Why is lldb telling me "variable not available"?

Hi all, I’m working on improving the debugging experience for C++20
coroutines when compiled with LLVM/Clang, and I could use some help
from someone who understands debug information and DWARF (knowledge of
coroutines isn't necessary, I don't think).

Specifically, I’m trying to improve lldb’s behavior when showing
variables in the current stack frame, when that frame corresponds to a
coroutine function.

To illustrate my problem, I uploaded C++ source, LLVM IR, a DWARF
dump, and a shellscript demonstrating the invocations I used to create
each of these, as a gist on GitHub:
https://gist.github.com/modocache/670bc38e5a5ea2e0a3d6bafe8ea9c693

I'm looking at lldb's behavior on lines 23-40 of the C++ program,
https://gist.github.com/modocache/670bc38e5a5ea2e0a3d6bafe8ea9c693#file-test-cpp-L23-L40,
which I’ll paste below:

coro foo() {
  int i = 0;
  ++i;
  printf("%d\n", i); // 1
  // Breakpoint 1:
  //   (lldb) frame variable i
  //   (int) i = 1

  co_await suspend_always();
  int j = 0;
  ++i;
  ++j;
  printf("%d, %d\n", i, j); // 2, 1
  // Breakpoint 2:
  //   (lldb) frame variable i
  //   (int) i = <variable not available>
  //   (lldb) frame variable j
  //   (int) j = 1

Here 'foo' is a coroutine, and the comments denote commands executed
at the lldb prompt when stopped at a breakpoint placed at the 'printf'
above. At breakpoint 1, lldb correctly shows the value of 'i' to be 1.
At breakpoint 2, 'i' is shown as 'variable not available'. ('j',
however, is shown correctly.)

Looking at the LLVM IR debug info metadata, (and keeping in mind I'm
no expert at this stuff) I don't see anything out of the ordinary.
Coroutine passes outline sections of the coroutine function, and to
maintain state, they replace references to stack frame variables with
loads and stores onto a "coroutine frame" object. But, looking at the
IR in test.ll lines 221-238, the 'llvm.dbg.value' intrinsic is being
used to denote the location of the values, which I think should allow
lldb to print the correct values:
https://gist.github.com/modocache/670bc38e5a5ea2e0a3d6bafe8ea9c693#file-test-ll-L221-L238

The 'llvm.dbg.value' intrinsics reference metadata in slots !659 and
!668, and these seem correct to me as well:

!659 = !DILocalVariable(name: "i", scope: !660, file: !5, line: 24, type: !130)
!668 = !DILocalVariable(name: "j", scope: !660, file: !5, line: 32, type: !130)

Finally, I looked at the DWARF being produced for the program. When
broken at breakpoint two above, executing 'disassemble' at the lldb
prompt shows me the program counters for the region I'm interested in:

    0x401885 <+373>: movq   -0x8(%rbp), %rax
    0x401889 <+377>: movl   $0x0, 0x40(%rax)
    0x401890 <+384>: movl   0x28(%rax), %edx
    0x401893 <+387>: addl   $0x1, %edx
    0x401896 <+390>: movl   %edx, 0x28(%rax)
    0x401899 <+393>: movl   0x40(%rax), %edx
    0x40189c <+396>: addl   $0x1, %edx
    0x40189f <+399>: movl   %edx, 0x40(%rax)
->  0x4018a2 <+402>: movl   0x28(%rax), %esi

Specifically, I think 0x401893 and 0x40189c respectively show 'i' and
'j' being incremented by 1. Now, looking at the DWARF dump, I can see
that 'i' is live between 0x401893 and 0x40189c, and 'j' is live
between 0x40189c and 0x4018b9:
https://gist.github.com/modocache/670bc38e5a5ea2e0a3d6bafe8ea9c693#file-test-dwarfdump-txt-L4063-L4086

Sure enough, when I break in lldb within 'i' region 0x401893 and
0x40189c, 'frame variable i' succeeds. But outside of that region,
lldb tells me '(int) i = <variable not available>'. How do I improve
the debug info here such that lldb can show 'i' outside of that small
live range?

Any and all help, questions, comments, are very much appreciated!

- Brian Gesiak

Hi Brian,

Thanks for working on coroutines, the debugging experience, and in
particular thanks for the comprehensive write-up!,

Specifically, I’m trying to improve lldb’s behavior when showing
variables in the current stack frame, when that frame corresponds to a
coroutine function.

[...]

Everything in the IR appears correct to my eyes, although I know next
to nothing about coroutines and might have missed something. The
simplest explanation of why the variable location goes missing can be
seen in the disassembly:

    0x401885 <+373>: movq   -0x8(%rbp), %rax
    0x401889 <+377>: movl   $0x0, 0x40(%rax)
    0x401890 <+384>: X movl   0x28(%rax), %edx
    0x401893 <+387>: X addl   $0x1, %edx
    0x401896 <+390>: X movl   %edx, 0x28(%rax)
    0x401899 <+393>: X movl   0x40(%rax), %edx
    0x40189c <+396>: addl   $0x1, %edx
    0x40189f <+399>: movl   %edx, 0x40(%rax)
->  0x4018a2 <+402>: movl   0x28(%rax), %esi

Where I've marked with 'X' before the mnemonic the instructions that
the variable location list covers. The location of "i" is correctly
given as edx from its load to its store, and ends when edx is
overwritten with the value of "j". In all the rest of the code, the
variables value is in memory, and the DWARF data doesn't record this.

Ideally debug info would track variables when they're stored to memory
-- however we don't automatically know whether any subsequent store to
memory will overwrite that variable, and so we don't track locations
into memory. PR40628 [0] is an example of what can go wrong, where we
described a variable as being in memory, but didn't know when that
location was overwritten.

If whatever's producing the coroutine IR has guarantees about where
and when variables are loaded/stored from/to memory, it should be
possible to put more information into the IR, so that the rest of LLVM
doesn't have to guess. For example, this portion of IR:

  %15 = load i32, i32* %i.reload.addr62, align 4, !dbg !670
  call void @llvm.dbg.value(metadata i32 %15, metadata !659, metadata
!DIExpression()), !dbg !661
  %inc19 = add nsw i32 %15, 1, !dbg !670
  call void @llvm.dbg.value(metadata i32 %inc19, metadata !659,
metadata !DIExpression()), !dbg !661
  store i32 %inc19, i32* %i.reload.addr62, align 4, !dbg !670

Could have a call to llvm.dbg.addr(metadata i32 *%i.reload.addr66,
...) inserted after the store, indicating that the variable is located
in memory. This should work (TM) so long as that memory is never
overwritten with something that isn't the current value of "i" on
every path after the call to llvm.dbg.addr; and on every path after
the call to llvm.dbg.addr, when the variable is loaded form memory,
there's a call to llvm.dbg.value to indicate that the variable is
located somewhere other than memory now.

Providing that extra information should improve the location coverage
for your example, certainly when unoptimised. However, I believe (80%)
this method isn't safe against optimisation, because (for example)
dead stores can be deleted by LLVM passes without deleting the call to
llvm.dbg.addr, pointing the variable location at a stale value in
memory. Unfortunately I'm not aware of a facility or technique that
protects against this right now. (CC Reid who I think ran into this
last?).

Note that there's some support for tracking variables through stack
spills in post-isel debug data passes, however those loads and stores
operate in well defined ways, and general loads and stores might not.

[0] https://bugs.llvm.org/show_bug.cgi?id=40628

Apologies for the slow response here Jeremy. Your reply has been incredibly helpful so far, I just need to try adding ‘llvm.dbg.addr’ myself to confirm that works. Thank you! - Brian Gesiak

Thanks all, especially Jeremy, for your help.

Everything in the IR appears correct to my eyes, although I know next
to nothing about coroutines and might have missed something.

Yes, good point. I think a better explanation of the coroutine passes
is in order. The coro-split pass first uses 'llvm::LowerDbgDeclare' to
replace all llvm.dbg.declare for %i with llvm.dbg.value, and then
replaces all uses of the '%i = alloca i32' instruction with a
getelementptr instruction like this one:

%i.reload.addr63 = getelementptr inbounds %_Z3foov.Frame,
%_Z3foov.Frame* %FramePtr, i32 0, i32 7, !dbg !651

In other words, the value of %i is stored on the frame object, on the
heap, at an offset of 7 into the frame. I'm beginning to think a
fundamental fix for this issue would be to stop replacing
llvm.dbg.declare with llvm.dbg.value, and instead replace the
llvm.dbg.declare with llvm.dbg.addr that points the debugger to the %i
variable's new permanent location as an offset into the coroutine
frame object. Does this approach make sense to people on this mailing
list, who probably know more about how these intrinsics work than I
do?

If whatever's producing the coroutine IR has guarantees about where
and when variables are loaded/stored from/to memory, it should be
possible to put more information into the IR, so that the rest of LLVM
doesn't have to guess. For example, this portion of IR:

  %15 = load i32, i32* %i.reload.addr62, align 4, !dbg !670
  call void @llvm.dbg.value(metadata i32 %15, metadata !659, metadata
!DIExpression()), !dbg !661
  %inc19 = add nsw i32 %15, 1, !dbg !670
  call void @llvm.dbg.value(metadata i32 %inc19, metadata !659,
metadata !DIExpression()), !dbg !661
  store i32 %inc19, i32* %i.reload.addr62, align 4, !dbg !670

Could have a call to llvm.dbg.addr(metadata i32 *%i.reload.addr66,
...) inserted after the store, indicating that the variable is located
in memory.

I tried multiple approaches to manually inserting an llvm.dbg.addr
after the store instruction, as per your suggestion, Jeremy. I used
llc to compile the IR into an object file that I then linked, and
inspected the DWARF generated for the file. Unfortunately, inserting
dbg.addr that operated on the reloaded values didn't lead to any
change in the DWARF that was produced -- specifically, this didn't
make a difference:

call void @llvm.dbg.addr(metadata i32* %i.reload.addr62, metadata
!873, metadata !DIExpression()), !dbg !884

I also tried adding a dbg.addr that attempted to point the debugger to
the %i variable's location at its offset on the coroutine frame:

call void @llvm.dbg.addr(metadata %_Z3foov.Frame* %FramePtr, metadata
!873, metadata !DIExpression(DW_OP_plus, 28)), !dbg !884

This changed the live ranges for %i in the DWARF that was output, but
not in a way that made the %i variable visible to the debugger. Again,
I wonder if maybe the correct path forward here is to, instead of
attempting to expand the live ranges in the DWARF for %i, to instead
signal in the debug info that the value of %i is always available to
read from the 7 offset into the coroutine frame. If this makes sense,
then I need to learn how to convey that via the llvm.dbg intrinsics --
llvm.dbg.addr sounds like what I want, doesn't it?

Again, thanks for all the help on this list. (PS: I've also been
enjoying reading the proposals you sent, Jeremy, for post-ISel debug
info salvaging!) Please let me know if my thinking above sounds right
or misguided in any way -- thanks!

- Brian Gesiak

Hi Brian,

In other words, the value of %i is stored on the frame object, on the
heap, at an offset of 7 into the frame. I'm beginning to think a
fundamental fix for this issue would be to stop replacing
llvm.dbg.declare with llvm.dbg.value, and instead replace the
llvm.dbg.declare with llvm.dbg.addr that points the debugger to the %i
variable's new permanent location as an offset into the coroutine
frame object. Does this approach make sense to people on this mailing
list, who probably know more about how these intrinsics work than I
do?

This matches a few similar use cases that I'm aware of -- certain
kinds of struct that are passed-by-value according to the language,
but passed-by-reference according to ABI, are treated in that way. In
general, the downside is that the debugger can only observe variable
values when they get written to memory, not when they're computed, as
dbg.values and dbg.declares aren't supposed to be mixed. Observing
variable values slightly later might be an improvement over the
current situation.

Although, I don't think this will work immediately, see below,

I tried multiple approaches to manually inserting an llvm.dbg.addr
after the store instruction, as per your suggestion, Jeremy. I used
llc to compile the IR into an object file that I then linked, and
inspected the DWARF generated for the file. Unfortunately, inserting
dbg.addr that operated on the reloaded values didn't lead to any
change in the DWARF that was produced -- specifically, this didn't
make a difference:

call void @llvm.dbg.addr(metadata i32* %i.reload.addr62, metadata
!873, metadata !DIExpression()), !dbg !884

Ouch, I tried this myself, and ran into the same difficulty. I'd
missed that all your functions are marked "optnone" / -O0, which means
a different instruction-selection pass (FastISel) runs, and it turns
out FastISel isn't aware of dbg.addrs existence. Even better, FastISel
doesn't manage to lower any debug intrinsic (including dbg.declare)
that refers to a GEP, because it doesn't have a register location (the
GEP gets folded into a memory addressing mode).

I've hacked together some support in [0], that allows dbg.addr's of
GEPs to be handled. A single dbg.addr at the start of the function
(and no dbg.values) should get you the same behaviour as a
dbg.declare. I suspect the reason why this problem hasn't shown up in
the past is because the coroutine code being generated hits a gap
between "optimised" and "not optimised": I believe all variables in
code that isn't optimised get their own storage (and so will always
have a stack or register location). Wheras in the coroutine code
you're generating the variable address doesn't get storage.

If [0] is useful for you I can get that landed; it'd be good to hear
whether this resolves the dbg.addr intrinsics not having an affect on
the output.

[0] https://github.com/jmorse/llvm-project/commit/40927e6c2b71ec914d937287a0c2ca6c52c01f6b

Awesome, thanks Jeremy.

Hi Brian,

> In other words, the value of %i is stored on the frame object, on the
> heap, at an offset of 7 into the frame. I'm beginning to think a
> fundamental fix for this issue would be to stop replacing
> llvm.dbg.declare with llvm.dbg.value, and instead replace the
> llvm.dbg.declare with llvm.dbg.addr that points the debugger to the %i
> variable's new permanent location as an offset into the coroutine
> frame object. Does this approach make sense to people on this mailing
> list, who probably know more about how these intrinsics work than I
> do?

This matches a few similar use cases that I'm aware of -- certain
kinds of struct that are passed-by-value according to the language,
but passed-by-reference according to ABI, are treated in that way. In
general, the downside is that the debugger can only observe variable
values when they get written to memory, not when they're computed, as
dbg.values and dbg.declares aren't supposed to be mixed. Observing
variable values slightly later might be an improvement over the
current situation.

This is very reassuring, thank you!

Although, I don't think this will work immediately, see below,

> I tried multiple approaches to manually inserting an llvm.dbg.addr
> after the store instruction, as per your suggestion, Jeremy. I used
> llc to compile the IR into an object file that I then linked, and
> inspected the DWARF generated for the file. Unfortunately, inserting
> dbg.addr that operated on the reloaded values didn't lead to any
> change in the DWARF that was produced -- specifically, this didn't
> make a difference:
>
> call void @llvm.dbg.addr(metadata i32* %i.reload.addr62, metadata
> !873, metadata !DIExpression()), !dbg !884

Ouch, I tried this myself, and ran into the same difficulty. I'd
missed that all your functions are marked "optnone" / -O0, which means
a different instruction-selection pass (FastISel) runs, and it turns
out FastISel isn't aware of dbg.addrs existence. Even better, FastISel
doesn't manage to lower any debug intrinsic (including dbg.declare)
that refers to a GEP, because it doesn't have a register location (the
GEP gets folded into a memory addressing mode).

I've hacked together some support in [0], that allows dbg.addr's of
GEPs to be handled. A single dbg.addr at the start of the function
(and no dbg.values) should get you the same behaviour as a
dbg.declare.

Maybe I'm running into the limitations of how far I can get here by
hacking up the LLVM IR directly, but I've made a couple of tries here
without any luck. I put the IR files and the DWARF dumps of
executables built from that IR in another GitHub gist,
https://gist.github.com/modocache/8a7b12eb42012990ba534787c4a47275.
Here's a summary of the attempts I've made -- all of them with your
patch included in my llvm-project tree:

1. Don't lower llvm.dbg.declare to llvm.dbg.value at all, no other
changes -- doesn't work, in the way I expected it to not work

I tried removing the call to llvm::LowerDbgDeclare that's being made
by the coroutine passes. The IR and DWARF dump after having done so
are in the files 'repro.declare.ll' and 'repro.declare.dwarf.txt'.

Using an executable built from this IR, I can use lldb to break at the
line where the declare exists,
https://gist.github.com/modocache/8a7b12eb42012990ba534787c4a47275#file-repro-declare-ll-L167,
using the command 'b repro.cpp:24'. Running 'e i' at this point prints
an incorrect value, '(int) $0 = 24742', but at least 'i' is found.

Then, continuing past a coroutine "suspend point" (a point after which
the value of 'i' is stored into an offset of the coroutine frame
object, an outlined coroutine function 'foo.resume' is invoked, 'i' is
loaded out of an offset on the coroutine frame object, and normally a
llvm.dbg.value call would have been generated for that load), and then
breaking at 'b repro.cpp:32', executing 'e i' results in 'error: <user
expression 1>:1:1: use of undeclared identifier 'i''. This fits my
mental model, I think: 'i' was declared in the function 'foo', and its
DWARF information
https://gist.github.com/modocache/8a7b12eb42012990ba534787c4a47275#file-repro-declare-dwarf-txt-L3711
only specifies its offset from the 'foo' frame. After the suspend
point, lldb is stopped in 'foo.resume', and the 'foo' frame is no
longer active, so lldb determines 'i' is out of scope. Makes sense!

2. Don't lower llvm.dbg.declare at all, sed to automatically replace
all llvm.dbg.declare with llvm.dbg.addr -- doesn't work, in an
unexpected way

The outputs from this are in the files repro.addr.ll and
repro.addr.dwarf.txt. My understanding based on
https://llvm.org/docs/SourceLevelDebugging.html#llvm-dbg-addr is that
llvm.dbg.addr should work at least as well as llvm.dbg.declare does.
However, when I use it and break at
https://gist.github.com/modocache/8a7b12eb42012990ba534787c4a47275#file-repro-addr-ll-L167
('b repro.cpp:24'), executing 'e i' in lldb outputs 'error: <user
expression 1>:1:1: use of undeclared identifier 'i''. This threw me
for a loop -- maybe I'm misunderstanding the relationship between
llvm.dbg.addr and llvm.dbg.declare?

- Brian Gesiak

I haven’t fully parsed this thread (sorry!), but I wanted to briefly mention that the SafeStack & ASan passes both do something similar (I think): move local variables backed by allocas onto a separate stack. These passes use replaceDbgDeclare to rewrite dbg.declares s.t. they point into the new stack. After that, llvm presumably runs LowerDbgDeclare (usually via InstCombine), but all the inserted dbg.values have useful locations relative to the new stack.

It could be worth investigating whether replaceDbgDeclare is a good fit for the coro-split pass.

vedant

Vedant, thank you! I had meant to ask if any of this reminded you all
of something else that I could emulate. I'll look into uses of
'replaceDbgDeclare' in SafeStack/ASan. - Brian

Vedant, Jeremy,

Thanks a ton! I copied ASan's use of 'replaceDbgDeclare', think that worked!

https://github.com/modocache/llvm-project/commit/afbc04e1dcba has some
extremely quick and dirty changes I made (with no tests!), and a link
to a Gist with the LLVM IR and DWARF produced,
https://gist.github.com/modocache/6f29093ba2827946011b422ed3bd2903.

There's only one kink: the spot I placed it has 'replaceDbgDeclare'
called multiple times, and it reuses the result of a single
getelementptr instruction. Because of this, only the last invocation
results in correct debug info being produced. Put another way: when I
place a breakpoint in my example on the last instance that %i is
reloaded, repro.cpp:45 at
https://gist.github.com/modocache/6f29093ba2827946011b422ed3bd2903#file-repro-cpp-L45,
I get the correct debug info: i = 3, j = 2. In every other spot, I get
'error: Couldn't materialize: couldn't get the value of variable i:
variable not available'. Still, is great progress for me.

I think to work around this error, I need to do 2 things:

1. Call replaceDbgDeclare only once.
2. Call it with the correct arguments -- you may have noticed in
https://github.com/modocache/llvm-project/commit/afbc04e1dcba#diff-f1204f42e8cb9e3718892e63059f8347R743
that I'm applying an offset of 0, which doesn't make much sense. I
need to figure out how to use this function such that it encodes in
DWARF to load at an offset from the coroutine frame, just as a
getelementptr would.

- Brian

Vedant, Jeremy,

Thanks a ton! I copied ASan's use of 'replaceDbgDeclare', think that worked!

https://github.com/modocache/llvm-project/commit/afbc04e1dcba has some
extremely quick and dirty changes I made (with no tests!), and a link
to a Gist with the LLVM IR and DWARF produced,
https://gist.github.com/modocache/6f29093ba2827946011b422ed3bd2903.

There's only one kink: the spot I placed it has 'replaceDbgDeclare'
called multiple times, and it reuses the result of a single
getelementptr instruction. Because of this, only the last invocation
results in correct debug info being produced. Put another way: when I
place a breakpoint in my example on the last instance that %i is
reloaded, repro.cpp:45 at
https://gist.github.com/modocache/6f29093ba2827946011b422ed3bd2903#file-repro-cpp-L45,
I get the correct debug info: i = 3, j = 2. In every other spot, I get
'error: Couldn't materialize: couldn't get the value of variable i:
variable not available'. Still, is great progress for me.

I think to work around this error, I need to do 2 things:

1. Call replaceDbgDeclare only once.
2. Call it with the correct arguments -- you may have noticed in
https://github.com/modocache/llvm-project/commit/afbc04e1dcba#diff-f1204f42e8cb9e3718892e63059f8347R743
that I'm applying an offset of 0, which doesn't make much sense. I
need to figure out how to use this function such that it encodes in
DWARF to load at an offset from the coroutine frame, just as a
getelementptr would.

Haven't looked at this closely, but (2) sounds right, the goal is to let the compiler know that variables which used to live in one memory location have moved to another. If you have the precise address of the variable's new location within the 'coro' frame, you don't need to specify the offset parameter to replaceDbgDeclare ("NewAddress" is exact). More typically, instrumentation passes maintain a pointer to the new frame and a table of offsets for various variables, so the common pattern is 'replaceDbgDeclare(oldAllocaOrGEPForVar, newFramePtr, ..., offsetIntoNewFramePtrForVar)'.

vedant

Jeremy, Vedant, thank you both for your help! I think I have something
working pretty well: https://reviews.llvm.org/D75338

I'd greatly appreciate code review here -- in the course of writing
the patch I definitely encountered gaps in my knowledge about debug
info. In particular, I wasn't able to use the '-mllvm -use-dbg-addr'
option successfully, and I don't quite understand why. Having read the
docs in https://llvm.org/docs/SourceLevelDebugging.html, I'd expect
the patch to work just as well whether using 'dbg.declare' or
'dbg.addr', but in practice when using 'dbg.addr' I couldn't find any
variable entries in the DWARF that was output. Maybe I'm not using
that option correctly?

I hope to spend some more time soon learning more about debug info
outside of this particular patch, so thanks again for all your help.

- Brian Gesiak