breakpoint command

lldb looks like it’s adding some offset for the breakpoint command? The disassemble command looks like it works fine, but the breakpoint command is using an address fairly far into printf in this case. Before I head off to investigate, does anyone have any idea what might be going on here or suggestions on where to start?

Thanks.
-Mike

(lldb) disassemble -n printf
libc.so.6`__printf at printf.c:30:
0x7f961e2a2840: subq $216, %rsp

libc.so.6`__printf + 7 at printf.c:30:
0x7f961e2a2847: testb %al, %al
0x7f961e2a2849: movq %rsi, 40(%rsp)
0x7f961e2a284e: movq %rdx, 48(%rsp)
0x7f961e2a2853: movq %rcx, 56(%rsp)
0x7f961e2a2858: movq %r8, 64(%rsp)
0x7f961e2a285d: movq %r9, 72(%rsp)
0x7f961e2a2862: je 0x7f961e2a289b ; __printf + 91 at printf.c:34
0x7f961e2a2864: movaps %xmm0, 80(%rsp) <=============== breakpoint gets set here
0x7f961e2a2869: movaps %xmm1, 96(%rsp)
0x7f961e2a286e: movaps %xmm2, 112(%rsp)
0x7f961e2a2873: movaps %xmm3, 128(%rsp)
0x7f961e2a287b: movaps %xmm4, 144(%rsp)
0x7f961e2a2883: movaps %xmm5, 160(%rsp)
0x7f961e2a288b: movaps %xmm6, 176(%rsp)
0x7f961e2a2893: movaps %xmm7, 192(%rsp)
0x7f961e2a289b: leaq 224(%rsp), %rax
0x7f961e2a28a3: movq %rdi, %rsi
0x7f961e2a28a6: leaq 8(%rsp), %rdx
0x7f961e2a28ab: movl $8, 8(%rsp)
0x7f961e2a28b3: movl $48, 12(%rsp)
0x7f961e2a28bb: movq %rax, 16(%rsp)
0x7f961e2a28c0: leaq 32(%rsp), %rax
0x7f961e2a28c5: movq %rax, 24(%rsp)
0x7f961e2a28ca: movq 3556935(%rip), %rax
0x7f961e2a28d1: movq (%rax), %rdi
0x7f961e2a28d4: callq 0x7f961e2979c0 ; _IO_vfprintf_internal at vfprintf.c:211
0x7f961e2a28d9: addq $216, %rsp
0x7f961e2a28e0: ret

(lldb) b printf
Breakpoint 2: where = libc.so.6__printf + 36 at printf.c:30, address = 0x00007f961e2a2864 (lldb) breakpoint set -n printf Breakpoint 3: where = libc.so.6__printf + 36 at printf.c:30, address = 0x00007f961e2a2864

(lldb) target modules lookup -s printf -v
1 symbols match ‘printf’ in /home/mikesart/data/src/blah2/build/blah:
Address: 0x0000000000000000 (0x0000000000000000)
Summary: 0x0000000000000000
0x0000000000000000
2 symbols match ‘printf’ in /lib/x86_64-linux-gnu/libc.so.6:
Address: libc.so.6[0x0000000000053840] (libc.so.6…text + 215200)
Summary: libc.so.6__printf at printf.c:30 Module: file = "/lib/x86_64-linux-gnu/libc.so.6", arch = "x86_64" CompileUnit: id = {0x0009273a}, file = "/build/buildd/eglibc-2.15/stdio-common/printf.c", language = "ISO C:1989" Function: id = {0x0009367e}, name = "__printf", range = [0x00007f961e2a2840-0x00007f961e2a28e1) FuncType: id = {0x0009367e}, decl = printf.c:29, clang_type = "int (const char *)" Blocks: id = {0x0009367e}, range = [0x7f961e2a2840-0x7f961e2a28e1) LineEntry: [0x00007f961e2a2840-0x00007f961e2a2864): /build/buildd/eglibc-2.15/stdio-common/printf.c:30 Variable: id = {0x000936a3}, name = "format", type= "const char *", location = rdi, decl = printf.c:29 Variable: id = {0x000936b3}, name = "arg", type= "va_list", location = DW_OP_fbreg(-216), decl = printf.c:31 Variable: id = {0x000936c2}, name = "done", type= "int", location = rax, decl = printf.c:32 Address: libc.so.6[0x0000000000053840] (libc-2.15.so..text + 215200) Summary: libc.so.6__printf at printf.c:30
Module: file = “/lib/x86_64-linux-gnu/libc.so.6”, arch = “x86_64”
CompileUnit: id = {0x0009273a}, file = “/build/buildd/eglibc-2.15/stdio-common/printf.c”, language = “ISO C:1989”
Function: id = {0x0009367e}, name = “__printf”, range = [0x00007f961e2a2840-0x00007f961e2a28e1)
FuncType: id = {0x0009367e}, decl = printf.c:29, clang_type = “int (const char *)”
Blocks: id = {0x0009367e}, range = [0x7f961e2a2840-0x7f961e2a28e1)
LineEntry: [0x00007f961e2a2840-0x00007f961e2a2864): /build/buildd/eglibc-2.15/stdio-common/printf.c:30
Symbol: id = {0x00001ebc}, range = [0x0000000000053840-0x00000000000538e1), name=“_IO_printf”
Variable: id = {0x000936a3}, name = “format”, type= “const char *”, location = rdi, decl = printf.c:29
Variable: id = {0x000936b3}, name = “arg”, type= “va_list”, location = DW_OP_fbreg(-216), decl = printf.c:31
Variable: id = {0x000936c2}, name = “done”, type= “int”, location = rax, decl = printf.c:32

gdb works as expected:

(gdb) disassemble printf
Dump of assembler code for function __printf:
0x00007ffff7054840 <+0>: sub rsp,0xd8
0x00007ffff7054847: test al,al
0x00007ffff7054849: mov QWORD PTR [rsp+0x28],rsi
0x00007ffff705484e: mov QWORD PTR [rsp+0x30],rdx
0x00007ffff7054853: mov QWORD PTR [rsp+0x38],rcx
0x00007ffff7054858: mov QWORD PTR [rsp+0x40],r8
0x00007ffff705485d: mov QWORD PTR [rsp+0x48],r9
0x00007ffff7054862: je 0x7ffff705489b
0x00007ffff7054864: movaps XMMWORD PTR [rsp+0x50],xmm0
0x00007ffff7054869: movaps XMMWORD PTR [rsp+0x60],xmm1
0x00007ffff705486e: movaps XMMWORD PTR [rsp+0x70],xmm2
0x00007ffff7054873: movaps XMMWORD PTR [rsp+0x80],xmm3
0x00007ffff705487b: movaps XMMWORD PTR [rsp+0x90],xmm4
0x00007ffff7054883: movaps XMMWORD PTR [rsp+0xa0],xmm5
0x00007ffff705488b: movaps XMMWORD PTR [rsp+0xb0],xmm6
0x00007ffff7054893: movaps XMMWORD PTR [rsp+0xc0],xmm7
0x00007ffff705489b: lea rax,[rsp+0xe0]
0x00007ffff70548a3: mov rsi,rdi
0x00007ffff70548a6: lea rdx,[rsp+0x8]
0x00007ffff70548ab: mov DWORD PTR [rsp+0x8],0x8
0x00007ffff70548b3: mov DWORD PTR [rsp+0xc],0x30
0x00007ffff70548bb: mov QWORD PTR [rsp+0x10],rax
0x00007ffff70548c0: lea rax,[rsp+0x20]
0x00007ffff70548c5: mov QWORD PTR [rsp+0x18],rax
0x00007ffff70548ca: mov rax,QWORD PTR [rip+0x364647] # 0x7ffff73b8f18
0x00007ffff70548d1: mov rdi,QWORD PTR [rax]
0x00007ffff70548d4: call 0x7ffff70499c0 <_IO_vfprintf_internal at vfprintf.c:211>
0x00007ffff70548d9: add rsp,0xd8
0x00007ffff70548e0: ret
End of assembler dump.
(gdb) b printf
Breakpoint 2 at 0x7ffff7054840: file printf.c, line 30.

In case you haven't looked at this yet, lldb has an option to skip the function prologue when setting function breakpoints. The setting is configurable via 'settings set target.skip-prologue false|true'. The default is true.

It looks like you have some debug info for libc.so since you are able to resolve some line numbers. My guess is it's getting this prologue offset information from the DWARF info.

Matt

Oh, brilliant. That gets the breakpoint set correctly at the start of
printf and it's hit now. And yes - the split symbol support means we have
full symbols for all the system stuff (that has installed symbols).

I'll investigate what is going on with the prologue dwarf stuff - something
is broken there since the prologue shouldn't put us in the middle of printf.

Thank you very much Matt.
-Mike

We do skip the prologue by default. We use the DWARF line tables to see where the prologue end is. If the DWARF doesn't have a prologue end marker in it, we try and figure this out ourselves by using the second line table entry in the line table. I am guessing there is either bad DWARF line table info made by the compiler in this case (with an bad prologue end marker), or no prologue end marker.

You can also see the some of the line table using:

(lldb) image dump line-table printf.c

If we are picking the second entry, the breakpoint you see us setting will probably be the second address in the line table.

There is also a bug here where we must have two "__printf" symbols because the disassembly is broken up:

(lldb) disassemble -n printf
libc.so.6`__printf at printf.c:30:
   0x7f961e2a2840: subq $216, %rsp

libc.so.6`__printf + 7 at printf.c:30:
   0x7f961e2a2847: testb %al, %al
   0x7f961e2a2849: movq %rsi, 40(%rsp)
   0x7f961e2a284e: movq %rdx, 48(%rsp)
   0x7f961e2a2853: movq %rcx, 56(%rsp)
   0x7f961e2a2858: movq %r8, 64(%rsp)
   0x7f961e2a285d: movq %r9, 72(%rsp)

The symbol table probably contains a "__printf" symbol with zero size and one with a larger size (there is a bug already files for this issue with "malloc" on bugzilla.

The ELF symbol table parser should try to only emit one symbol when it can by coalescing the two "__printf" symbols into one. The disassembly output should look like this:

(lldb) disassemble -n printf
libc.so.6`__printf at printf.c:30:
   0x7f961e2a2840: subq $216, %rsp
   0x7f961e2a2847: testb %al, %al
   0x7f961e2a2849: movq %rsi, 40(%rsp)
   0x7f961e2a284e: movq %rdx, 48(%rsp)
   0x7f961e2a2853: movq %rcx, 56(%rsp)
   0x7f961e2a2858: movq %r8, 64(%rsp)
   0x7f961e2a285d: movq %r9, 72(%rsp)

If you send me the ELF file that contains printf, I can check it out the line table and probably the symbol issue for you.

We do skip the prologue by default. We use the DWARF line tables to see
where the prologue end is. If the DWARF doesn't have a prologue end marker
in it, we try and figure this out ourselves by using the second line table
entry in the line table. I am guessing there is either bad DWARF line table
info made by the compiler in this case (with an bad prologue end marker),
or no prologue end marker.

You can also see the some of the line table using:

(lldb) image dump line-table printf.c

This doesn't look right. It shouldn't have duplicates like that I assume?

(lldb) image dump line-table printf.c
Line table for /build/buildd/eglibc-2.15/stdio-common/printf.c in `libc.so.6
0x00007f159c281840: /build/buildd/eglibc-2.15/stdio-common/printf.c:30
0x00007f159c281864: /build/buildd/eglibc-2.15/stdio-common/printf.c:30
0x00007f159c28189b: /build/buildd/eglibc-2.15/stdio-common/printf.c:34
0x00007f159c2818a3: /build/buildd/eglibc-2.15/stdio-common/printf.c:30
0x00007f159c2818a6: /build/buildd/eglibc-2.15/stdio-common/printf.c:35
0x00007f159c2818ab: /build/buildd/eglibc-2.15/stdio-common/printf.c:34
0x00007f159c2818ca: /build/buildd/eglibc-2.15/stdio-common/printf.c:35
0x00007f159c2818d9: /build/buildd/eglibc-2.15/stdio-common/printf.c:39
0x00007f159c2818e1: /build/buildd/eglibc-2.15/stdio-common/printf.c:39

...
If you send me the ELF file that contains printf, I can check it out the
line table and probably the symbol issue for you.

I put a zip file up here with the SO and the debug SO.

https://docs.google.com/file/d/0B05iIR_mx6ApWXYzMmFzaWNSZ3M/edit?usp=sharing

Thanks Greg.
-Mike

So I did the following command:

(lldb) image dump symtab --sort address

               Debug symbol
               >Synthetic symbol
               >>Externally Visible
               >>>
Index UserID DSX Type File Address/Value Load Address Size Flags Name
------- ------ --- ------------ ------------------ ------------------ ------------------ ---------- ----------------------------------
[ 4971] 5012 Code 0x0000000000053840 0x00000000000000a1 0x00000002 __printf
[ 5637] 5678 X Code 0x0000000000053840 0x00000000000000a1 0x00000012 _IO_printf
[ 6482] 6523 X Code 0x0000000000053840 0x00000000000000a1 0x00000012 printf

Note there are 3 symbols for this address as shown above. Currently we should the first one (__printf), but since this isn't the external name (note the X marking the latter two as external), we should probably show one of the external symbols.

The "symbol.st_info" is in bits 7:0 of the flags, and the "symbol.st_other" is in bits 15:8. So the flags are:

0x00000002 __printf STB_LOCAL | STT_FUNC
0x00000012 _IO_printf STB_GLOBAL | STT_FUNC
0x00000012 printf STB_GLOBAL | STT_FUNC

So there isn't much to be able to do with these symbols. Does anyone know why we have 3 symbols? Are these just aliases? Is there anyway to know that we should use "printf" by inspecting any other sections?

I checked out the line table for printf and found that there is indeed no "prologue_end" marker in the line table state machine, so we are setting a breakpoint on the second line table entry in the sequence which maps to 0x0000000000053864 as we saw in your previous output (which was slid to 0x7f961e2a2864). So everything is working as expected here given the information we have.

% ~/Documents/src/dwarfutils/trunk/build/Release/dwarfdump --debug-line=0x0001ef1b libc-2.15.so

The "symbol.st_info" is in bits 7:0 of the flags, and the
"symbol.st_other" is in bits 15:8. So the flags are:

0x00000002 __printf STB_LOCAL | STT_FUNC
0x00000012 _IO_printf STB_GLOBAL | STT_FUNC
0x00000012 printf STB_GLOBAL | STT_FUNC

So there isn't much to be able to do with these symbols. Does anyone know
why we have 3 symbols? Are these just aliases? Is there anyway to know that
we should use "printf" by inspecting any other sections?

The printf libc source code looks like this:

  24 #undef printf
...
  28 int
  29 __printf (const char *format, ...)
...
  41 #undef _IO_printf
  42 ldbl_strong_alias (__printf, printf);
  43 /* This is for libg++. */
  44 ldbl_strong_alias (__printf, _IO_printf);

So yes, there are two strong aliases and one local symbol. gdb seems to
prefer the local one when running stacktraces and doing "disassemble
printf" or "disassemble _IO_printf". I'll investigate this further though.

I checked out the line table for printf and found that there is indeed no
"prologue_end" marker in the line table state machine, so we are setting a
breakpoint on the second line table entry in the sequence which maps to
0x0000000000053864 as we saw in your previous output (which was slid to
0x7f961e2a2864). So everything is working as expected here given the
information we have.

If we have no prologue_end marker, would it make sense to not skip anything?

Thanks Greg.
-Mike

The "symbol.st_info" is in bits 7:0 of the flags, and the "symbol.st_other" is in bits 15:8. So the flags are:

0x00000002 __printf STB_LOCAL | STT_FUNC
0x00000012 _IO_printf STB_GLOBAL | STT_FUNC
0x00000012 printf STB_GLOBAL | STT_FUNC

So there isn't much to be able to do with these symbols. Does anyone know why we have 3 symbols? Are these just aliases? Is there anyway to know that we should use "printf" by inspecting any other sections?

The printf libc source code looks like this:

  24 #undef printf
...
  28 int
  29 __printf (const char *format, ...)
...
  41 #undef _IO_printf
  42 ldbl_strong_alias (__printf, printf);
  43 /* This is for libg++. */
  44 ldbl_strong_alias (__printf, _IO_printf);

So yes, there are two strong aliases and one local symbol. gdb seems to prefer the local one when running stacktraces and doing "disassemble printf" or "disassemble _IO_printf". I'll investigate this further though.

I checked out the line table for printf and found that there is indeed no "prologue_end" marker in the line table state machine, so we are setting a breakpoint on the second line table entry in the sequence which maps to 0x0000000000053864 as we saw in your previous output (which was slid to 0x7f961e2a2864). So everything is working as expected here given the information we have.

If we have no prologue_end marker, would it make sense to not skip anything?

I don't know what more recent gcc's do, but they used to omit the prologue_end marker, and instead use the convention that the first line table entry for the function is the beginning of the function, and the second is the end of the prologue. gdb has relied on that for a long long time (in absence of the prologue_end and back to the stabs days when there was no such thing.) So not skipping the prologue in this case seems like throwing the baby out with the bath water. Better would be to get your compiler to get it right...

Jim

The "symbol.st_info" is in bits 7:0 of the flags, and the "symbol.st_other" is in bits 15:8. So the flags are:

0x00000002 __printf STB_LOCAL | STT_FUNC
0x00000012 _IO_printf STB_GLOBAL | STT_FUNC
0x00000012 printf STB_GLOBAL | STT_FUNC

So there isn't much to be able to do with these symbols. Does anyone know why we have 3 symbols? Are these just aliases? Is there anyway to know that we should use "printf" by inspecting any other sections?

The printf libc source code looks like this:

  24 #undef printf
...
  28 int
  29 __printf (const char *format, ...)
...
  41 #undef _IO_printf
  42 ldbl_strong_alias (__printf, printf);
  43 /* This is for libg++. */
  44 ldbl_strong_alias (__printf, _IO_printf);

So yes, there are two strong aliases and one local symbol. gdb seems to prefer the local one when running stacktraces and doing "disassemble printf" or "disassemble _IO_printf". I'll investigate this further though.

Sounds like showing __printf is ok them since this matches the sources.

I checked out the line table for printf and found that there is indeed no "prologue_end" marker in the line table state machine, so we are setting a breakpoint on the second line table entry in the sequence which maps to 0x0000000000053864 as we saw in your previous output (which was slid to 0x7f961e2a2864). So everything is working as expected here given the information we have.

If we have no prologue_end marker, would it make sense to not skip anything?

I agree with Jim here and I would vote to still try and skip something. The strongest reason I have for this recommendation is that the debug info for args and locals _relies_ on being past the prologue, and doesn't describe the location correctly for the arguments and locals in the prologue. So when users set a breakpoint on "printf" and they stop there, all variable information is usually bogus until the prologue has been executed. Users don't like seeing garbage values (they file bugs) and then stepping over the first line and then saying "now the variables are correct".

I agree it'd be nice for the compiler to get it right, but we have folks
that are going to try to use this debugger, they're going to try to set a
breakpoint, and when it sets that breakpoint in the middle of the function
they will just switch back to gdb - they rightfully won't care about
prologue end markers or compiler bugs.

I'm all for figuring out how to skip something when we don't have that
info, but I'd prefer to error on the side of caution. Having some locals
with garbage in them seems better to me than having your breakpoint not hit
at all or hit after something important you wanted to debug.

For whatever it's worth, this is broken with gcc 4.6, and it appears to be
broken in gcc 4.8. It's tough to tell what happens with lldb & gcc 4.8
because something else is broken there - I think lldb has trouble with
dwarf4.

This is the debug_line info gcc 4.8 produces (identical to gcc 4.6) for the
little program down below.

.debug_line: line number info for a single cu
Source lines (from CU-DIE at .debug_info offset 0x0000000b):

            NS new statement, BB new basic block, ET end of text sequence
            PE prologue end, EB epilogue begin
            IA=val ISA number, DI=val discriminator value
<pc> [row,col] NS BB ET PE EB IS= DI= uri: "filepath"
0x00400900 [ 6, 0] NS uri: "/home/mikesart/data/src/blah2/blah.cpp"
0x00400932 [ 6, 0] NS
0x00400959 [ 10, 0] NS
0x00400986 [ 11, 0] NS
0x004009ac [ 14, 0] NS
0x004009bc [ 15, 0] NS
0x004009be [ 18, 0] NS
0x004009cd [ 19, 0] NS
0x004009e1 [ 20, 0] NS
0x004009e6 [ 21, 0] NS
0x004009e8 [ 21, 0] NS ET

   1 #include <stdio.h>
   2 #include <stdarg.h>
   3
   4 int
   5 myprintf (const char *format, ...)
   6 {
   7 va_list arg;
   8 int done;
   9
  10 va_start (arg, format);
  11 done = vfprintf (stdout, format, arg);
  12 va_end (arg);
  13
  14 return done;
  15 }
  16
  17 int main( int argc, char *argv )
  18 {
  19 myprintf("blah is %d\n", 5);
  20 return 0;
  21 }

> If we have no prologue_end marker, would it make sense to not skip anything?

I don't know what more recent gcc's do, but they used to omit the prologue_end marker, and instead use the convention that the first line table entry for the function is the beginning of the function, and the second is the end of the prologue. gdb has relied on that for a long long time (in absence of the prologue_end and back to the stabs days when there was no such thing.) So not skipping the prologue in this case seems like throwing the baby out with the bath water. Better would be to get your compiler to get it right...

I agree with Jim here and I would vote to still try and skip something. The strongest reason I have for this recommendation is that the debug info for args and locals _relies_ on being past the prologue, and doesn't describe the location correctly for the arguments and locals in the prologue. So when users set a breakpoint on "printf" and they stop there, all variable information is usually bogus until the prologue has been executed. Users don't like seeing garbage values (they file bugs) and then stepping over the first line and then saying "now the variables are correct".

I agree it'd be nice for the compiler to get it right, but we have folks that are going to try to use this debugger, they're going to try to set a breakpoint, and when it sets that breakpoint in the middle of the function they will just switch back to gdb - they rightfully won't care about prologue end markers or compiler bugs.

Doesn't GDB do the same thing here? The Apple version of GDB (GPL v2 did), maybe GDB was changed (the top of tree GPL v3 version) to deal with lame compiler debug info. Maybe in absence of the end of prologue marker, we should just use our disassembly based approach.

I'm all for figuring out how to skip something when we don't have that info, but I'd prefer to error on the side of caution. Having some locals with garbage in them seems better to me than having your breakpoint not hit at all or hit after something important you wanted to debug.

I agree.

For whatever it's worth, this is broken with gcc 4.6, and it appears to be broken in gcc 4.8. It's tough to tell what happens with lldb & gcc 4.8 because something else is broken there - I think lldb has trouble with dwarf4.

This is the debug_line info gcc 4.8 produces (identical to gcc 4.6) for the little program down below.

.debug_line: line number info for a single cu
Source lines (from CU-DIE at .debug_info offset 0x0000000b):

            NS new statement, BB new basic block, ET end of text sequence
            PE prologue end, EB epilogue begin
            IA=val ISA number, DI=val discriminator value
<pc> [row,col] NS BB ET PE EB IS= DI= uri: "filepath"
0x00400900 [ 6, 0] NS uri: "/home/mikesart/data/src/blah2/blah.cpp"
0x00400932 [ 6, 0] NS
0x00400959 [ 10, 0] NS
0x00400986 [ 11, 0] NS
0x004009ac [ 14, 0] NS
0x004009bc [ 15, 0] NS
0x004009be [ 18, 0] NS
0x004009cd [ 19, 0] NS
0x004009e1 [ 20, 0] NS
0x004009e6 [ 21, 0] NS
0x004009e8 [ 21, 0] NS ET

   1 #include <stdio.h>
   2 #include <stdarg.h>
   3
   4 int
   5 myprintf (const char *format, ...)
   6 {
   7 va_list arg;
   8 int done;
   9
  10 va_start (arg, format);
  11 done = vfprintf (stdout, format, arg);
  12 va_end (arg);
  13
  14 return done;
  15 }
  16
  17 int main( int argc, char *argv )
  18 {
  19 myprintf("blah is %d\n", 5);
  20 return 0;
  21 }

So we can change our approach to use disassembly when there is no prologue end marker. Try that out and let us know how it works out.

Greg

I don't know what more recent gcc's do, but they used to omit the
prologue_end marker, and instead use the convention that the first line
table entry for the function is the beginning of the function, and the
second is the end of the prologue. gdb has relied on that for a long long
time (in absence of the prologue_end and back to the stabs days when there
was no such thing.)

I think Jim is absolutely right here with his very useful reply. It looks
like gdb ignores the prologue end stuff entirely (which makes sense since
it doesn't look like gcc outputs it), and it sets the breakpoint on the
second line table entry. But in this case:

0x00400764 [ 6, 0] NS uri: "/home/mikesart/data/src/blah2/blah.cpp"

0x00400796 [ 6, 0] NS
0x004007bd [ 10, 0] NS
0x004007ea [ 11, 0] NS

gdb is treating 0x004007bd as the second line table entry, and lldb is
treating 0x00400796 as the second. This is pretty clear with the
disassembly output down below.

The problem in this case is that second block with the sse movaps
instructions isn't run, so the breakpoint is never hit.

So I think the change would be in how lldb handles what the second line
table entry is? Ie, skip duplicate entries.
-Mike

(lldb) disassemble -n myprintf -m
blah`myprintf(char const*, ...) at blah.cpp:6
   5 myprintf (const char *format, ...)
   6 {
   7 va_list arg;
blah[0x400764]: pushq %rbp
blah[0x400765]: movq %rsp, %rbp
blah[0x400768]: subq $224, %rsp
blah[0x40076f]: movq %rsi, -168(%rbp)
blah[0x400776]: movq %rdx, -160(%rbp)
blah[0x40077d]: movq %rcx, -152(%rbp)
blah[0x400784]: movq %r8, -144(%rbp)
blah[0x40078b]: movq %r9, -136(%rbp)
blah[0x400792]: testb %al, %al
blah[0x400794]: je 0x4007b6 ; myprintf(char const*,
...) + 82 at blah.cpp:6
blah`myprintf(char const*, ...) + 50 at blah.cpp:6
   5 myprintf (const char *format, ...)
   6 {
   7 va_list arg;
blah[0x400796]: movaps %xmm0, -128(%rbp)
blah[0x40079a]: movaps %xmm1, -112(%rbp)
blah[0x40079e]: movaps %xmm2, -96(%rbp)
blah[0x4007a2]: movaps %xmm3, -80(%rbp)
blah[0x4007a6]: movaps %xmm4, -64(%rbp)
blah[0x4007aa]: movaps %xmm5, -48(%rbp)
blah[0x4007ae]: movaps %xmm6, -32(%rbp)
blah[0x4007b2]: movaps %xmm7, -16(%rbp)
blah[0x4007b6]: movq %rdi, -216(%rbp)
blah`myprintf(char const*, ...) + 89 at blah.cpp:10
   9
   10 va_start (arg, format);
   11 done = vfprintf (stdout, format, arg);
blah[0x4007bd]: movl $8, -208(%rbp)
blah[0x4007c7]: movl $48, -204(%rbp)
blah[0x4007d1]: leaq 16(%rbp), %rax
blah[0x4007d5]: movq %rax, -200(%rbp)
blah[0x4007dc]: leaq -176(%rbp), %rax
blah[0x4007e3]: movq %rax, -192(%rbp)
blah`myprintf(char const*, ...) + 134 at blah.cpp:11

(gdb) disassemble /m myprintf
Dump of assembler code for function myprintf(char const*, ...):
6 {
   0x0000000000400764 <+0>: push rbp
   0x0000000000400765 <+1>: mov rbp,rsp
   0x0000000000400768: sub rsp,0xe0
   0x000000000040076f: mov QWORD PTR [rbp-0xa8],rsi
   0x0000000000400776: mov QWORD PTR [rbp-0xa0],rdx
   0x000000000040077d: mov QWORD PTR [rbp-0x98],rcx
   0x0000000000400784: mov QWORD PTR [rbp-0x90],r8
   0x000000000040078b: mov QWORD PTR [rbp-0x88],r9
   0x0000000000400792: test al,al
   0x0000000000400794: je 0x4007b6
   0x0000000000400796: movaps XMMWORD PTR [rbp-0x80],xmm0
   0x000000000040079a: movaps XMMWORD PTR [rbp-0x70],xmm1
   0x000000000040079e: movaps XMMWORD PTR [rbp-0x60],xmm2
   0x00000000004007a2: movaps XMMWORD PTR [rbp-0x50],xmm3
   0x00000000004007a6: movaps XMMWORD PTR [rbp-0x40],xmm4
   0x00000000004007aa: movaps XMMWORD PTR [rbp-0x30],xmm5
   0x00000000004007ae: movaps XMMWORD PTR [rbp-0x20],xmm6
   0x00000000004007b2: movaps XMMWORD PTR [rbp-0x10],xmm7
   0x00000000004007b6: mov QWORD PTR [rbp-0xd8],rdi
7 va_list arg;
8 int done;
9
10 va_start (arg, format);
   0x00000000004007bd: mov DWORD PTR [rbp-0xd0],0x8
   0x00000000004007c7: mov DWORD PTR [rbp-0xcc],0x30
   0x00000000004007d1: lea rax,[rbp+0x10]
   0x00000000004007d5: mov QWORD PTR [rbp-0xc8],rax
   0x00000000004007dc: lea rax,[rbp-0xb0]
   0x00000000004007e3: mov QWORD PTR [rbp-0xc0],rax
11 done = vfprintf (stdout, format, arg);
   0x00000000004007ea: mov rax,QWORD PTR [rip+0x20082f] #
0x601020 <stdout@@GLIBC_2.2.5>
   0x00000000004007f1: lea rdx,[rbp-0xd0]
   0x00000000004007f8: mov rcx,QWORD PTR [rbp-0xd8]
   0x00000000004007ff: mov rsi,rcx
   0x0000000000400802: mov rdi,rax
   0x0000000000400805: call 0x400670 <vfprintf@plt>
   0x000000000040080a: mov DWORD PTR [rbp-0xb4],eax
12 va_end (arg);
13
14 return done;
   0x0000000000400810: mov eax,DWORD PTR [rbp-0xb4]
15 }
   0x0000000000400816: leave
   0x0000000000400817: ret

I've added to this patch:

http://llvm-reviews.chandlerc.com/D960

Modified Symbol::GetPrologueByteSize() in source/Symbol/Symbol.cpp and
Function::GetPrologueByteSize() in source/Symbol/Function.cpp. They should
both now skip line entries until they find a line number that differs.

One question: would it make sense to have Symbol::GetPrologueByteSize()
check to see if it is an exact alias to a function and if so, use the
Function::GetPrologueByteSize() routine? That way we'd get the prologue end
data instead of just skipping lines?

In any case, breakpoint setting in the printf case works as it should now:

(lldb) b printf
Breakpoint 2: where = libc.so.6`__printf + 91 at printf.c:34, address =
0x00007f8dba8b189b
(lldb) b __printf
Breakpoint 3: where = libc.so.6`__printf + 91 at printf.c:34, address =
0x00007f8dba8b189b
(lldb) b _IO_printf
Breakpoint 4: where = libc.so.6`__printf + 91 at printf.c:34, address =
0x00007f8dba8b189b
(lldb) disassemble -n printf
libc.so.6`__printf at printf.c:30:
   0x7f8dba8b1840: subq $216, %rsp

libc.so.6`__printf + 7 at printf.c:30:
   0x7f8dba8b1847: testb %al, %al
   0x7f8dba8b1849: movq %rsi, 40(%rsp)
   0x7f8dba8b184e: movq %rdx, 48(%rsp)
   0x7f8dba8b1853: movq %rcx, 56(%rsp)
   0x7f8dba8b1858: movq %r8, 64(%rsp)
   0x7f8dba8b185d: movq %r9, 72(%rsp)
   0x7f8dba8b1862: je 0x7f8dba8b189b ; __printf + 91 at
printf.c:34
   0x7f8dba8b1864: movaps %xmm0, 80(%rsp)
   0x7f8dba8b1869: movaps %xmm1, 96(%rsp)
   0x7f8dba8b186e: movaps %xmm2, 112(%rsp)
   0x7f8dba8b1873: movaps %xmm3, 128(%rsp)
   0x7f8dba8b187b: movaps %xmm4, 144(%rsp)
   0x7f8dba8b1883: movaps %xmm5, 160(%rsp)
   0x7f8dba8b188b: movaps %xmm6, 176(%rsp)
   0x7f8dba8b1893: movaps %xmm7, 192(%rsp)
   0x7f8dba8b189b: leaq 224(%rsp), %rax
   0x7f8dba8b18a3: movq %rdi, %rsi
   0x7f8dba8b18a6: leaq 8(%rsp), %rdx
   0x7f8dba8b18ab: movl $8, 8(%rsp)
   0x7f8dba8b18b3: movl $48, 12(%rsp)
   0x7f8dba8b18bb: movq %rax, 16(%rsp)
   0x7f8dba8b18c0: leaq 32(%rsp), %rax
   0x7f8dba8b18c5: movq %rax, 24(%rsp)
   0x7f8dba8b18ca: movq 3556935(%rip), %rax
   0x7f8dba8b18d1: movq (%rax), %rdi
   0x7f8dba8b18d4: callq 0x7f8dba8a69c0 ;
_IO_vfprintf_internal at vfprintf.c:211
   0x7f8dba8b18d9: addq $216, %rsp
   0x7f8dba8b18e0: ret

I'd love to hear feedback on this patch and the split symbol patch overall.
I've built it on the Mac using cmake & ninja (won't run locally without
debugserver though?). I'm still having issues with xcodebuild randomly
failing to link on my Mac, and it seems to rebuild the whole thing when I
launch xcodebuild again which takes forever. (This Mac thing has been
pretty frustrating.) I've run the entire test suite on Linux, and it says
there are some new failure cases but when I run them individually (not
through ninja check-lldb) they work. So I'm finding it a bit difficult to
verify all this stuff. I'm also gone all next week, so if folks are worried
about this large patch it might make sense to wait until early July. I
would like to get it looked over and in at some point because the merging
is becoming more painful.

Thanks.
-Mike

Hi,

I'm reading the patch but only got through a small portion of it thus far. The interface changes look all right, to me at least. I'm not too familiar with generating split-file debug info, but I imagine it's possible to do so with some flags to GCC (if not Clang)? If so, can you add some tests that do that too? I'd love to see this functionality go in and stay unbroken :slight_smile:

In any case, it's probably best to have someone else with more object file experience review the patch too as my knowledge is somewhat limited in that dept. I'll post any comments I might come up with on phabricator.

Cheers,
Dan

Thanks for looking at this Daniel. I finally got the Mac stuff building and
running. I'm building lldb with cmake+ninja, and debugserver with
"xcodebuild -scheme debugserver", then I set LLDB_SERVER_PATH and
everything is working a lot more consistently for me. I also got the tests
to run and fixed a couple of Mac bugs. So the latest patch is updated here:

http://llvm-reviews.chandlerc.com/D1027

However when I try to run the full suite of Mac tests (via: ./dotest.py -i
--compiler `xcrun -find clang`), it successfully runs several and then
python crashes with the issue down below.

I'm off to Montana next week, so I'll try to get some test cases for this
split symbol support, and hopefully some other folks can verify some of
this and we can work to get it committed sometime in early July.

Have a great weekend.
-Mike

Process: Python [77522]
Path:
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 2.7.2 (2.7.2)
Code Type: X86-64 (Native)
Parent Process: bash [68645]
User ID: 501

PlugIn Path: /Users/USER/*/_lldb.so
PlugIn Identifier: _lldb.so
PlugIn Version: ??? (0)

Date/Time: 2013-06-21 16:13:05.685 -0700
OS Version: Mac OS X 10.8.4 (12E55)
Report Version: 10
Sleep/Wake UUID: 5773AD08-BB53-4028-9656-588ED4D88FB7

Interval Since Last Report: 200369 sec
Crashes Since Last Report: 2
Per-App Crashes Since Last Report: 2
Anonymous UUID: FA7AC80A-95B5-623E-D9BA-4A199914D12E

Crashed Thread: 5 <lldb.process.internal-state(pid=85810)>

Exception Type: EXC_BAD_ACCESS (SIGBUS)
Exception Codes: KERN_PROTECTION_FAILURE at 0x00000001185aa180

VM Regions Near 0x1185aa180:
    Stack 00000001185a8000-00000001185aa000 [ 8K]
rw-/rwx SM=COW
--> STACK GUARD 00000001185aa000-00000001185ab000 [ 4K]
---/rwx SM=NUL stack guard for thread 5
    Stack 00000001185ab000-000000011862d000 [ 520K]
rw-/rwx SM=COW thread 5

Application Specific Information:
HandleCommand(command = "process attach -p 85810")

Thread 0:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff870af0fa __psynch_cvwait + 10
1 libsystem_c.dylib 0x00007fff81826fe9 _pthread_cond_wait +
869
2 _lldb.so 0x00000001042b5182
lldb_private::Condition::Wait(lldb_private::Mutex&, lldb_private::TimeValue
const*, bool*) + 146 (Condition.cpp:92)
3 _lldb.so 0x00000001042c566d
lldb_private::Predicate<bool>::WaitForValueEqualTo(bool,
lldb_private::TimeValue const*, bool*) + 205 (Predicate.h:338)
4 _lldb.so 0x0000000104366b05
lldb_private::Listener::WaitForEventsInternal(lldb_private::TimeValue
const*, lldb_private::Broadcaster*, lldb_private::ConstString const*,
unsigned int, unsigned int, std::__1::shared_ptr<lldb_private::Event>&) +
805 (Listener.cpp:433)
5 _lldb.so 0x0000000104366dec
lldb_private::Listener::WaitForEventForBroadcasterWithType(lldb_private::TimeValue
const*, lldb_private::Broadcaster*, unsigned int,
std::__1::shared_ptr<lldb_private::Event>&) + 92 (Listener.cpp:464)
6 _lldb.so 0x0000000104670d44
lldb_private::Process::WaitForStateChangedEvents(lldb_private::TimeValue
const*, std::__1::shared_ptr<lldb_private::Event>&) + 132 (Process.cpp:1336)
7 _lldb.so 0x00000001046709a1
lldb_private::Process::WaitForProcessToStop(lldb_private::TimeValue const*,
std::__1::shared_ptr<lldb_private::Event>*) + 577 (Process.cpp:1241)
8 _lldb.so 0x00000001041bdb29
CommandObjectProcessAttach::DoExecute(lldb_private::Args&,
lldb_private::CommandReturnObject&) + 2185 (CommandObjectProcess.cpp:634)
9 _lldb.so 0x0000000104506190
lldb_private::CommandObjectParsed::Execute(char const*,
lldb_private::CommandReturnObject&) + 528 (CommandObject.cpp:1038)
10 _lldb.so 0x00000001044e38a6
lldb_private::CommandInterpreter::HandleCommand(char const*,
lldb_private::LazyBool, lldb_private::CommandReturnObject&,
lldb_private::ExecutionContext*, bool, bool) + 20134
(CommandInterpreter.cpp:1828)
11 _lldb.so 0x00000001040a5684
lldb::SBCommandInterpreter::HandleCommand(char const*,
lldb::SBCommandReturnObject&, bool) + 292 (SBCommandInterpreter.cpp:123)
12 _lldb.so 0x00000001040980bf
_wrap_SBCommandInterpreter_HandleCommand__SWIG_0 + 831
(LLDBWrapPython.cpp:9296)
13 _lldb.so 0x0000000103ffaa0b
_wrap_SBCommandInterpreter_HandleCommand + 923 (LLDBWrapPython.cpp:9404)
14 org.python.python 0x0000000103708f72 PyEval_EvalFrameEx +
3557
15 org.python.python 0x0000000103708147 PyEval_EvalCodeEx +
1934
16 org.python.python 0x000000010370e8df 0x1036f1000 + 121055
17 org.python.python 0x000000010370a63a PyEval_EvalFrameEx +
9389
18 org.python.python 0x0000000103708147 PyEval_EvalCodeEx +
1934

I added standalone debug support to FreeBSD's base system build last
year. Assuming you have an executable 'exe.full' containing debug
data, it's basically just:

objcopy --strip-debug --add-gnu-debuglink=exe.debug exe.full exe
objcopy --only-keep-debug exe.full exe.debug

(I'm not sure if any linkers support creating standalone debug
directly, but ld from FreeBSD's rather-old binutils doesn't.)

Ah, cool, thanks Ed!

I can confirm those commands (invoked in reverse order) work for me on
Ubuntu 12.10. So it should be possible to do some Makefile-magic with
objcopy to test the split-debug implementation in an automatic fashion.

I had a chance to try Mike's patch (D960) and can confirm that it works as
advertised on binaries that are split with the objcopy approach.

Cheers,
Dan

Just pinging. Does anyone have any thoughts on whether this makes sense
look at?

Thanks.
-Mike

That sounds fine to me. Since the symbol's start address is already in section/offset format, it will be a really quick lookup within the module itself to resolve the function.

Greg

Something like the below look ok?

Thanks Greg.
-Mike

uint32_t
Symbol::GetPrologueByteSize ()
{
    ...
            const Address &base_address = m_addr_range.GetBaseAddress();
            Function *function =
base_address.CalculateSymbolContextFunction();
            if (function)
            {
                // Functions have line entries which can also potentially
have end of prologue information.
                // So if this symbol points to a function, use the prologue
information from there.
                m_type_data = function->GetPrologueByteSize();
            }
            else
            {
                ... do regular symbol prologue calculations ...