inability to set breakpoints in lldb

i have been investigating several lldb bugs that seem to be related.

bug 19974 describes lldb’s inability to set a breakpoint in an included file and bug 20297 describes lldb’s inability to set a breakpoint when using ccache. i have also found a similar case where lldb is unable to set a breakpoint when a separate preprocess step has been used before compiling.

note: ccache uses a preprocess step to determine which source files need to be rebuilt so using ccache could be considered the same as using a separate preprocess step.

all three cases are similar in that the c/c++ preprocessor is involved and alters the way symbols are added to the object files.

the following attached files are write-ups of some of my findings for the different cases
ccache - “lldb and ccache.rtf”
preprocessed files - “lldb and preprocessed files.rtf”
include files - “lldb and include files.rtf”

all three cases generate the inability to set breakpoints

on the Mac, all three cases create object files with “SOL” symbols for preprocessed or included source files.

the following attached file describes the object files created for the preprocessed test case on the Mac:
“preprocessed object files.rtf”

on ubuntu, it is not as obvious to spot differences in the object files when examining object files that that lldb is unable to set breakpoints.

lldb is unable to set breakpoints on both OSX and ubuntu for all three cases. gdb is able to set breakpoints on both OSX and ubuntu for all three cases.

using XCode, i have tried a couple different ways of stepping through lldb to see why it is having trouble setting breakpoints.

From top-down, i have traced setting breakpoints in Breakpoint.cpp down thru BreakpointResolver.cpp. The intended source file does not seem to be in the list of compute units, but i don’t know why.

From bottom-up, i have tried to follow how ObjectFileMachO.cpp parses symbols, but i’m not sure that this is the correct place to look. Also, since this is both a MachO problem and an Elf problem, the problem isn’t likely to be here anyway.

Does anyone familiar with lldb’s architecture have any ideas where in the code i should be looking or have a plan-of-attack to suggest?

thanks,
doug

lldb and ccache.rtf (5.37 KB)

lldb and preprocessed files.rtf (4.6 KB)

preprocessed object files.rtf (9.24 KB)

lldb and include files.rtf (2.13 KB)

Are any of these bugs fixed by setting:

settings set target.inline-breakpoint-strategy always

in your .lldbinit file?

Jim

Hey Doug,

Can you give Jim’s suggestin a try and see if that affects anything?

Thanks,
Todd

Or should I say “suggesti’n?” Er, hmm… Suggestion :wink:

adding that setting to .lldbinit appears to fix all three cases (ccache, preprocess and include). this was tested in xcode with individual test cases that i created.

i wonder why that setting is not on by default?

Because it vastly improves performance for large Objective C and C++ projects when setting file and line breakpoints. See our troubleshooting page for more details:

http://lldb.llvm.org/troubleshooting.html

what if the logic was changed to do what it does now, but if the “set breakpoint” returns 0 locations, then try again with target.inline-breakpoint-strategy set?

adding that setting to .lldbinit appears to fix all three cases (ccache, preprocess and include).

I want to say “Nice!”, except for this part…

Because it vastly improves performance for large Objective C and C++ projects when setting file and line breakpoints. See our troubleshooting page for more details:

Do you guys have any numbers on what kind of hit this creates? Doug’s analysis seemed to indicate there were links we could follow that might eliminate the need to search everywhere, and perhaps narrow a search for a symbol to just one of a couple places? So maybe there’s some in-between ground here where we don’t have to search everywhere for a symbol?

(I’m under the impression from what I read above that if we turn on this option, it means “always search all modules for symbol/breakpoint resolution”?)

actually the strategy i suggested would always be slow if ccache was used or if someone employed a unity build (large number of .cpp files included in a single .cpp file in order to reduce link times). “set breakpoint” would always return 0 locations until target.inline-breakpoint-strategy was set.

doug

it appears that there are a number of possible options for addressing lldb’s inability to set breakpoints when using ccache, with preprocess files and with include files

  1. continue using “settings set target.inline-breakpoint-strategy always” to get around the problem. however, many/most users aren’t aware of this setting and aren’t aware that they can get around the problem. also, the name doesn’t imply that this setting will fix the problem for ccache or for preprocessed files
  2. improve the search speed so it’s feasible to search all include files when working with large projects. this could eliminate the need for “settings set target.inline-breakpoint-strategy always”. i don’t know how involved this effort would be.
  3. add a two-stage file search if the “set breakpoint” initially finds 0 locations. i have included a possible patch for this with a description below
  4. add .c/.cpp/.m include files to the list of files to search. (don’t add .h files). this would fix the problem with ccache and preprocessed files. this would also make unit builds work. “settings set target.inline-breakpoint-strategy always” would still be needed for inline routines in .h files. i don’t know how well this fits in with the current system of searching modules and compile_units.
  5. allow different platforms to select how they wish to handle this. the best strategy for xcode may be different than the best strategy for android.
  6. some combination of options above

two-stage file search

target.cpp.diff (1.86 KB)

include.zip (1.83 KB)

preprocess.zip (1.82 KB)

I would vote to switch over to using "always" as the default setting and then let people who run into performance problems set it to "headers"

I am not fond of the two state approach because you might be trying to set a breakpoint a shared library that isn't loaded yet.

More comments below:

it appears that there are a number of possible options for addressing lldb's inability to set breakpoints when using ccache, with preprocess files and with include files
    1) continue using "settings set target.inline-breakpoint-strategy always" to get around the problem. however, many/most users aren't aware of this setting and aren't aware that they can get around the problem. also, the name doesn't imply that this setting will fix the problem for ccache or for preprocessed files

We should just make it the default. I don't like that we miss breakpoints for no good reason and rely upon an obscure setting to fix this.

    2) improve the search speed so it's feasible to search all include files when working with large projects. this could eliminate the need for "settings set target.inline-breakpoint-strategy always". i don't know how involved this effort would be.

Search speed isn't the problem, it is the number of files that get touched on disk. There probably won't be any performance issues on other systems until fission becomes real, but at Apple we leave the debug info in the .o files and when debugging large projects we end up having to open thousands of .o files. Is is the file cache getting warmed up that causes the performance problems.

    3) add a two-stage file search if the "set breakpoint" initially finds 0 locations. i have included a possible patch for this with a description below

I don't like this because if you have a shared library that isn't loaded yet, you will get no matches, then try again with inlines. It also doesn't help catch actual inlined locations in the case where you might have an inlined function and an actual concrete instance when the inlining depth gets exceeded.

    4) add .c/.cpp/.m include files to the list of files to search. (don't add .h files). this would fix the problem with ccache and preprocessed files. this would also make unit builds work. "settings set target.inline-breakpoint-strategy always" would still be needed for inline routines in .h files. i don't know how well this fits in with the current system of searching modules and compile_units.
    5) allow different platforms to select how they wish to handle this. the best strategy for xcode may be different than the best strategy for android.
    6) some combination of options above

I just suggest changing the default to "always" and we can have people that have performance problems change their setting to get the performance back.

Greg

here is a patch that sets the default to eInlineBreakpointsAlways. it also changes the associated comment text, since the old text was eInlineBreakpointsHeaders-default-centric

i manually tested it with lldb using one of my test cases and ran check-lldb on ubuntu to make sure it wasn’t breaking other things

I think the patch might be missing from that mail. Here it is though:

InlineBreakpointsAlways.patch (2.62 KB)

The text is wrong in the help text. It should't mention "eInlineBreakpointsHeaders", but it should mention "headers" (the actual text you would type for the settings set command). Other than that, it looks good.

i’ll fix that and create a new patch

Just replace "eInlineBreakpointsHeaders" with "headers" and you are good to go.

Doug - I’ll get this in.

This change went up here, with a couple tweaks to documentation (per Greg and another minor grammatical fix):

svn commit
Sending source/Target/Target.cpp
Transmitting file data .
Committed revision 218405.