I’ve answered questions like Debugging LLDB using LLDB a few times, and have had colleagues ask me the same. So I think it’s time we had some documentation on the subject.
Even if it can be summarised as “yes, you can debug a debugger with a debugger” It’s sufficiently weird it’s worth writing down I think.
So I want to crowd source/be lazy about what should be in it. Tell me if there’s any tips you know or wish you knew about debugging lldb.
My starting list:
Building lldb with debug information.
That one cmake setting that exports all lldb symbols that I forget the name of.
The basic running lldb under lldb (or gdb).
Setting the prompt so you can tell which one is which.
How to tell when you’ve returned to the debugee lldb (since it won’t print a fresh prompt when you resume).
How lldb-server platform spawns gdb mode instances and the implications for debugging.
How to debug lldb-server in gdb mode, remotely or otherwise.
How to use follow process to debug a gdb instance spawned from a platform (which I think is possible, but I am yet to try it).
I’ll take that and your suggestions and write up a new docs page for folks to review.
fwiw I personally prefer to run the debugee-lldb in one window, and then attach to it from lldb in a separate window, so the two lldb’s aren’t fighting for console input and I don’t have to change the prompts to disambiguate which one I’m typing in to.
I guess the one case where I often run lldb on lldb is when I’m looking to catch a crash, and not interacting with the top lldb intentionally before that. I still have the problem if I have top-lldb > lldb > a.out and I press control-c, top-lldb will see it first and give me the top-lldb prompt.
I haven’t touched lldb code in quite some time, but I think I remember some of the tests were written in python. Maybe a few words on debugging those would be helpful.
One thing that I’ve been struggling with lately - how do I debug lldb using lldb-vscode, and type in commands? I can blast in commands with -o in the args property in launch.json, but once I’m running I can’t figure out how to interactively type in commands.
In general, I think our vscode solution could use more documentation and attention. All we’ve really got is @vadimcn 's CodeLLDB on the marketplace. It’s a good solution, but we should do more with lldb-vscode.
Most of our tests are written in Python actually. We have a little snippet on the website but I agree there’s room for improvement!
Do folks run their just-built lldb from the build dir? A few years ago I considered having CMake generate a .lldbinit in the build root with to change the inferior LLDB’s prompt.
+1, especially when running under tmux, this can avoid editline getting confused.
Cool, didn’t know there was prior art. The “security” feature is the reason I didn’t pursue it further. But if folks think that’s useful, we could special case the build dir in a debug/assert build or with a CMake variable.
I’m a big fan of this. I would also like to contribute to this cause with what I can.
In addition to the list you’ve provided, some other things I think would be useful:
General lldb tips
How to use some more advanced breakpoint features (e.g. conditional breakpoints)
How to enable lldb logging and how to redirect it to a file for later analysis
Somewhat related: It might be useful to create an “intro to gdb-remote protocol”. We have a nice list of each packet and what it is supposed to do, but no intro to the subject itself.
How to manipulate the environment variables of the inferior (I’ve seen some confusion from end users on this actually)
How to use apropos to find commands or settings that are possibly relevant
A more complete introduction to lldb’s python integration
How to use it, how to write new things, etc. What we have today is a useful introduction, but I tried to use it the other day and found it rather terse.
How our swig binding code is laid out (I’ve had to learn this the hard way a few times, documentation would have been nice).
Strategies for alternate ways of accomplishing common tasks. For example, if you try to evaluate an expression but it fails, what are some alternate ways you could attempt to answer the question that the expression evaluation would have answered?
I’ve been thinking about improving documentation for a while, I’d also like to contribute these semi-related ideas:
A guide on automating lldb workflows. More specifically, how to combine the lldb command line flags and python integration to automatically perform certain tasks on things like corefiles and bad binaries.
A more beginner-friendly way to tell users about certain aspects of their debugging session. For example, how do I know if I lldb found my debug info? How do I perform source remappings? etc.
+1 on this. My experience debugging lldb with itself using 1 terminal is fraught with confusion and frustration. Attaching to an existing lldb instance in a separate terminal makes it extremely clear what is happening where.
Perhaps @clayborg or some of his colleagues from Meta can better help you with this. IIRC they are using lldb-vscode quite extensively.
Worth mentioning that there are some data formatters for lldb data types (in lldb/examples/summaries/lldb). We should also extend those and if possible set up some testing for them so they don’t bit-rot. The situation isn’t as bad as llvm/clang, lldb has few data structures that you can’t just look at. But OTOH, having a Module * show its name w/o having to drill down into the structure is such a time saver.
This are all good ideas, sounds like we have got 2 things:
How to debug lldb using lldb.
How to make the best use of lldb, regardless of debugee (but of course including lldb itself).
Between GDB’s documentation and our scattered text files the information is there but I agree it could be put together.
Also it would be a good place to state (/restate, it’s probably somewhere on the website) our goals with regards to compatibility with combinations of lldb/gdb and lldb-server/gdbserver and other “compatible” stubs.
It would be nice if LLDB had more material towards formatter writers. For instance, today I’m using script print(...) to debug various expressions I’d like to put into a formatter. I think it would be easier to have a Python debugger inside formatters if my whole LLDB session was Python-scripted. I tried, but resorted to print-style debugging in the end.
One from me is how to reduce crashes involving lldb and Linux ptrace down into small examples that don’t need lldb. For example when you find a kernel crash.
lldb’s instance of the Python interpreter is a pretty vanilla interpreter, so it does all the things a normal Python interpreter does, including if you put breakpoint() in the code of a Python formatter, it will break into the python debugger when the formatter is executing, and you can use all the pdb commands to step through the python, inspect variables, etc.
That’s obvious in hindsight, but a little magical and it’s not obvious you would expect it to work. So we should indeed include that in the documentation. This doesn’t just work for formatters, but for all the python affordances in lldb.
I think another good documentation add would be how to use pdb to debug API tests. I generally resort to adding in print() or self.runCmd() calls do get data out, but that can become very cumbersome.
Indeed, though again, the message is "anywhere that lldb is running python code, pdb (or really whatever python debugger you have installed) should interrupt that python code execution and give you control. Putting breakpoint() anywhere in the test file will stop execution.
I use this all the time, not just for python debugging. For instance, it’s often the case that I want to debug lldb when it’s performing a particular line in the test file, but there’s lots of code before that that is also going to hit the breakpoints I need to debug the line I want to. So I just put breakpoint() in the test before that line, run:
lldb-dotest -p <TestFile.py> -f <test_case> -d
Then attach with lldb, disable all my breakpoints, and continue. Then when I hit the python breakpoint in the test, I reenable all my breakpoints in lldb, run continue at the pdb prompt and I’m all set to debug the line I wanted to…
I think another good documentation add would be how to use pdb to debug API tests. I generally resort to adding in print() or self.runCmd() calls do get data out, but that can become very cumbersome.
Anything to do with debugserver I have no idea, so if someone wants to add that on top that would be great. I’ve just said the architecture is mostly the same.
I’ll go through the rest of the answers/requests and what I’m familiar with I’ll put up patches for, what I’m not, I’ll politely request an expert to document it for me
It’s sufficiently weird it’s worth writing down I think.