Debugging a running process with lldb

Matthew_Gardiner · May 14, 2014, 9:09am

Hi folks

On reading the documentation for lldb am I correct in understanding that for a running inferior program being debugged by lldb, that the input reader, i.e. the (lldb) prompt should be active at all times?

For instance, from "http://lldb.llvm.org/tutorial.html":

"The commands that currently work while running include interrupting the process to halt execution ("process interrupt"), getting the process status ("process status"), breakpoint setting and clearing (" breakpoint [set|clear|enable|disable|list] ..."), and memory reading and writing (" memory [read|write] ...")."

I interpret this to mean whenever debugging a program regardless of the run/stop state of any of it's threads, lldb must always print:

(lldb)

after processing the last command, since we require an active prompt in order to process the issuing of the above commands (from the lldb user) when the inferior is running.

In that case I have found a bug on my 64-bit linux build. I firstly launch lldb to debug a simple inferior (64-bit linux process), stop it on entry, then resume it. The (lldb) prompt does not reappear after I have resumed:

$ lldb ./forever
Traceback (most recent call last):
   File "<string>", line 1, in <module>
ImportError: No module named lldb.embedded_interpreter
Current executable set to './forever' (x86_64).
(lldb) process launch -s
Process 1901 launching
Process 1901 launched: './forever' (x86_64)
Process 1901 stopped
* thread #1: tid = 1901, 0x0000003675a011f0, name = 'forever', stop reason = trace
     frame #0: 0x0000003675a011f0
-> 0x3675a011f0: movq %rsp, %rdi
    0x3675a011f3: callq 0x3675a046e0
    0x3675a011f8: movq %rax, %r12
    0x3675a011fb: movl 0x21eb97(%rip), %eax
(lldb) process continue
Process 1901 resuming

Are other people seeing this bug?

thank you
Matthew Gardiner

Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Keep up to date with CSR on our technical blog, www.csr.com/blog, CSR people blog, www.csr.com/people, YouTube, www.youtube.com/user/CSRplc, Facebook, www.facebook.com/pages/CSR/191038434253534, or follow us on Twitter at www.twitter.com/CSR_plc.
New for 2014, you can now access the wide range of products powered by aptX at www.aptx.com.

Matthew_Gardiner · May 14, 2014, 10:22am

More thoughts:

I've read http://lldb.llvm.org/tutorial.html in more depth now (appreciating that the debugger and the inferior require separate stdins.)
I see there's an option, "--no-stdin", proposing to mitigate the issue I found

"If you attach to a process, or launch a process with the "|--no-stdin|" option, the command interpreter is always available to enter commands. This might be a little disconcerting to gdb users when always have an

(lldb)| prompt. This allows you to set a breakpoint, etc without having

to explicitly interrupt the program you are debugging:

(lldb) process continue

(lldb) breakpoint set --name stop_here |
"

However, it seems that --no-stdin is not present as a "process launch" option anymore, but rather we have "--no-stdio". (Perhaps a typo, somewhere?). I tried this option:

(lldb) process launch -s --no-stdio
Process 2344 launching
Process 2344 launched: './forever' (x86_64)
Process 2344 stopped
* thread #1: tid = 2344, 0x0000003675a011f0, name = 'forever', stop reason = trace
     frame #0: 0x0000003675a011f0
-> 0x3675a011f0: movq %rsp, %rdi
    0x3675a011f3: callq 0x3675a046e0
    0x3675a011f8: movq %rax, %r12
    0x3675a011fb: movl 0x21eb97(%rip), %eax
(lldb) process continue
Process 2344 resuming

The interpreter prompt still fails to appear, so I still think I'm either seeing a bug, or that the documentation is incomplete as to describing how to run lldb such that the command interpreter, is always available when the inferior is running.

thanks
Matt

Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Keep up to date with CSR on our technical blog, www.csr.com/blog, CSR people blog, www.csr.com/people, YouTube, www.youtube.com/user/CSRplc, Facebook, www.facebook.com/pages/CSR/191038434253534, or follow us on Twitter at www.twitter.com/CSR_plc.
New for 2014, you can now access the wide range of products powered by aptX at www.aptx.com.

Greg_Clayton1 · May 14, 2014, 5:17pm

If we launch a process and hookup a pty to its stdin/out/err, then we shouldn't get a prompt. It is probably a bug where if you launch with --no-stdio that the LLDB prompt isn't always active.

For attach, we will always have a prompt.

So seems this is a bug that --no-stdio is still pushing a Process IOHandler and taking over the console while the process is running.

jingham · May 14, 2014, 5:56pm

Note that on OS X with the current TOT lldb, "process launch --no-stdin" works as advertised.

Jim

Matthew_Gardiner · May 15, 2014, 5:29am

Greg Clayton wrote:

If we launch a process and hookup a pty to its stdin/out/err, then we shouldn't get a prompt.

Sorry, I don't understand this. Surely if we attach the inferior's (i.e. what you termed the process) standard IOs to a pty, then we *will* get a prompt, since then lldb can use it's terminal with no corruption from the inferiors in/out?

(I'm not actually clear about the relationship between the "process launch" -t and -n commands. I do accept that they are mutually exclusive. I assume, -n means that the stdin/out/err of the inferior is closed, so that lldb has complete use of the foreground, so it could use it's prompt. I assume -t means attach the inferior to a different pty, so again, lldb should still be able to use it's prompt.)

It is probably a bug where if you launch with --no-stdio that the LLDB prompt isn't always active.

For attach, we will always have a prompt.

So seems this is a bug that --no-stdio is still pushing a Process IOHandler and taking over the console while the process is running.

So you are saying if I do
(lldb) process launch -s --no-stdio
then
(lldb) process continue

then I should always see:
(lldb)
back on my terminal?

Todd, could you try this (a couple of times, in case I'm seeing something intermittent) please, to see if your observations agree?

thanks
Matt

Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Keep up to date with CSR on our technical blog, www.csr.com/blog, CSR people blog, www.csr.com/people, YouTube, www.youtube.com/user/CSRplc, Facebook, www.facebook.com/pages/CSR/191038434253534, or follow us on Twitter at www.twitter.com/CSR_plc.
New for 2014, you can now access the wide range of products powered by aptX at www.aptx.com.

Matthew_Gardiner · May 15, 2014, 5:33am

jingham@apple.com wrote:

Note that on OS X with the current TOT lldb, "process launch --no-stdin" works as advertised.

Sorry to sound pedantic but on my build, when I invoke "(lldb) help process launch", I see

-n ( --no-stdio )
Do not set up for terminal I/O to go to running process.

not --no-stdin.

Matt

Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Keep up to date with CSR on our technical blog, www.csr.com/blog, CSR people blog, www.csr.com/people, YouTube, www.youtube.com/user/CSRplc, Facebook, www.facebook.com/pages/CSR/191038434253534, or follow us on Twitter at www.twitter.com/CSR_plc.
New for 2014, you can now access the wide range of products powered by aptX at www.aptx.com.

Matthew_Gardiner · May 15, 2014, 8:53am

Matthew Gardiner wrote:

Greg Clayton wrote:

If we launch a process and hookup a pty to its stdin/out/err, then we shouldn't get a prompt.

Sorry, I don't understand this. Surely if we attach the inferior's (i.e. what you termed the process) standard IOs to a pty, then we *will* get a prompt, since then lldb can use it's terminal with no corruption from the inferiors in/out?

(I'm not actually clear about the relationship between the "process launch" -t and -n commands. I do accept that they are mutually exclusive. I assume, -n means that the stdin/out/err of the inferior is closed, so that lldb has complete use of the foreground, so it could use it's prompt. I assume -t means attach the inferior to a different pty, so again, lldb should still be able to use it's prompt.)

It is probably a bug where if you launch with --no-stdio that the LLDB prompt isn't always active.

For attach, we will always have a prompt.

So seems this is a bug that --no-stdio is still pushing a Process IOHandler and taking over the console while the process is running.

So you are saying if I do
(lldb) process launch -s --no-stdio
then
(lldb) process continue

then I should always see:
(lldb)
back on my terminal?

Todd, could you try this (a couple of times, in case I'm seeing something intermittent) please, to see if your observations agree?

thanks
Matt

Hi Greg,

Actually, my colleague just brought his MacBook (OSX 10.9.2) into work, this morning. He installed pre-built xcode/lldb (lldb-310.2.37) on his Mac, and I can confirm that on mac, when we invoked:

(lldb) process launch -s -n
Process 1002 lanched....
(lldb) process continue
Process 1002 resuming
(lldb)

then we see the interpreter prompt return after the "continue". So what I reported originally is a linux build bug. (Perhaps I'll start digging into this, if I get any time...).

However, using "process launch -s -t" on the mac, was buggy, a terminal-emulator was spawned, but (presumably) lldb and the child terminal fail to synchronise using the UNIX socket.

(lldb) process launch -s -t
error: failed to launch or debug process
(lldb)

Anyway, sorry for my earlier rant ;-), but at least now I'm a little clearer about the intended semantics...

Matt

Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Keep up to date with CSR on our technical blog, www.csr.com/blog, CSR people blog, www.csr.com/people, YouTube, www.youtube.com/user/CSRplc, Facebook, www.facebook.com/pages/CSR/191038434253534, or follow us on Twitter at www.twitter.com/CSR_plc.
New for 2014, you can now access the wide range of products powered by aptX at www.aptx.com.

Greg_Clayton1 · May 15, 2014, 5:21pm

So to clear things up:

-t or --tty will attempt to launch the process in a _separate_terminal_window_ (open a new terminal window and launch your process within it so the process gets its own window and we don't have to shared IO) and should never be used with -n (--no-stdio). If they currently can be run together then this is a bug. --tty isn't supported on Linux as far as I know and the linux launch code should return an error saying this isn't supported. The --tty option is designed to be used when you want to debug anything that uses curses (emacs, vi, etc) and where having a different terminal window with its own stdin/out/err is required.

--no-stdio should, as you surmised, launch a process with /dev/null as the stdin/out/err. It also means we should always have a prompt.

To to summarize:

(lldb) process launch ...

No LLDB prompt while running as all input goes to process (though if CTRL+C is pressed and we will stop the target for you)

(lldb) process launch --tty -- ...

LLDB prompt always visible, process launches in a different terminal window, not supported on linux, should never be used with --no-stdio and if our options allow it, this is a bug

(lldb) process launch --no-stdio -- ...

LLDB prompt always visible (if not, then a bug, probably only on the linux side).

Let me know if I

Matthew_Gardiner · May 16, 2014, 5:01am

Greg Clayton wrote:

So to clear things up:

-t or --tty will attempt to launch the process in a _separate_terminal_window_ (open a new terminal window and launch your process within it so the process gets its own window and we don't have to shared IO) and should never be used with -n (--no-stdio). If they currently can be run together then this is a bug. --tty isn't supported on Linux as far as I know and the linux launch code should return an error saying this isn't supported. The --tty option is designed to be used when you want to debug anything that uses curses (emacs, vi, etc) and where having a different terminal window with its own stdin/out/err is required.

--no-stdio should, as you surmised, launch a process with /dev/null as the stdin/out/err. It also means we should always have a prompt.

To to summarize:

(lldb) process launch ...

No LLDB prompt while running as all input goes to process (though if CTRL+C is pressed and we will stop the target for you)

(lldb) process launch --tty -- ...

LLDB prompt always visible, process launches in a different terminal window, not supported on linux, should never be used with --no-stdio and if our options allow it, this is a bug

(lldb) process launch --no-stdio -- ...

LLDB prompt always visible (if not, then a bug, probably only on the linux side).

Thanks for confirming this.
I might have a look at fixing the linux bug in a few weeks if the rest of my work permits.
Matt

Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Keep up to date with CSR on our technical blog, www.csr.com/blog, CSR people blog, www.csr.com/people, YouTube, www.youtube.com/user/CSRplc, Facebook, www.facebook.com/pages/CSR/191038434253534, or follow us on Twitter at www.twitter.com/CSR_plc.
New for 2014, you can now access the wide range of products powered by aptX at www.aptx.com.