Remote debugging a docker process

Subprojects LLDB
1

Greetings and salutations!

I am trying to remotely debug a process running inside of a Docker container. I can connect to lldb-server from my host, but can't launch a debugging process. I can debug the target locally, inside or outside of the container.

Container:

lldb-server-4.0 platform --verbose --listen "*:5000"
Connection established.

Host:

$ lldb

(lldb) target create target/debug/hist

(lldb) platform connect connect://localhost:5000
Platform: remote-linux
Triple: x86_64-pc-linux-gnu
OS Version: 4.15.0 (4.15.0-45-generic)
Kernel: #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019
Hostname: 4ce058c8dba3
Connected: yes
WorkingDir: /seraphim
(lldb) run
error: connect remote failed (Connection refused)
error: process launch failed: Connection refused

Docker is a containerization system that sandboxes the processes it manages in various ways. Processes inside of the container are running on a virtualized network stack and do not know the IP address of their host and cannot communicate to the outside except on "published" ports, for instance.

A mail [1] on this list dating to 2017 suggested the problem is that the gdbserver child process spawned by `process launch` can't talk to Docker over the firewall. However, I don't believe it's the issue - or at least, not the only one.

I isolated this problem by binding 5000 and 5001 in the container to the same values in the host. Then, I restricted the acceptable range of gdbserver ports to just 5001, using the flags suggested in the email.

lldb-server-4.0 platform --verbose --listen "*:5000" --min-gdbserver-port 5001 --max-gdbserver-port 5001

This had no apparent effect.

I also found a pull request [2] from 2018 which suggested that the problem is the virtualized IP address in the container. That is a promising direction, but unfortunately, that patch was abandoned and nothing took its place.

There are traces of this issue all over the net, but none of them that I have found were ever resolved. I think that remote-debugging a Docker container is an increasingly important use-case for lldb-remote, and if anyone is interested in this, I'm happy to work with you to hammer it out.

Unanswered Stack Overflow [3]

A bug filed on Swift's tracker [4]

[1] http://lists.llvm.org/pipermail/lldb-dev/2017-February/012004.html

[2] https://reviews.llvm.org/D42845

[3] linux - Remote LLDB debugging - Docker container - Stack Overflow

[4] https://bugs.swift.org/browse/SR-3596?attachmentViewMode=list

2

Then, I restricted the acceptable range of
gdbserver ports to just 5001, using the flags suggested in the email.

lldb-server-4.0 platform --verbose --listen "*:5000" --min-gdbserver-port 5001 --max-gdbserver-port 5001

                                                                                                      ^^^^
There should be 5002 as --max-gdbserver-port is in reality "max+1" value and
not "max" value.

The options --min-gdbserver-port and --max-gdbserver-port get ignored this way.

Submitted:
  Sanity check --max-gdbserver-port
  ⚙ D58962 Sanity check --max-gdbserver-port

Replied with a working Docker example:
  Remote LLDB debugging - Docker container
  linux - Remote LLDB debugging - Docker container - Stack Overflow

# Add an option 'allow-all-hosts' to permit lldb debugging inside a Docker container

[2] ⚙ D42845 Add an option 'allow-all-hosts' to permit lldb debugging inside a Docker container

I also haven't found a need for this new proposed option.

Jan

3

Thank you, Jan. --gdbserver-port was the flag I needed.

I also found that lldb-4.0 (packaged with Ubuntu 16.04) had another error with remote debugging that exhibited the same symptom. Upgrading lldb and setting --gdbserver-port were both required to remote debug my docker container. But now it works beautifully!