How does attach work on non-Windows?

Subprojects LLDB
1

On Windows, when we attach to process, we basically invoke a system call which tells the kernel to kick off the process necessary for a debugger to be able to communicate with the process.

The end result of all this is that eventually the OS itself will generate a breakpoint in the inferior by injecting an additional thread into the inferior and breaking on that thread.

LLDB picks this up, and the result is that LLDB stops and waits for the user to continue the inferior just as it would with any other breakpoint, and if you were to get a backtrace you might see something like this:

looking at: Stack traces for SBProcess: pid = 12588, state = stopped, threads = 2, executable = test_with_dwarf_and_attach_to_process_with_id_api
Stack trace for thread id=0x3428 name=None queue=None stop reason=none
frame #0: 0xffffffffffffffff ntdll.dll`DbgBreakPoint + 1

Stack trace for thread id=0x4314 name=None queue=None stop reason=none
frame #0: 0x00000077908c2c None`None + -18446744071703589843

My question is: Do other platforms have this concept of an OS-generated breakpoint? Or when you attach to the process, would the first breakpoint opcode encountered by the inferior be one which was created by the user through some command from the debugger?

This creates a problem for some of our tests, because we have this extra breakpoint that is messing up the stack frame expectations unless we issue a manual continue first to get past the initial breakpoitn and to the first user breakpoint.

2

Slightly related, but do other platforms have a way to check from an inferior if a debugger is present?

We need to do this frequently from the test inferiors, and I see lots of different approaches used in the test programs, none of which work correctly on Windows.

3

On OS X and most Unixes, you call ptrace(pid, PT_ATTACH) or some procfs call to initiate the attach, and then wait on the pid until it comes back with a SIGSTOP signal when the attach is completed. How the wait is done differs from system to system, but this is the general model. I have not heard of a UNIX that uses actual trap instructions placed by the kernel in the process to stop it.

Jim

4

There is a way on OS X. There is a sysctl that will give you information on the current process state, and one of the bits you get back says whether the process is being traced. sysctl's are a generic UNIX thing, but I don't know if the bit OS X uses is shared with other Unix's.

Jim

5

Assuming we can find a reasonable way to detect this on all platforms, can I replace current wait-for-debugger-attach code in the test inferiors to use this method? It’s all very racy right now, and there are combinations of sleeps and loops in the executables sometimes working together with sleeps in the test cases to synchronize the test and the executable. If we had a common method that allowed inferiors to just say “wait until some debugger is attached to me” I think some of our problems would go away.

Would you mind posting a code snippet for how to do this on OS X so someone familiar with FreeBSD and/or Linux can comment on if there’s a similar one for their platform?

6

I seem to recall on the Linux side that this gets a bit complicated. Attaching vs. launching get different sequences of stops, as do the number of hops through possible shells in between a launch and an attach. So we pass through a skip-stop count (which I think both OS X and Linux use) to figure out how many of them we need to skip before we say we’re attached.

Are we at the point where everything is using lldb-server/debugserver everywhere on OS X/Linux/*BSD even for local debugging? If so, all those platforms probably have a point in the gdb protocol that they know they’re attached and ready to go. If we have an analog on the Windows side, is it just a matter of exposing that for the client code that attaches to the lldb-server/debugserver/{windows-equivalent}-server?

7

There's even a Tech Note for how to do this:

I think you should be able to see that w/o a developer account, but if you can't, here's the code:

#include <assert.h>
#include <stdbool.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/sysctl.h>

static bool AmIBeingDebugged(void)
    // Returns true if the current process is being debugged (either
    // running under the debugger or has a debugger attached post facto).
{
    int junk;
    int mib[4];
    struct kinfo_proc info;
    size_t size;

    // Initialize the flags so that, if sysctl fails for some bizarre
    // reason, we get a predictable result.

    info.kp_proc.p_flag = 0;

    // Initialize mib, which tells sysctl the info we want, in this case
    // we're looking for information about a specific process ID.

    mib[0] = CTL_KERN;
    mib[1] = KERN_PROC;
    mib[2] = KERN_PROC_PID;
    mib[3] = getpid();

    // Call sysctl.

    size = sizeof(info);
    junk = sysctl(mib, sizeof(mib) / sizeof(*mib), &info, &size, NULL, 0);
    assert(junk == 0);

    // We're being debugged if the P_TRACED flag is set.

    return ( (info.kp_proc.p_flag & P_TRACED) != 0 );
}

The document warns that this is an "unstable API" OTOH it's been there since 2004 at least so...

If most platforms have ways to tell if you have been attached to, I have no objections to using that directly.

Jim

8

On Linux, according to the proc(5) man page, you look at /proc/<pid>/status.

There's a field "TracerPid" that gives the pid of the tracing process. The value is 0 if not being traced.

TracerPid: 13849

TracerPid: 0

/proc/pid/stat has a field called "flags" that contains kernel flags, but they don't have anything useful to determine if a process is being traced.

9

Slightly related, but do other platforms have a way to check from an inferior if a debugger is present?

We need to do this frequently from the test inferiors, and I see lots of different approaches used in the test programs, none of which work correctly on Windows.

This is the proper way to do this under windows:

   BOOL WINAPI IsDebuggerPresent(void);

This related function is often helpful:

  void WINAPI DebugBreak(void);

10

The main issue I see with all these APIs is that they are
non-blocking. That is, the test executable cannot simply say "wait
until a debugger attaches", but it will have to do something like:
while (! attached) {
  sleep(X);
  attached = figure_out_if_i_am_attached_using_architecture_specific_methods();
}

This does not sound to me like it will be significantly more stable
then the currently used method:
while (! attached) sleep(X);
where the debugger flips the variable after it attaches.

However, I would definitely welcome providing a default implementation
of wait_for_debugger_attach() (regardless of how it is implemented),
which all tests can use, instead of each one rolling out its own.

Another possible implementation would be to use some standard IPC
mechanism (pipes, signals, sockets, ...) for waking up the inferior
once the debugger is ready. The test inferior could e.g., wait on a
pipe and the debugger will write a single byte there once it attaches.
Since we control both the test runner and the test inferior, this is
enough and you don't need any fancy APIs. The tricky part here would
be to make sure this works during remote debugging.

That said, I don't think the current issues with the attach tests are
caused by this problem. The current protocol, however awkard it may
be, should still work IMO, but we are still seeing flakyness in all
tests that do attaches. I think we have some other issues here and I
am suspecting races in other parts of the system. I was planning to
take a look at these soon...

LLDB picks this up, and the result is that LLDB stops and waits for the user
to continue the inferior just as it would with any other breakpoint, and if
you were to get a backtrace you might see something like this:

looking at: Stack traces for SBProcess: pid = 12588, state = stopped,
threads = 2, executable = test_with_dwarf_and_attach_to_process_with_id_api
Stack trace for thread id=0x3428 name=None queue=None stop reason=none
  frame #0: 0xffffffffffffffff ntdll.dll`DbgBreakPoint + 1

This sounds like a pretty strange way to stop the inferior (I was
quite surprised that ^C also injects a thread into a program), however
it is fundamentally equivalent to what the other platforms do. When we
attach on Linux, the inferior also comes out stopped. The difference
is that we get a SIGSTOP as a stop reason, while on windows you get a
breakpoint in a different thread. It sounds to me like the tests
should not rely on a specific thread being selected after the attach.
Most tests don't do backtraces right after attach since you're likely
to get very unpredictable results on any platform. Those that do
should be fixed to set a breakpoint on the place that interests them,
resume and do a backtrace once they hit that breakpoint. I vaguelly
recall some tests that actually do try to backtrace on attach on
purpose (for example, to see if you can backtrace out of a syscall).
If they exist, we should fix them to first select the right thread for
the backtracing, so they don't try to use the DbgBreakPoint thread.

11

It’s not that it relies on a specific thread being selected, because as you can see there are 2 threads in that trace. The problem is that the second frame is not even yet in the main function, it’s in the startup code because of how early the attach process happens (which itsels is probably actually racy, since attach is user-initiated and might happen before or after main function has entered. But the test checks for main.c as the source file of the second frame, so this si where the probelm lies.

12

Ah yes, our old friend TestHelloWorld. This guy definitely needs to be
fixed. I haven't actually looked at the code before to see why it was
so flaky, but now it all makes sense....

I would just use the "usual" protocol of "expr release_child = 1"
here, but if you wanna go crazy, then go ahead... :slight_smile:

13

Well I’m xfailing it for now, but this other method seems kind of hackish to me because it means the inferior and the debugger have to coordinate with each other, which means the test has to know about the executable and the executable has to know about the test. I’d rather remove one direction of this coupling.

Eventually my plan is to introduce a test_util.a which all of the test executables link against. We can provide a platform-independent wait_for_debugger_to_attach() method here that just does the right thing on all platforms. This way the tests can be written without having to do this funny business :slight_smile:

14

What about fixing lldb to prefer to debug via launch instead of attach depending on the platform???

I don't see how you could write a wait_for_debugger_to_attach() that would work reliably.

15

In what instances do you think it wouldn’t work? At least on Windows it’s trivial. Jim provided some code that would work on OSX, and someone else provided a method earlier in the thread that should work for Linux. What are the specific reliability concerns you have?

Changing lldb to prever debug via launch instead of attach wouldn’t necessarily fix the problem, because you still need to write tests that test attach sometimes.

16

I'm basing this off our own experiences - we use a similar method to
test Android and iOS platforms, and they work most of the time, but fail
randomly, about once every 25 runs, because one of the 50 or so test
cases couldn't sync up with the debug server on the device. Do other
people see similar problems?

If you're running the tests natively, I bet you'll be fine.

17

> Well I'm xfailing it for now, but this other method seems kind of hackish
> to me because it means the inferior and the debugger have to coordinate
> with each other, which means the test has to know about the executable
and
> the executable has to know about the test. I'd rather remove one
direction
> of this coupling.
>
> Eventually my plan is to introduce a test_util.a which all of the test
> executables link against. We can provide a platform-independent
> wait_for_debugger_to_attach() method here that just does the right thing
on
> all platforms. This way the tests can be written without having to do
this
> funny business :slight_smile:

What about fixing lldb to prefer to debug via launch instead of attach
depending on the platform???

The largest section of code getting debugged in production on the Apple
side is in attached scenarios (often by requirement), so reducing attach
test coverage isn't something I think we'd want to do at least on Apple
platforms.

I don't see how you could write a wait_for_debugger_to_attach() that would
work reliably.

What about building a helper method that we mandate main() call for
attach-enabled tests, that loops in a known function and has an exit
scenario that the debugger can control. Then, on attach, we set a
breakpoint on something (which can be shared out several different ways,
either by variable address lookup if we have symbols, or could be added to
the environment by the helper method, etc.) Then the protocol becomes:

The lldb test would do something like this:
1. attach to process
2. collect breakpoint information, which may require the inferior to be
running more if we attach before main has even fired. No worries there,
though.
3. set breakpoint at collected spot.
4. when synchronization breakpoint is hit, we now know we're in a good
startup spot.

And if a test might be used both in an attach-synchronous mode and not
attached at all, that helper method could be controlled (again perhaps by
environment variable or something else) to skip doing the waiting step if
it doesn't expect to get attached to by the debugger.

All that could be put behind helpers on the lldb test side to more or less
hide all traces of it. Test inferiors would need a call to a helper method
that could be fully defined in a header to simplify getting that code
brought in.

That would eliminate races in attach tests by creating an agreed
synchronization point.

-Todd

18

So a few things:

1 - on MacOSX we can reliably launch a process via posix_spawn() with a flag set that sets the process to stop at __dyld_start which is the first instruction in the program. So our launch then attach always works reliably because when we launch the process set set this posix_spawn() flag.

2 - on linux I believe this is solved by doing a manual fork(), sync(), exec().

3 - on windows we have a sync problem for the launch first then attach due to this extra thread. Is this the case?

Seems like you should be able to launch with Use the CreateProcess() function, set CREATE_SUSPENDED and then attach no?

I would rather not solve this problem that is windows only by requiring that all test suite functions link to a .a file. We should be able to launch a process and debug it using the launch then attach method that is used by our platform.

Greg

19

Or is this just a problem with a bad test? Are we telling LLDB to launch the test and LLDB is doing the launch + attach in the windows platform and that is what is hosing up the attach? Or is this a flawed test where the test just manually launches the process with no hopes of syncing with the process we are attaching to?

20

I think it’s mostly a problem of tests. Basically we have tests which are written in such a way that the inferior and the debugger need to reach a synchronization point before the test can proceed to test what it wants to test.