Can't debug with a -g compiled binary as a non-root user on OS X 10.11 El Capitan

I can only seem to debug our binary as the root user om 10.11. I rebooted at one point, and lldb did work briefly with a system user but then after the machine ran for a bit, it proceeded to not work again. Rebooted again, and again, lldb failes with this error…

lldb /opt/aspera/bin/ascp
(lldb) target create “/opt/aspera/bin/ascp”
2015-10-02 14:24:17.091 lldb[1721:12884] Metadata.framework [Error]: couldn’t get the client port
Current executable set to ‘/opt/aspera/bin/ascp’ (x86_64).
(lldb) r -i ~/.ssh/id_rsa /tmp/mp_source/* localhost:/tmp/mp_dest/
error: process exited with status -1 (unable to attach)

As root, I can reproduce the error:

root# lldb /opt/aspera/bin/ascp
(lldb) target create “/opt/aspera/bin/ascp”
2015-10-02 14:30:40.515 lldb[1864:14630] Metadata.framework [Error]: couldn’t get the client port
Current executable set to ‘/opt/aspera/bin/ascp’ (x86_64).
(lldb) r -i /var/root/.ssh/id_rsa /tmp/mp_source/* localhost:/tmp/mp_dest/
Process 1866 launched: ‘/opt/aspera/bin/ascp’ (x86_64)

Session Stop (Error: Session initiation failed, Server process failed to start: permissions?)
Process 1866 exited with status = 1 (0x00000001)

I have another machine running OS X 10.9 and lldb where everything works flawlessly.

The problem with out binary seems to be that OS X is prohibiting our binary from starting another process (even as root). Not sure if this is the right list for that question though. Assume it’s something to do with 10.11’s security model.

Hi Tony,

This is the right list.

Are you using an LLDB that you built locally? If so, can you move aside the debugserver that you find somewhere under in your LLDB.framework bundle directory, and make a symlink to the debugserver that comes out of your /Applications/Xcode.app bundle? Your official Xcode.app one should be in a location like:
/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/debugserver

The other thing it could be is that I think your lldb_codesign cert may need to be recreated on a new OS. I seem to recall the instructions there indicate the code signing cert does not survive OS updates but I might be mistaken.

I suspect the symlink will resolve your issue, though. With tighter security, it is likely that a home-built debugserver is no longer going to work without being Apple signed. We may need to adjust the Xcode build to create a symlink to the official one if that’s the case.

-Todd

Hi Tony. There are new security mechanisms in Mac OS X 10.11 El Capitan, called System Integrity Protection, but I don't recognize this failure mode. Try a stripped down example, like

$ echo 'int main () { }' > /tmp/a.c
$ xcrun clang /tmp/a.c -o /tmp/a.out
$ xcrun lldb /tmp/a.out
(lldb) br s -n main
(lldb) r

That should work. That's a baseline, then we can start working on why your example isn't working.

The fact that it doesn't work as root makes it less likely it's an unsigned debugserver / missigned debugserver issue. You can run an unsigned / mis-signed lldb as root and it will still work on os x 10.11, as well as a signed one run by a user account.

Is the binary you're running under the debugger signed? I think it needs the get-task-allow entitlement if the debugger is going to attach/run it.

The lldb I'm using is from XCode 7.0.1.

I can debug my binary if I run lldb as root.

I eventually figured out my actual bug from log messages without lldb. Turns out, Mac OS X 10.11 El Capitan doesn't allow the root user to deploy binaries to /usr/bin which our installer does so that our binary is in the default ssh environment path. I setup a custom ~/.ssh/environment and configured PermitUserEnvironment to yes in /etc/sshd_config and that let my binary run normally.

But, still, I can't seem to run lldb as a normal system user with our binary.

Tony

(resending my reply with a Cc to lldb-dev - System Integrity Protection in Mac OS X 10.11 El Capitan has impact on lldb in several important ways, I think others will be interested.)

Yes, System Integrity Protection in El Capitan includes the restriction that nothing can be modified in directories like /usr, /System, /bin, /sbin, even by the root user. /usr/local is excepted from these restrictions.

SIP also restricts the debugger from being able to attach to system processes. e.g. trying to attach to Mail.app will fail, regardless of whether you're root or a regular user, official lldb or self-built/self-signed lldb.

More complete details on exactly what SIP includes:

https://developer.apple.com/library/prerelease/ios/documentation/Security/Conceptual/System_Integrity_Protection_Guide/System_Integrity_Protection_Guide.pdf

J