Running LLDB tests

warren.paul · January 28, 2011, 6:37pm

I’ve been playing around with LLDB and trying to get it to load some symbol files generated by the toolchains we support. So far so good, but I need to start doing more lookups, etc. and would like to script these tests. I thought I’d start with the LLDB tests and build up from there. But when I run the tests I get mostly failures followed by a crash. Am I running them incorrectly, or missing some steps, … ?? Any help would be greatly appreciated. Also, I haven’t seen any logging in the Dwarf reader yet but have hit a few asserts. Any tips for having my test scripts report anything the debugger doesn’t like or expect in the Dwarf?

Thanks,
Warren

Here’s the output of the tests:

./dotest.py -v

Session logs for test failures/errors will go into directory ‘2011-01-28-12_17_51’

Johnny_Chen · January 28, 2011, 7:13pm

Hi Warren,

If you look in the ‘2011-01-28-12_17_51’ directory, you should see some files explaining the detailed traces
which lead to the failure reporting. For example,

[10:04:58] johnny:/Volumes/data/lldb/svn/trunk/test $ ls -l 2011-01-28-09_57_01
total 64
-rw-r–r-- 1 johnny admin 3171 Jan 28 10:00 ExpectedFailure-TestNamespace.NamespaceTestCase.test_with_dsym_and_run_command.log
-rw-r–r-- 1 johnny admin 3129 Jan 28 10:00 ExpectedFailure-TestNamespace.NamespaceTestCase.test_with_dwarf_and_run_command.log
-rw-r–r-- 1 johnny admin 2173 Jan 28 09:59 ExpectedFailure-TestObjCMethods2.FoundationTestCase2.test_NSArray_expr_commands_with_dsym.log
-rw-r–r-- 1 johnny admin 2130 Jan 28 09:59 ExpectedFailure-TestObjCMethods2.FoundationTestCase2.test_NSArray_expr_commands_with_dwarf.log
-rw-r–r-- 1 johnny admin 2639 Jan 28 09:59 ExpectedFailure-TestObjCMethods2.FoundationTestCase2.test_NSString_expr_commands_with_dsym.log
-rw-r–r-- 1 johnny admin 2596 Jan 28 09:59 ExpectedFailure-TestObjCMethods2.FoundationTestCase2.test_NSString_expr_commands_with_dwarf.log
-rw-r–r-- 1 johnny admin 1492 Jan 28 09:58 Failure-TestClassTypes.ClassTypesTestCase.test_with_dsym_and_expr_parser.log
-rw-r–r-- 1 johnny admin 1450 Jan 28 09:58 Failure-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_expr_parser.log
[10:53:21] johnny:/Volumes/data/lldb/svn/trunk/test $ tail -20 2011-01-28-09_57_01/Failure-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_expr_parser.log

Expecting start string: Breakpoint created: 1: name = ‘C’, locations = 1
Not matched

FAIL

Traceback (most recent call last):
File “/Volumes/data/lldb/svn/trunk/test/class_types/TestClassTypes.py”, line 53, in test_with_dwarf_and_expr_parser
self.class_types_expr_parser()
File “/Volumes/data/lldb/svn/trunk/test/class_types/TestClassTypes.py”, line 165, in class_types_expr_parser
startstr = “Breakpoint created: 1: name = ‘C’, locations = 1”)
File “/Volumes/data/lldb/svn/trunk/test/lldbtest.py”, line 802, in expect
msg if msg else EXP_MSG(str, exe))
AssertionError: False is not True : Breakpoint created successfully

To rerun this test, issue the following command from the ‘test’ directory:

./dotest.py -A i386 -C clang -v -t -f ClassTypesTestCase.test_with_dwarf_and_expr_parser
[10:54:35] johnny:/Volumes/data/lldb/svn/trunk/test $

You can do further debugging from the Python stack traceback to have some idea of how/why the test failed.
It could even be that the Test*.py itself is incorrect.

Right now, the test suite runs as a single Python invocation, so if there is a seg fault, the test suite aborts abruptly.
You can exclude certain test cases from being run while investigating the seg faults, however, by customizing
your own copy of blacklist.py in the test directory. The file contains some example usages.

For lldb loggings, you can look at the lldbLoggings() function within dotest.py file, it currently use some environment
variables to set the logging destination and the logging options. I’ll ask Greg whether there are any specific thing to
turn on just for Dwarf reader things.

Thanks.

Greg_Clayton1 · January 28, 2011, 7:18pm

Are you running these on MacOSX?
If so, did you follow the instructions in "lldb/docs/code-signing.txt"?

You will need to generate your own code signing certificate and then debug a program at least once since the first time you used this code signing certificate a dialog will pop up asking for permission where you will have to enter your username/password. After this you should be able to debug.

So to fix this, make sure you can debug something:
./lldb /bin/ls
(lldb) run

...

If this works, then the test suite should run for you.

Greg Clayton

warren.paul · January 28, 2011, 10:30pm

I had done the first part of code-signing.txt, up to "The next steps are
necessary on SnowLeopard". The rest sounded like a workaround for an OS
bug, and I have 10.6.5, so I figured I see if I could build LLDB without
it. I can, and I can debug it as well, so I assume that means I don't
need the steps from "The next steps are necessary on SnowLeopard"?

I looked through a few of the failure logs and it looks like lldb is
failing to launch the process. I can have it launch a process when I run
lldb from the command line, so perhaps the test script is finding a
different lldb? I've only build the debug version, so I'm hoping
dotest.py is finding it at ./../build/Debug/?

My Python version is Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49).

Any ideas?

Thanks,
Warren

jingham · January 28, 2011, 10:42pm

Try completing all the steps in that doc. I can't debug at all on 10.6.6 unless I do all the steps I described in that little document. So it is at least worth giving that a shot - maybe you are lucking out a little, but not quite enough, so that it works directly but not under Python? Note the binary that needs to get properly codesigned is debugserver, since that's the tool that has to have the rights to access the task port of the debugee. It doesn't matter whether lldb is properly codesigned at present.

If you want to know for sure which LLDB you are using (Python just uses the framework and not the lldb binary), then do:

(setenv DYLD_PRINT_LIBRARIES YES; ./dotest.py)

or the bash equivalent, and then watch the testsuite run, and note from whence it is picking up LLDB.framework.

Jim

Greg_Clayton1 · January 28, 2011, 11:00pm

You should need the code signing for any SnowLeopard system as far as I know. You need to reboot sometimes after doing the code signing to make things work in all cases. It sounds like you have debugging from lldb working, but not when it is running from Python.

I am guessing task_for_pid() is failing when being run under python.

The easiest thing to try is reboot and see if it works after a reboot. If not, then run python test with some environment variables set (the following is for bash):

LLDB_DEBUGSERVER_LOG_FILE=/tmp/debugserver.txt LLDB_DEBUGSERVER_LOG_FLAGS=0x800e0e ./dotest.py -v enum_types

This causes the tester to run just the tests in the "enum_types" directory which will run two tests. When you are done you should have a file in /tmp that you can send to me directly and I can take a look.

If you have a laptop, please do try and reboot first. I won't go into details, but there are some unexptected things that can make task_for_pid() fail which means we can't debug processes.

Greg Clayton

warren.paul · February 9, 2011, 10:32pm

I forgot to respond to this earlier, but doing the rest of the steps in
the signing doc and rebooting fixed the problem.

Thanks,
Warren