Hi there,
I’m a google summer of code student this year, on FreeBSD part.
And I’m writing to introduce my project, we are aiming to support kernel debugging on FreeBSD for LLDB.
Ed Maste (emaste@freebsd.org) is my mentor for this project, I’ll be working closely with him.
Here is the project page:
https://wiki.freebsd.org/SummerOfCode2014/LLDB_KernelDebugging
This is my LLDB fork:
https://github.com/cosql/lldb
Any comments are more than welcome.
Hi there,
I'm a google summer of code student this year, on FreeBSD part.
And I'm writing to introduce my project, we are aiming to support kernel
debugging on FreeBSD for LLDB.
Ed Maste (emaste@freebsd.org <mailto:emaste@freebsd.org>) is my mentor
for this project, I'll be working closely with him.Here is the project page:
SummerOfCode2014/LLDB_KernelDebugging - FreeBSD Wiki
Looks like a cool project. Congrat for the selection!
This is my LLDB fork:
GitHub - cosql/lldb: Mirror of official lldb git repository located at http://llvm.org/git/lldb. Updated hourly.
Welcome and good luck 
Sylvestre
Good luck with your project, Mike!
I’ll be interested in seeing what kind of work you end up doing there. Maybe we’ll be able to leverage that on the Linux side as well
Your project will be great! Below I will outline a few things to keep you on track:
Core file kernel debugging:
- You shouldn't have to do too much to the current ProcessELFCore except help it to locate hint address for the list of shared libraries and the main kernel file in memory. This is done in:
virtual lldb::addr_t
ProcessELFCore::GetImageInfoAddress ();
- Most of the work will be done in your DynamicLoader subclass that then uses the hint from the ProcessELFCore::GetImageInfoAddress() as to where the shared libraries are. This new DynamicLoader subclass will be used for both live kernel debugging and for Core file debugging and should be able to track down all kernel extensions that are loaded.
Live kernel debugging:
- You will make a new process subclass like you were saying unless the GDB remote protocol is used to talk to the kernel?
- You should re-use your dynamic loader from the Core file step here
Extra credit:
- Get threads that are context switched in memory to show up. In the MacOSX kernel debugger we have the notion of 1 core we can talk to through the KDP interface which is the code that runs the debugger. There are tons of threads in the kernel that are sitting in memory. We have a python plugin that will supply a list of memory threads back to lldb. So even though the kernel debug interface only supports N cores, we might also be able to show M number of memory threads and see their backtraces, registers, etc. The MacOSX kernel implements this using python code that can use kernel globals to get the list of in memory threads and create them on the fly. There is a very simple example of this in:
svn cat http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/operating_system.py
Let us know if you have any questions as far as implementation and architecture goes. The MacOSX kernel process subclass does things correctly so keep an eye on that plug-in when you have questions or need guidance.
Greg Clayton
Thanks for the insight Greg - a few comments:
Core file kernel debugging:
- You shouldn't have to do too much to the current ProcessELFCore except help it to locate hint address for the list of shared libraries and the main kernel file in memory. This is done in:
Note that kernel core files on FreeBSD aren't (currently) ELF files,
and may be a different format on different architectures, so there's
some extra work here. Dumps are written in a contiguous blob
(typically to swap) at the time of a crash, and then copied to the
filesystem after the next reboot.
The dump function for amd64/x86_64 is minidumpsys in
minidump_machdep.c[1], and consists of:
1. Dump header (stripped during copy to filesystem)
2. Minidump header
3. Copy of kernel message buffer
4. Bitmap of pages present in the dump
5. Page directory pages
6. Pages themselves
7. Trailer
All of this is abstracted by the libkvm library, which provide
kvm_read and kvm_write functions which a VA address parameter. I
think this is the right way to introduce support for FreeBSD kernel
cores at first, as I'll explain below.
Live kernel debugging:
- You will make a new process subclass like you were saying unless the GDB remote protocol is used to talk to the kernel?
We currently have separate ways of interacting with the kernel for
remote and local debugging. For remote debugging the kernel has a
built-in GDB protocol stub which is available over a serial or
firewire interface, and this should mostly work with the existing LLDB
support. We do local debugging (examining threads, global objects,
etc.) via /dev/mem, abstracted by the same kvm_read and kvm_write
functions in libkvm. Since local FreeBSD debugging will be done on a
FreeBSD host (by definition), using libkvm makes sense. And since
we'll have that, using it also for cores makes sense to get started.
Of course we'll want to allow debugging a FreeBSD kernel core on OS X
or Linux too. I think the way forward for us there is to create a
utility in the FreeBSD base system that converts the minidump format
to ELF, rather than teaching LLDB about our core formats in a
cross-platform way. This utility could be built-in to savecore(8),
the tool which reads the dump out of a raw partition and writes to the
filesystem. There is also ongoing discussion about bringing a KDP
target to the kernel for live debugging.
Extra credit:
- Get threads that are context switched in memory to show up. In the MacOSX kernel debugger we have the notion of 1 core we can talk to through the KDP interface which is the code that runs the debugger.
We have code[2] for this already in kgdb, our GDB 6.1 port with kernel
support, and we should be able to re-use most of it as is. The kernel
threads just appear as regular threads in kgdb:
(kgdb) info threads
827 Thread 102126 (PID=28738: kgdb) sched_switch
(td=0xfffffe021882b000, newtd=0xfffffe0008392920, flags=<value
optimized out>)
at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
826 Thread 102334 (PID=28737: sudo) sched_switch
(td=0xfffffe02d2669920, newtd=0xfffffe0008392920, flags=<value
optimized out>)
at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
825 Thread 104699 (PID=28715: more) sched_switch
(td=0xfffffe049cec7920, newtd=0xfffffe0008390000, flags=<value
optimized out>)
at /tank/emaste/src/git-stable-9/sys/kern/sched_ule.c:1904
...
-Ed
[1] http://svnweb.freebsd.org/base/head/sys/amd64/amd64/minidump_machdep.c?annotate=257216#l219
[2] http://svnweb.freebsd.org/base/head/gnu/usr.bin/gdb/kgdb/kthr.c?annotate=246893
Hi,
Small hint from me: when it comes to obtaining test data post-mortem kernel debugging I have found
qemu dump-guest-memory
is invaluable - cheapest and easiest way. It worked great for linux + x86_64, it also works for x86 and is easy to extended on other architectures, and also I see no technical obstacles for using it with FreeBSD.
Also if there is intrest in getting crash alike functionality for linux in community into lldb please let me know, would be more than happy to collaborate and share my work in this area.
I am far from feature parity from crash and at some point I found fitting into lldb’s Process and DynamicLoader API unnecessary burden, with not that much gain. That was especially due to my limited resources, at prototyping stage and also lack of freedom in introducing heavier dependecies (eg. libarchive), that made my work progress at satisfying rate. It might have been just my lacking technical competences, nothing to discourage from going this way
But I might revisit and reconsider this decision, or at least share parts that I believe might be useful. Now I am bit more confident about shape and form, and see bit better where I am going…
Good luck, and have fun with your project, I am sure it’ll be very interesting.
Cheers,
/P