JIT debugging and Windows

Hi Andrew,

I’m also one of the Julia core developers and the guy currently working on
debugging support. I was given a patch to add MCJIT debugging support
to LLDB with the disclaimer that it probably doesn’t work. I haven’t tried it
yet, so I can’t say how close it is, but it’s probably at least a starting point.

I’d be happy to coordinate any efforts.

  • Keno

Hi Keno,

I found this old patch that hooks into the GDB JIT registration mechanism which we were planning to use as a starting point:

http://lists.cs.uiuc.edu/pipermail/lldb-dev/2010-December/000314.html

If you have anything else we’d love to take a look!

Cheers,
Andrew

I have this (as I said, I haven’t tried it). It seems to be a different patch.

MCJIT-debugging.patch (26 KB)

Looks great, thanks Keno. I’ll look into this next week and keep you posted.

Cheers,
Andrew

I’ve started looking into this and it’s actually not that difficult to do (the JIT part at least). I think the most difficult thing is
figuring out where to put the functionality. The way I see it, the JIT interface is almost a DynamicLoader, with the difference
that asking it about things like whether or not loading a dynamic library is safe doesn’t make much sense. Another problem
is that currently there’s a 1:1 relationship between Processes and DynamicLoaders. Perhaps a new class of Plugins
(JITLoader?) could be added with a 1:n relationship to processes. Some of the interface (I’m thinking
DidLaunch,DidAttach,Get/SetStopWhenImagesChange) could be pulled out into an abstract base class. Does that make
sense. I’d be happy to implement this once we decide on the design.

Keno

Hi Keno,

I noticed the same thing and agree that the JIT loader doesn’t perfectly fit as a DynamicLoader, the suggestion from a couple of years back on the list was to make the Process → DynamicLoader relationship 1:n but I prefer your suggestion of adding a new plugin type (JITLoader). Right now there would only be one JITLoader type for the GDB interface.

The patch could probably be fixed up as-is as a first step (leaving the JIT loading bits as a growth on the DynamicLoaderPOSIXDYLD) and we could move it out into its own plugin type once that’s working. I should have time this coming week to get this working but if you’ll also have time then I’m happy to leave it with you.

Cheers,
Andrew

I’ve gotten pretty far,
Code is here: https://github.com/loladiro/lldb
It’s interacting correctly with the debug interface and I’ve got it loading the object file from memory correctly for
MachO/DWARF (working on ELF right now). Currently not much use as it doesn’t seem to be looking at the
debug information that were just added (I can see them via image dump sections and I can set breakpoints
correctly though, so that’s a plus). For DWARF, there’s a few changes for upstream LLVM needed. I’ll put them
up if anybody’s interested, otherwise, I’ll just keep chugging along and wait until I have more complete patch.

  • Keno

This looks great. I’m still in holiday mode but will tackle any outstanding ELF-related work when I’m back on Thursday, feel free to ping me or the list again if you get any further. Thanks for pushing on this!

Cheers,
Andrew

I’m currently in holiday mode as well. A couple of things that I didn’t get to that need to be resolved (these are pretty general questions on lldb itself, so I’d appreciate input from anybody on this list):

  • I’m using a regular rather than an internal breakpoint, because the latter does not get updated when shared libraries are loaded (Is this intended? Is there a recommended workaround?)

  • I’m not seeing the function in backtraces even though I can set breakpoints. Is there anything else that needs to be done/ Do the backtraces use different information that is perhaps not relocated properly? If so, how can I check?

  • I’m also not seeing line numbers, etc. Probably related to the second point.

  • Keno

Actually, never mind, I am getting backtraces (Woot!!). Still not getting line numbers though.

Actually, never mind, I am getting backtraces (Woot!!). Still not getting
line numbers though.

That's really exciting!

--Tim

Hi Keno,

This all sounds great, can you check in the progress you’ve made so far with seeing JITed functions in backtraces? I can try to figure out what’s happening with the missing line numbers.

Regarding the breakpoints a quick look at Breakpoint.cpp seems to show that this is intentional, there’s an explicit check for !IsInternal() inside Breakpoint::ModulesChanged() which I would guess is what applies here. I think we would want these breakpoints to be internal though so that users don’t see them and if we make sure that the JITLoader gets an event whenever a new shared library is loaded then we can also add more explicit logging around when the JIT register code function has actually been found and resolved.

Thanks!
Andrew

Everything I’ve been doing with that was upstream llvm to get the relocations right rather than lldb. https://github.com/loladiro/lldb/commit/991583df4d052cb7dab692e657c4ced4d01ff384 is up to date (the next commit on master has a few debugging things I was trying out, but that commit is functionally complete).

Yes, I thought about that solution as well. Seems like the way to go.

I’ve attached a patch to add support for JITed ELF objects to the work you did in your lldb repo. This was taken almost verbatim from the original patch you posted so all credit goes to you. :slight_smile: With this patch we now have breakpoints and full stack traces into the code JITed by our KL compiler on Linux, including line numbers and variable info.

Since it sounds like you’re on OSX I’m guessing your upstream LLVM changes included adding the calls to the GDBRegistrar in RuntimeDyldMachO.cpp? It looks like they’re only in there for ELF right now. Let me know if there’s anything else I might need for OSX and I’ll investigate what’s happening with the missing line numbers there.

ELFJIT.patch (12.3 KB)

Oh, this is great. I can’t wait to try it. I’ll clean up the upstream LLVM patch and post it. It’s not all that big, it’s basically adding those calls and updating the appropriate structs in memory (just like ELF does).

Hey Keno, I gave this a quick run on OSX after adding code to the RuntimeDyldMachO to call the JIT registration function. lldb catches the breakpoint correctly and I can see that it’s found the symbols I’d expect in the debug info but it looks like relocations aren’t being applied so it’s not resolving symbols in my sample backtrace. Since you mentioned you’ve been doing some work related to the relocations (in LLVM itself) I’ll leave this for now but feel free to ping me anytime if you want me to try again with any updated code.

Cheers!
Andrew

Hey Keno,

We were busy getting a release out so I’d let this sit for the last little while but I’d like to make a few small fixes here and then see if we can’t get this applied to LLDB trunk if you agree. The changes I’d like to make are:

  • Make the JIT breakpoint internal (as we discussed)

  • Make each JITed object an ObjectFile that’s part of a single Module owned by the JITLoader rather than each one being a Module

  • Update the text of some of the JIT log messages

  • Rename the GDBJIT dir to just GDB (being picky, but the word JIT should be implied :))

If you’re ok with that I’ll try to get those changes done in the next week or two and then start poking at getting a Windows process plugin up and running. Right now on my end this works only on Linux but it’d be great to get it going on OSX as well if you have any additional insight there.

If you’re curious about how the new JIT debugging looks in our software there’s a video here: https://vimeo.com/85356121 Note that this should mostly work without any modification to debug Julia code on Linux too if the Dwarf filenames you’re using are full file paths to the source files.

Cheers,
Andrew

Hi Andrew,
I’be been busy getting our debug information up to snuff to actually make use of this, so I haven’t had much time to work on this. If you’re willing to take those changes up, I’d be more than happy for you to do them, as I will be quite busy myself over the next two weeks or so. This is gonna be awesome. Good work!

Keno