Module management questions

I have an app that's dynamically generating and JITing code, and will have many such cases in the course of its run. They need to be JITed separately, because I need to execute the first batch before I know what the second one will be. Of course, I *still* need to execute the first after the need for the second arises, so I need to retain the JITed machine code for all the functions I dynamically compile.

I think I just discovered the hard way that I can't keep adding to a single Module and ExecutionEngine, and re-optimizing -- certain combinations of optimization passes evidently are not happy doing this. In particular, I seem to have special problems with templated (but not explicitly 'inline') C++ functions, when llvm inlining passes are used. It seems to JIT fine, no errors, but I end up with my JITed code crashing upon execution (seg faults, and occasionally unknown opcodes).

First question: can anybody confirm my theory, that it is not in fact safe/proper to add more functions to an already-optimized Module/ExecutionEngine and then perform more passes on it again?

Assuming that is the case... I'm very concerned about leaks and resource usage (including memory) if I need to create a new module and/or EE for every one of these sets of functions that I need to dynamically compile.

Second question: what's the best way to proceed? What's the approved way to completely free a Module/EE? And can I do so and still retain the JITed machine code to continue executing?

In principle this ought to work, if you're careful. Are you sure you're not generating code that calls into functions that got totally inlined away? Are you running the Verifier pass at regular intervals?

--Owen

In principle this ought to work, if you're careful. Are you sure you're not generating code that calls into functions that got totally inlined away?

How would I know?

Are you running the Verifier pass at regular intervals?

Yes, both before and after the set of optimization passes.

So let me clarify what I'm doing: I have a whole bunch of C++ functions (including some inline and templated functions) that the dynamically-compiled user code may need, so I precompile that into bitcode with llvm-g++, and seed the Module with that bitcode, then add the user code (translated by our app from a custom programming language into LLVM IR) and then optimize. But I need to do this several times, for several user programs, which can come along irregularly during the execution of our app, and I never want to throw away the JITed machine code, though once I JIT I should not need the IR I generated for the user program.

Also, an update: I've found that I'm totally safe if I create (and seed) a new Module and ExecutionEngine every time I need to JIT user code. But that's very slow and also a pig on memory use since I never free the Module or EE.

I also found by experimentation that I seem to be able to keep a single Module and merely make a new EE for each new bit of user code -- but that is only safe if I move certain routines (that the user code may call) from the llvm-g++-ified code into a module statically compiled by g++. That is, if those few routines are just function calls to LLVM and not attempted to be inlined, all seems well as long as I don't reuse the EE's. This approach seems stable for now and saves most of the memory, but still smells like I'm working around an LLVM bug.

So far I've had a hard time actually reproducing a crash in anything other than our full app, but if somebody wants to work with me on tracking the underlying bug, I can try to narrow it down to a simple example.

In the mean time, perhaps somebody can answer these questions for me: who owns the machine code that is returned by ExecutionEngine::getPointerToFunction? Is that "static" once JITed? Owned by the EE? If I delete the Module and/or EE after JITing, can I still call the JITed code? If not, may I suggest as a future feature either the ability for the client app to take complete ownership of the JIT code, or else to ask the Module/EE to release as many resources and memory as possible except the callable JIT code?

In principle this ought to work, if you're careful. Are you sure you're not generating code that calls into functions that got totally inlined away?

How would I know?

Are you running the Verifier pass at regular intervals?

Yes, both before and after the set of optimization passes.

So let me clarify what I'm doing: I have a whole bunch of C++ functions (including some inline and templated functions) that the dynamically-compiled user code may need, so I precompile that into bitcode with llvm-g++, and seed the Module with that bitcode, then add the user code (translated by our app from a custom programming language into LLVM IR) and then optimize. But I need to do this several times, for several user programs, which can come along irregularly during the execution of our app, and I never want to throw away the JITed machine code, though once I JIT I should not need the IR I generated for the user program.

The llvm::Function object cannot be destroyed, because it is still
used. However, you can free the IR by calling F->deleteBody() to save
memory.

Also, an update: I've found that I'm totally safe if I create (and seed) a new Module and ExecutionEngine every time I need to JIT user code. But that's very slow and also a pig on memory use since I never free the Module or EE.

I also found by experimentation that I seem to be able to keep a single Module and merely make a new EE for each new bit of user code -- but that is only safe if I move certain routines (that the user code may call) from the llvm-g++-ified code into a module statically compiled by g++. That is, if those few routines are just function calls to LLVM and not attempted to be inlined, all seems well as long as I don't reuse the EE's. This approach seems stable for now and saves most of the memory, but still smells like I'm working around an LLVM bug.

So far I've had a hard time actually reproducing a crash in anything other than our full app, but if somebody wants to work with me on tracking the underlying bug, I can try to narrow it down to a simple example.

In the mean time, perhaps somebody can answer these questions for me: who owns the machine code that is returned by ExecutionEngine::getPointerToFunction? Is that "static" once JITed? Owned by the EE? If I delete the Module and/or EE after JITing, can I still call the JITed code? If not, may I suggest as a future feature either the ability for the client app to take complete ownership of the JIT code, or else to ask the Module/EE to release as many resources and memory as possible except the callable JIT code?

You can free the machine code yourself by saying
EE->freeMachineCodeForFunction(F) . If you destroy the EE, it will
also free the machine code.

Reid

Thanks, but unfortunately, this is exactly the opposite of what I want to do. I need to retain the machine code indefinitely, but I want to free all possible other resources that are not strictly needed simply to jump into the machine code and execute it. In particular, the Modules appear to be the culprit in the memory consumption.

If I call ExecutionEngine::removeModule (after JITing a function from the module), does that mean that I can subsequently no longer call the function? Or can I? Can I delete the Module at that point and still use the machine code?

I'm pretty sure you cannot delete the module and still execute the
code. The JIT uses pointers to Function objects in its internal data
structures, and if you try to free one while it's holding a reference,
I believe you'll get an assertion error. The Module owns the
Functions, so deleting it will free them.

I would imagine that most memory is used in the IR of the functions,
which you can delete as I described in my last message. Did that
help?

Reid

Hi Larry,

I faced the same issue and solved it by writing my own JITMemoryManager.
This is the class that is responsible for allocating memory for the JITed
functions. So your derived implementation can let the application take
ownership of this memory before destructing the entire execution engine.
createJIT takes a JITMemoryManager as parameter, so it's straightforward to
make it use your derived class.

Good luck!

Nicolas

I would imagine that most memory is used in the IR of the functions,
which you can delete as I described in my last message. Did that
help?

It helps a little, but it seems that most of the memory consumption is elsewhere. I haven't had the time to spelunk through the Module internals enough to say for sure what's using up the memory.

But...