JIT Leaks?

Paolo_Invernizzi · July 14, 2007, 2:39pm

Hi all,

I'm playing with the JIT trying to find out the best way to handle a genetic algorithm that we are developing: I need to:

1) generate about 50.000 functions a time
2) JIT them and execute them over a large dataset
3) Discard everything
4) ... stuff...
5) Loop to 1 again and again and again...

I've the code running, under LLVM 2.0 on OS X 10.4.10 but it leaks, so I started to tweak the standard "fibonacci" example to understand how to do in a proper way the step 3.

- Simply deleting the execution engine instance in the fibonacci code and looping all the main code give me memory leaks. Can anybody confirm this?

- Conserving the module, the execution engine and the module provider, but releasing the fibonacci function with:

EE->freeMachineCodeForFunction(FibF);
FibF->eraseFromParent();

and recreating the function from scratch leaks even more. Idem, someone can confirm this?

Thanks in advance.
Paolo Invernizzi

Holger_Schurig1 · July 14, 2007, 5:17pm

- Simply deleting the execution engine instance in the fibonacci code
and looping all the main code give me memory leaks. Can anybody
confirm this?

- Conserving the module, the execution engine and the module
provider, but releasing the fibonacci function with:

You can find out what exactly leaks with the help of valgrind.

Anton_Korobeynikov · July 14, 2007, 5:56pm

Holger.

You can find out what exactly leaks with the help of valgrind.

It seems, that Paolo is on Mac OS X. No valgrind there

Gordon_Henriksen1 · July 14, 2007, 8:29pm

All is not lost…

http://developer.apple.com/documentation/Performance/Conceptual/ManagingMemory/Articles/FindingLeaks.html

— Gordon

void · July 15, 2007, 5:39am

On OS X, you can use the "leaks" command to find out what's leaked in a particular process. It's not exact, but may give a good indication of what's wrong.

-bw

Paolo_Invernizzi · July 15, 2007, 10:28am

First, I'm not sure if deleting the ExecutionEngine is all I need to clean-up... so I started with a minimal test just to check

int main( int argc, char **argv ){
     while( true ){
         Module *M = new Module("M");
         Function *F = cast<Function>(M->getOrInsertFunction("F", Type::Int32Ty, (Type*)0));
         BasicBlock *BB = new BasicBlock("BB",F);
         new ReturnInst( ConstantInt::get(Type::Int32Ty,1), BB );
         std::vector<GenericValue> Args(0);
         ExistingModuleProvider *MP = new ExistingModuleProvider(M);
         ExecutionEngine *EE = ExecutionEngine::create(MP, false);
         GenericValue GV = EE->runFunction(F, Args);
         delete EE;
     }
}

The memory goes up, and OS X leaks command returns pages of:

Leak: 0x011036b0 size=16
         0x00000000 0x00000000 0x00000000 0x01102988 .............)..
Leak: 0x011036a0 size=16
         0x00000000 0x00000000 0x00000000 0x01102988 .............)..
Leak: 0x01103680 size=16
         0x00000000 0x00000000 0x00000000 0x01102c78 ............x,..
Leak: 0x01103670 size=16
         0x00000000 0x00000000 0x00000000 0x01102728 ............('..
Leak: 0x01103640 size=16
         0x00000000 0x00000000 0x00000000 0x01102978 ............x)..
Leak: 0x01103630 size=16
         0x00000000 0x00000000 0x00000000 0x01103368 ............h3..
Leak: 0x01103620 size=16
         0x00000000 0x00000000 0x00000000 0x01102a58 ............X*..
Leak: 0x01103610 size=16
         0x00000000 0x00000000 0x00000000 0x01103368 ............h3..

So all I know is that It's a size 16 leak *grin*.

But MallocDebug reports as leaks

2.4M start
2.4M operator new(unsigned long)
2.4M MDmalloc
2.4M main
2.4M llvm::PMDataManager::add(llvm::Pass*, bool)
2.4M llvm::LoopPass::assignPassManager(llvm::PMStack&, llvm::PassManagerType)
2.4M llvm::LLVMTargetMachine::addPassesToEmitMachineCode(llvm::FunctionPassManager&, llvm::MachineCodeEmitter&, bool)
2.4M llvm::JIT::JIT[in-charge](llvm::ModuleProvider*, llvm::TargetMachine&, llvm::TargetJITInfo&)
2.4M llvm::JIT::create(llvm::ModuleProvider*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
2.4M llvm::FunctionPass::assignPassManager(llvm::PMStack&, llvm::PassManagerType)
2.4M llvm::ExecutionEngine::create(llvm::ModuleProvider*, bool, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
2.4M _start
2.4M 0x1
2.4M llvm::PMTopLevelManager::schedulePass(llvm::Pass*)
2.4M llvm::FunctionPassManagerImpl::addTopLevelPass(llvm::Pass*)

I hope this is usefull, I'm pretty new to OS X.

Cheers, Paolo Invernizzi

Anton_Korobeynikov · July 15, 2007, 12:11pm

Hello, Paolo.

I hope this is usefull, I'm pretty new to OS X.

Well. From your example you have got the only valgrind results:

==28336== 58 bytes in 2 blocks are possibly lost in loss record 70 of
127
==28336== at 0x4020A92: operator new(unsigned)
(in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==28336== by 0x410010B: std::string::_Rep::_S_create(unsigned,
unsigned, std::allocator<char> const&)
(in /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/libstdc++.so.6.0.3)
==28336== by 0x4102A04:
(within /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/libstdc++.so.6.0.3)
==28336== by 0x4102B25: std::string::string(char const*,
std::allocator<char> const&)
(in /usr/lib/gcc/i686-pc-linux-gnu/3.4.6/libstdc++.so.6.0.3)
==28336== by 0x85530D6:
__static_initialization_and_destruction_0(int, int)
(MachineFunction.cpp:40)
==28336== by 0x855311D:
_GLOBAL__I__ZN4llvm19MachineFunctionPass6virtfnEv
(MachineFunction.cpp:484)
==28336== by 0x86E1A37:
(within /home/asl/proj/llvm/build_debug/Debug/examples/JITLeak)
==28336== by 0x8355D50:
(within /home/asl/proj/llvm/build_debug/Debug/examples/JITLeak)
==28336== by 0x86E197A: __libc_csu_init
(in /home/asl/proj/llvm/build_debug/Debug/examples/JITLeak)
==28336== by 0x418E7F3: (below main) (in /lib/libc-2.5.so)

clattner · July 15, 2007, 7:33pm

First, I'm not sure if deleting the ExecutionEngine is all I need to
clean-up... so I started with a minimal test just to check

Is this llvm 2.0 or llvm svn head? Several minor memory leaks have been fixed since llvm 2.0.

-Chris

int main( int argc, char **argv ){
    while( true ){
        Module *M = new Module("M");
        Function *F = cast<Function>(M->getOrInsertFunction("F",
Type::Int32Ty, (Type*)0));
        BasicBlock *BB = new BasicBlock("BB",F);
        new ReturnInst( ConstantInt::get(Type::Int32Ty,1), BB );
        std::vector<GenericValue> Args(0);
        ExistingModuleProvider *MP = new ExistingModuleProvider(M);
        ExecutionEngine *EE = ExecutionEngine::create(MP, false);
        GenericValue GV = EE->runFunction(F, Args);
        delete EE;
    }
}

The memory goes up, and OS X leaks command returns pages of:

Leak: 0x011036b0 size=16
        0x00000000 0x00000000 0x00000000
0x01102988 .............)..
Leak: 0x011036a0 size=16
        0x00000000 0x00000000 0x00000000
0x01102988 .............)..
Leak: 0x01103680 size=16
        0x00000000 0x00000000 0x00000000
0x01102c78 ............x,..
Leak: 0x01103670 size=16
        0x00000000 0x00000000 0x00000000 0x01102728 ............
('..
Leak: 0x01103640 size=16
        0x00000000 0x00000000 0x00000000
0x01102978 ............x)..
Leak: 0x01103630 size=16
        0x00000000 0x00000000 0x00000000
0x01103368 ............h3..
Leak: 0x01103620 size=16
        0x00000000 0x00000000 0x00000000
0x01102a58 ............X*..
Leak: 0x01103610 size=16
        0x00000000 0x00000000 0x00000000
0x01103368 ............h3..

So all I know is that It's a size 16 leak *grin*.

But MallocDebug reports as leaks

2.4M start
2.4M operator new(unsigned long)
2.4M MDmalloc
2.4M main
2.4M llvm::PMDataManager::add(llvm::Pass*, bool)
2.4M llvm::LoopPass::assignPassManager(llvm::PMStack&,
llvm::PassManagerType)
2.4M llvm::LLVMTargetMachine::addPassesToEmitMachineCode
(llvm::FunctionPassManager&, llvm::MachineCodeEmitter&, bool)
2.4M llvm::JIT::JIT[in-charge](llvm::ModuleProvider*,
llvm::TargetMachine&, llvm::TargetJITInfo&)
2.4M llvm::JIT::create(llvm::ModuleProvider*, std::basic_string<char,
std::char_traits<char>, std::allocator<char> >*)
2.4M llvm::FunctionPass::assignPassManager(llvm::PMStack&,
llvm::PassManagerType)
2.4M llvm::ExecutionEngine::create(llvm::ModuleProvider*, bool,
std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
2.4M _start
2.4M 0x1
2.4M llvm::PMTopLevelManager::schedulePass(llvm::Pass*)
2.4M llvm::FunctionPassManagerImpl::addTopLevelPass(llvm::Pass*)

I hope this is usefull, I'm pretty new to OS X.

Cheers, Paolo Invernizzi

You can find out what exactly leaks with the help of valgrind.

It seems, that Paolo is on Mac OS X. No valgrind there

All is not lost…

http://developer.apple.com/documentation/Performance/Conceptual/
ManagingMemory/Articles/FindingLeaks.html

— Gordon

_______________________________________________
LLVM Developers mailing list
LLVMdev@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

_______________________________________________
LLVM Developers mailing list
LLVMdev@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-Chris

Paolo_Invernizzi · July 15, 2007, 7:11pm

Chris,

It's the 2.0. I'll try the head as soon as I can (I installed the 2.0 via macports).
All started because I was not sure if deleting the ExecutionEngine is all I have to do to cleanup the LLVM stuff.

Cheers, Paolo