Heap problems with 3.8.0rc2 in combination with vs2015 sp1

I recently upgraded from llvm 3.7.1 to a pre release of llvm (3.8.0rc2) in order to test some issues regarding bug 24233.
After upgrading I starting to see heap corruption messages in vs 2015 sp1 when my program exits.
“HEAP[ConsoleEngine.exe]: Invalid address specified to RtlValidateHeap( 0000000000290000, 0000000000318698 )”

Initially I only got it in Release build. Debug build seems to be fine, but recently I also got them in Debug build.
I traced it down how to trigger it in Debug Mode. It happens when I put the ExecutionEngine creation in the constructor in the cpp file.
If I put it in the h file then there’s no problem. This leads me to believe that it is probably not my code that is causing it since I never seem to have
this issue in the past (3.6.x - 3.7.x). I’ve include the stack traces below this email.

It fails when the dtor is called for the first module. MCJIT only has 2 modules, the first one only has function declarations. The second one is empty. No codegen is taking place.

I googled for the error message and it seems to be happen when you delete a resource twice.
Did something change for 3.8.0 regarding to function decl. creation in a module? Any tips on how to trace it down further? I can’t seem to make a small example demonstration the problem. It only seems to be triggered in a larger project (again: nearly code works fine in 3.6.x-3.7.x)

– Release mode stack trace
ntdll.dll!RtlReportCriticalFailure() Unknown
ntdll.dll!RtlpReportHeapFailure() Unknown
ntdll.dll!RtlpHeapHandleError() Unknown
ntdll.dll!RtlpLogHeapFailure() Unknown
ntdll.dll!string “Enabling heap debug options\n”() Unknown
ucrtbase.dll!free() Unknown
ConsoleEngine.exe!llvm::Function::vector deleting destructor'(unsigned int) C++ ConsoleEngine.exe!llvm::Module::~Module(void) C++ ConsoleEngine.exe!llvm::MCJIT::OwningModuleContainer::freeModulePtrSet(class llvm::SmallPtrSet<class llvm::Module *,4> &) C++ ConsoleEngine.exe!llvm::MCJIT::~MCJIT(void) C++ ConsoleEngine.exe!llvm::MCJIT::vector deleting destructor’(unsigned int) C++

– Debug Mode stack trace
ntdll.dll!RtlpBreakPointHeap() Unknown
ntdll.dll!string “Enabling heap debug options\n”() Unknown
ntdll.dll!RtlValidateHeap() Unknown
KernelBase.dll!HeapValidate() Unknown
ucrtbased.dll!_CrtIsValidHeapPointer() Unknown
ucrtbased.dll!_calloc_base() Unknown
ucrtbased.dll!_free_dbg() Unknown
ConsoleEngine.exe!operator delete(void * block) Line 21 C++
ConsoleEngine.exe!llvm::User::operator delete(void * Usr) Line 195 C++
ConsoleEngine.exe!llvm::Function::scalar deleting destructor'(unsigned int) C++ ConsoleEngine.exe!llvm::ilist_node_traits<llvm::Function>::deleteNode(llvm::Function * V) Line 160 C++ ConsoleEngine.exe!llvm::iplist<llvm::Function,llvm::SymbolTableListTraits<llvm::Function> >::erase(llvm::ilist_iterator<llvm::Function> where) Line 519 C++ ConsoleEngine.exe!llvm::iplist<llvm::Function,llvm::SymbolTableListTraits<llvm::Function> >::erase(llvm::ilist_iterator<llvm::Function> first, llvm::ilist_iterator<llvm::Function> last) Line 601 C++ ConsoleEngine.exe!llvm::iplist<llvm::Function,llvm::SymbolTableListTraits<llvm::Function> >::clear() Line 605 C++ ConsoleEngine.exe!llvm::Module::~Module() Line 61 C++ ConsoleEngine.exe!llvm::Module::scalar deleting destructor’(unsigned int) C++
ConsoleEngine.exe!llvm::MCJIT::OwningModuleContainer::freeModulePtrSet(llvm::SmallPtrSet<llvm::Module *,4> & MPS) Line 175 C++
ConsoleEngine.exe!llvm::MCJIT::OwningModuleContainer::~OwningModuleContainer() Line 82 C++
ConsoleEngine.exe!llvm::MCJIT::~MCJIT() Line 102 C++
ConsoleEngine.exe!llvm::MCJIT::`scalar deleting destructor’(unsigned int) C++
ConsoleEngine.exe!std::default_deletellvm::ExecutionEngine::operator()(llvm::ExecutionEngine * _Ptr) Line 1195 C++
ConsoleEngine.exe!std::unique_ptr<llvm::ExecutionEngine,std::default_deletellvm::ExecutionEngine >::~unique_ptr<llvm::ExecutionEngine,std::default_deletellvm::ExecutionEngine >() Line 1398 C++

I downloaded 3.8.0rc3 and I also have it in 3.8.0rc3.
I did set a data access breakpoint on the first function ptr that causes the invalid heap. This would allow me to break whenever someone is touching that address. It did not show double deletes during debugging. Further more I managed to narrow it down to 2 function calls:

// stupid code, but its just for triggering heap error
auto func = llvm::Function::Create(…);
if (func) func->eraseFromParent();

The erase from parent triggers the invalid heap. I used appverif.exe and gflags.exe (page heap checks) but they could not find anything.
The Function::Create is just a wrapper for new Function(). I also traced into this call, but did not see anything weird. It returns a valid ptr.
Somehow the ptr returned by the new seems to be broken. If i’m not mistaken the new operator is overloaded by llvm? I could not try to use the default new Function() since its constructor is not accessible from outside.

I’m kinda lost where to look next. There’s not much that could go wrong with creating a function and deleting it?
I’m using visual studio 2015 update1, and could not find any known issue regarding heaps. I’m at a point where I’m suspecting my compiler.
But its weird that the same compiler works fine with 3.7.1

any ideas would be appreciated

I made the llvm::Function() constructor public (for testing purpose) and used the non-overloaded new.

auto func = ::new llvm::Function(…)

if (func) func->eraseFromParent();

And the heap corruption is gone! Did something changed in llvm::User::new between 3.7.1 and 3.8.0 ?
I found a bug in llvm ?

I found the root cause, but I don’t know what’s the best approach to fix it.

Under windows, 64 bit, when a function is created the void User::operator new(size_t Size) operator allocates space + Use.
In the Use* the HasHungOffUses is set to true. So the ptr to the use* is returned as new object. This ptr is NOT the ptr that was allocated by the system. For that ptr you need ptr - word length. It’s important that we use free on the original ptr.

The void User::operator delete(void Usr) deletes the use when dtor is called. However, tracing through the Function() ctor I see that the
HasHungOffUses is set to False (!). And when we arrive at the User::delete, the HasHungOffUses has the wrong value causing it to flow into the else part.

void User::operator delete(void Usr) {
// Hung off uses use a single Use
before the User, while other subclasses
// use a Use allocated prior to the user.
User *Obj = static_cast<User *>(Usr);
if (Obj->HasHungOffUses) {
assert(!Obj->HasDescriptor && “not supported!”);

Use **HungOffOperandList = static_cast<Use **>(Usr) - 1;
// drop the hung off uses.
Use::zap(*HungOffOperandList, HungOffOperandList + Obj->NumUserOperands,
Delete */ true);
::operator delete(HungOffOperandList);
} else if (Obj->HasDescriptor) {
Use *UseBegin = static_cast<Use >(Usr) - Obj->NumUserOperands;
Use::zap(UseBegin, UseBegin + Obj->NumUserOperands, /
Delete */ false);

auto *DI = reinterpret_cast<DescriptorInfo *>(UseBegin) - 1;
uint8_t *Storage = reinterpret_cast<uint8_t *>(DI) - DI->SizeInBytes;
::operator delete(Storage);
} else {
Use *Storage = static_cast<Use >(Usr) - Obj->NumUserOperands;
Use::zap(Storage, Storage + Obj->NumUserOperands,
Delete */ false);
::operator delete(Storage);

The storage address is computed wrongly in this case, it’s off by 1 word length. And so visual studio sees that I’m not freeing the org allocated ptr and throws heap error. During debug when I force HasHungOffUses = true, everything is fine and the correct original ptr is computed and freed. So I’m guessing that that value should have stayed true.

I also have set a data breakpoint on HashHungOffUses when it’s accessed and did noticed that

ConsoleEngine.exe!llvm::Function::__autoclassinit2(unsigned __int64) C++
ConsoleEngine.exe!llvm::Function::Create(llvm::FunctionType * Ty, llvm::GlobalValue::LinkageTypes Linkage, const llvm::Twine & N, llvm::Module * M) Line 116 C++

caused it to be set to False. So do I need to trace into all the Function bases classes and find out where it is set to false? or do I need to fix the Storage address computation in the else part?


Turns out llvm initializes memory before the constructor is invoked. Visual studio has /sdl by default on, and the __autoclassinit2 will zero the memory before the constructor is reached causing all the setters to be default zero. which clears the hashungoff var and causing the delete to flow in wrong part.

When /sdl is enabled, the compiler generates code to perform these checks at run time:
— Performs class member initialization. Automatically initializes all class members to zero on object instantiation (before the constructor runs). This helps prevent the use of uninitialized data associated with class members that the constructor does not explicitly initialize.

Disabling the /SDL switch and everything works as intended. It might be handy to put this in the windows docs since it costed me quite some time to discover.

This is for reference to other users that are having heap issues:

Thank you for your investigation. Do maybe have a suggestion for a
patch that would make LLVM compatible with the /sdl switch?



I’ve been looking for a way to do it cleanly. But we don’t have an event/trigger when the constructor has been executed.
Also, apparently llvm depends on initializing memory before the constructor has been executed, so it’s best to disable the /sdl since the same construct might be used elsewhere in the llvm source.