JIT and atexit crash

Hi everyone, hi Lang,

I have a question about Orc JIT (and I think LLVM JIT in general).

I am jitting some C++ code that calls atexit and registers a function at some address 0xdeadbeef. Everything is fine, but when the host program (the one that JITs C++ code) shuts down, then I see a crash:

    error: memory read failed for 0xdeadbeef

With the following backtrace:

  * frame #0: 0x000000010b70b7a0
    frame #1: 0x00007fff8ae0c17f libsystem_c.dylib`__cxa_finalize_ranges + 339
    frame #2: 0x00007fff8ae0c4b2 libsystem_c.dylib`exit + 55
    frame #3: 0x00007fff8ad7723c libdyld.dylib`start + 8

After some debugging I think I understand what goes wrong.
Here is my hypothesis:

JIT allocates and maps some memory for the execution. Some function X at address 0xdeadbeef is part of this memory.
JIT calls a code that passes the X to atexit.
JIT deallocates and unmaps the memory used for execution (either via objectLayer.removeObjectSet or by calling JIT's destructors)
atexit (cxa_finalize_ranges) calls the X at 0xdeadbeef which does not belong to 'us' anymore, which leads to the crash.

Given that my assumption is correct what can we do about this? Is there anything that can be done to cover this case inside of the Orc engine?

Currently, I worked around this problem by resolving atexit to my custom function that does nothing.

  RuntimeDyld::SymbolInfo findSymbol(const std::string &Name) {
    if (Name == "_atexit") {
      return findSymbol("mull_custom_test_at_exit");
    }

    if (auto SymAddr = RTDyldMemoryManager::getSymbolAddressInProcess(Name))
      return RuntimeDyld::SymbolInfo(SymAddr, JITSymbolFlags::Exported);

    return RuntimeDyld::SymbolInfo(nullptr);
  }

  extern "C" int mull_custom_test_at_exit(void (* p)(void)) {
    return atexit(p);
  }

The solution removes the crash, but it is not optimal.

Best regards,
Alex.

JIT allocates and maps some memory for the execution. Some function X at address 0xdeadbeef is part of this memory.
JIT calls a code that passes the X to atexit.
JIT deallocates and unmaps the memory used for execution (either via objectLayer.removeObjectSet or by calling JIT's destructors)
atexit (cxa_finalize_ranges) calls the X at 0xdeadbeef which does not belong to 'us' anymore, which leads to the crash.

Sounds plausible.

Given that my assumption is correct what can we do about this? Is there
anything that can be done to cover this case inside of the Orc engine?

It's not the job of the Orc engine. Just don't use atexit.

Joerg

It's not the job of the Orc engine.

I could argue about this, but I won’t :slight_smile:

Just don't use atexit.

The problem is that I run third-party programs. I cannot control them.

Transform the atexit into equivalent code you can control, run it before the destructors of the JIT engine run?

Maybe the easiest workaround would be overriding symbol resolution for the function name and redirect it to your own version in static code (and hope it has no bad side effect on your use case).

I think when running 3rd party code, the only way to definitely avoid this kind of trouble is to never deallocate any code or data sections. Doug Binks mentioned that too in his cppcast about Runtime Compiled C++

Hi,

Not sure whether this matches your use case, but the Orc-based JIT used in LLI appears to be using llvm::orc::LocalCXXRuntimeOverrides () to override __cxa_atexit:

Best,
Matt

Thank you all for your input and your ideas.

Matt, this is very helpful. I used to do very similar thing myself, I’m glad that I can replace my code with class.

I am now curious whether we can include _at_exit into LocalCXXRuntimeOverrides or it should go into another place?

Lang, what do you think?

Hi,

Sorry for being late to this discussion. We had to solve this exact
problem. Specifically, our problem was on Windows where code compiled
using "clang -emit-llvm" calls atexit() to register the destructor of
static objects. We link-in runtime support code compiled this way along
with jitted code to create a module that we execute.

Here's our solution if this can help anyone. I can't paste the entire
source code due to copyright constraints. But, it should be easy to adapt
these pieces of code into something that can be compiled in a standalone
fashion...

Don't hesitate if you need help to get this working. It would be nice if
running Jitted code would require less scaffolding. :slight_smile:

Cheers,
Benoit

#ifdef PEPTIDE_WINDOWS
extern "C" void __chkstk();
#else
extern "C" int __cxa_atexit(void (*func) (void *), void * arg, void *
dso_handle);
#endif

namespace {

using namespace llvm;
using namespace Amino;

// Forward declaration of _Amino_atexit
extern "C" int _Amino_atexit(void (*cb)(), void** cbList);

/// \brief Pre-defined symbol cache entry for the internal Amino symbols
used
/// during jitting.
class SymbolCache {
private:
    /*----- types -----*/

    struct Entry {
        llvm::StringRef m_name;
        uint64_t m_addr;
    };

    using HashTable = ...;

    /*----- member functions -----*/

    template <typename Func>
    void insert(const char* name, Func* func) {
        m_table.insert(Entry(name, func));
    }

    /*----- static data members -----*/

    /// Hash table
    HashTable<Entry, SetTraits> m_table;

public:

    /*----- member functions -----*/

    // Constructor
    SymbolCache()
        : // Reserve enough space to avoid having to rehash while
filling-in
           // the table!
          m_table(0, 64) {
        insert("_Amino_atexit" ,
_Amino_atexit);
        insert("_Amino_dbgPrint" ,
_Amino_dbgPrint);
#ifdef PEPTIDE_WINDOWS
        insert("__chkstk" , __chkstk);
#else
        insert("__cxa_atexit" ,
__cxa_atexit);
#endif
    }

    uint64_t find(llvm::StringRef name) const {
        auto it = m_table.find(Entry(name));
        if (it == m_table.end()) return uint64_t(0);
        return it->m_addr;
    }
};

// Reserve enough space to avoid having to rehash while filling-in the
table!
const SymbolCache g_theSymbolCache;

//=========================================================================