Hi,
I am using clang::ToolInvocation class to compile some code in-memory:
clang::tooling::ToolInvocation ti
(
compilerArgs,
new clang::EmitBCAction(),
new clang::FileManager(clang::FileSystemOptions())
);
//filename is the name of the source file, e.g. “Somefile.cpp”
//sourcecode contains the source of the file
ti.mapVirtualFile( filename, sourcecode );
bool ret = ti.run();
In order to speed up compilation of several sources, I call the above code concurrently in separate threads, with each thread compiling its own source code. However, this does not work because clang::CompilerInstance calls llvm::Sys::RemoveFileOnSignal which in turn calls RegisterHandlers() which is not re-entrant.
Is there a way to make RegisterHandlers() re-entrant? I do call llvm::start_multi_threaded() before any of this, hoping that it will make llvm routines thread-safe, but to no avail.
Also, is there any other way of doing parallel compilation?
thx
Vikas.
Seems like it is a bug in either RemoveFileOnSignal(). For some
reason, the first one releases the mutex before calling the
RegisterHandlers().
bool llvm::sys::RemoveFileOnSignal(StringRef Filename,
std::string* ErrMsg) {
SignalsMutex.acquire();
...
SignalsMutex.release();
RegisterHandlers();
return false;
}
CC'ing Argyrios because this is important for libclang.
Argyrios: what do you think -- should we call RegisterHandlers() under a mutex?
Dmitri
Hi,
I am using clang::ToolInvocation class to compile some code in-memory:
clang::tooling::ToolInvocation ti
(
compilerArgs,
new clang::EmitBCAction(),
new clang::FileManager(clang::FileSystemOptions())
);
//filename is the name of the source file, e.g. “Somefile.cpp”
//sourcecode contains the source of the file
ti.mapVirtualFile( filename, sourcecode );
bool ret = ti.run();
In order to speed up compilation of several sources, I call the above code
concurrently in separate threads, with each thread compiling its own source
code. However, this does not work because clang::CompilerInstance calls
llvm::Sys::RemoveFileOnSignal which in turn calls RegisterHandlers() which
is not re-entrant.
Is there a way to make RegisterHandlers() re-entrant? I do call
llvm::start_multi_threaded() before any of this, hoping that it will make
llvm routines thread-safe, but to no avail.
Seems like it is a bug in either RemoveFileOnSignal(). For some
reason, the first one releases the mutex before calling the
RegisterHandlers().
bool llvm::sys::RemoveFileOnSignal(StringRef Filename,
std::string* ErrMsg) {
SignalsMutex.acquire();
…
SignalsMutex.release();
RegisterHandlers();
return false;
}
CC’ing Argyrios because this is important for libclang.
Argyrios: what do you think – should we call RegisterHandlers() under a mutex?
That, or get RegisterHandlers to use the mutex.
Also access to CallBacksToRun is not thread-safe (the whole file probably needs reviewing…)
But it’s not so clear to me what is the exact symptom, is there a crash occurring ? Vika, could you explain ?
Yes, there is a crash when lot of threads are used. I am using 60 threads to compile 60 source files and the crash happens 90% of the time. Here is the relevant stack trace:
(gdb) bt
#0 0x00007f30e11d9e4e in __libc_sigaction (sig=31, act=, oact=0x2c5ffa0)
at …/sysdeps/unix/sysv/linux/x86_64/sigaction.c:71
#1 0x0000000001c75636 in RegisterHandlers() ()
#2 0x0000000001c7579f in llvm::sys::RemoveFileOnSignal(llvm::StringRef, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >) ()
#3 0x0000000000ed3286 in clang::CompilerInstance::createOutputFile(llvm::StringRef, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&, bool, bool, llvm::StringRef, llvm::StringRef, bool, bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >) ()
#4 0x0000000000ed20c8 in clang::CompilerInstance::createOutputFile(llvm::StringRef, bool, bool, llvm::StringRef, llvm::StringRef, bool, bool) ()
#5 0x0000000000ed200b in clang::CompilerInstance::createDefaultOutputFile(bool, llvm::StringRef, llvm::StringRef) ()
#6 0x0000000000d166b9 in clang::CodeGenAction::CreateASTConsumer(clang::CompilerInstance&, llvm::StringRef) ()
#7 0x0000000000ef87a4 in clang::FrontendAction::CreateWrappedASTConsumer(clang::CompilerInstance&, llvm::StringRef) ()
#8 0x0000000000ef943e in clang::FrontendAction::BeginSourceFile(clang::CompilerInstance&, clang::FrontendInputFile const&) ()
#9 0x0000000000ed3e01 in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) ()
#10 0x000000000140dad9 in clang::tooling::FrontendActionFactory::runInvocation(clang::CompilerInvocation, clang::FileManager*, clang::DiagnosticConsumer*) ()
#11 0x000000000140d423 in clang::tooling::ToolInvocation::run() ()
Please let me know if you want me to create a working project to illustrate the crash.
thx
Vikas.