[windows libclang] libclang fails to delete obsolete temporary .pch files

Clang Frontend
1

Hi,

I use clang_reparseTranslationUnit with CXTranslationUnit_PrecompiledPreamble flag on windows.

I investigated why libclang.dll fails to delete old pch files on windows.

Modules keep an active file mapping on file until they are destroyed. Later on, this still active mapping will cause DeleteFileW to fail (silently…) .

There is at least two cases when it happens:

The first one is when some preamble has changed and a new pch needs to replace the current one, CleanPreambleFile , which purpose is to destroy the now old pch file will be called before the corresponding module is destroyed when the mapping will be recycled in ASTUnit::Parse.

The second one is when we destroy the translation unit with clang_disposeTranslationUnit, CleanPreambleFile which purpose is to destroy the pch file, is called before the corresponding module is destroyed in ~ModuleFile.

Below are the corresponding callstacks.

  1. libclang.dll!`anonymous namespace’::OnDiskData::CleanPreambleFile() Line 175 C++

  2. libclang.dll!erasePreambleFile(const clang::ASTUnit * AU) Line 144 C++

  3. libclang.dll!clang::ASTUnit::getMainBufferWithPrecompiledPreamble(const clang::CompilerInvocation & PreambleInvocationIn, bool AllowRebuild, unsigned int MaxLines) Line 1511 C++

  4. libclang.dll!clang::ASTUnit::Reparse(std::pair<std::basic_string<char,std::char_traits,std::allocator >,llvm::PointerUnion<char const *,llvm::MemoryBuffer const *> > * RemappedFiles, unsigned int NumRemappedFiles) Line 2116 C++

  5. libclang.dll!clang_reparseTranslationUnit_Impl(void * UserData) Line 3008 C++

  6. libclang.dll!llvm::sys::fs::mapped_file_region::~mapped_file_region() Line 833 C++

  7. libclang.dll!`anonymous namespace’::MemoryBufferMMapFile::~MemoryBufferMMapFile() C++

  8. libclang.dll!anonymous namespace'::MemoryBufferMMapFile::scalar deleting destructor’(unsigned int) C++

  9. libclang.dll!llvm::OwningPtrllvm::MemoryBuffer::~OwningPtrllvm::MemoryBuffer() Line 45 C++

  10. libclang.dll!clang::serialization::ModuleFile::~ModuleFile() Line 58 C++

  11. libclang.dll!clang::serialization::ModuleFile::`scalar deleting destructor’(unsigned int) C++

  12. libclang.dll!clang::serialization::ModuleManager::~ModuleManager() Line 230 C++

  13. libclang.dll!clang::ASTReader::~ASTReader() Line 7638 C++

  14. libclang.dll!clang::ASTReader::`scalar deleting destructor’(unsigned int) C++

  15. libclang.dll!llvm::OwningPtrclang::ExternalASTSource::~OwningPtrclang::ExternalASTSource() Line 45 C++

  16. libclang.dll!clang::ASTContext::~ASTContext() Line 790 C++

  17. libclang.dll!clang::ASTContext::`scalar deleting destructor’(unsigned int) C++

  18. libclang.dll!llvm::RefCountedBaseclang::ASTContext::Release() Line 54 C++

  19. libclang.dll!llvm::IntrusiveRefCntPtrInfoclang::ASTContext::release(clang::ASTContext * obj) Line 89 C++

  20. libclang.dll!llvm::IntrusiveRefCntPtrclang::ASTContext::release() Line 178 C++

  21. libclang.dll!llvm::IntrusiveRefCntPtrclang::ASTContext::~IntrusiveRefCntPtrclang::ASTContext() Line 148 C++

  22. libclang.dll!llvm::IntrusiveRefCntPtrclang::ASTContext::operator=(llvm::IntrusiveRefCntPtrclang::ASTContext S) Line 145 C++

  23. libclang.dll!clang::ASTUnit::Parse(llvm::MemoryBuffer * OverrideMainBuffer) Line 1169 C++

  24. libclang.dll!clang::ASTUnit::Reparse(std::pair<std::basic_string<char,std::char_traits,std::allocator >,llvm::PointerUnion<char const *,llvm::MemoryBuffer const *> > * RemappedFiles, unsigned int NumRemappedFiles) Line 2125 C++

  25. libclang.dll!clang_reparseTranslationUnit_Impl(void * UserData) Line 3008 C++

  26. libclang.dll!`anonymous namespace’::OnDiskData::CleanPreambleFile() Line 175 C++

  27. libclang.dll!`anonymous namespace’::OnDiskData::Cleanup() Line 183 C++

  28. libclang.dll!removeOnDiskEntry(const clang::ASTUnit * AU) Line 154 C++

  29. libclang.dll!clang::ASTUnit::~ASTUnit() Line 249 C++

  30. libclang.dll!clang::ASTUnit::`scalar deleting destructor’(unsigned int) C++

  31. libclang.dll!clang_disposeTranslationUnit(CXTranslationUnitImpl * CTUnit) Line 2945 C++

  32. libclang.dll!llvm::sys::fs::mapped_file_region::~mapped_file_region() Line 833 C++

  33. libclang.dll!`anonymous namespace’::MemoryBufferMMapFile::~MemoryBufferMMapFile() C++

  34. libclang.dll!anonymous namespace'::MemoryBufferMMapFile::scalar deleting destructor’(unsigned int) C++

  35. libclang.dll!llvm::OwningPtrllvm::MemoryBuffer::~OwningPtrllvm::MemoryBuffer() Line 45 C++

  36. libclang.dll!clang::serialization::ModuleFile::~ModuleFile() Line 58 C++

  37. libclang.dll!clang::serialization::ModuleFile::`scalar deleting destructor’(unsigned int) C++

  38. libclang.dll!clang::serialization::ModuleManager::~ModuleManager() Line 230 C++

  39. libclang.dll!clang::ASTReader::~ASTReader() Line 7638 C++

  40. libclang.dll!clang::ASTReader::`scalar deleting destructor’(unsigned int) C++

  41. libclang.dll!llvm::OwningPtrclang::ExternalASTSource::~OwningPtrclang::ExternalASTSource() Line 45 C++

  42. libclang.dll!clang::ASTContext::~ASTContext() Line 790 C++

  43. libclang.dll!clang::ASTContext::`scalar deleting destructor’(unsigned int) C++

  44. libclang.dll!llvm::RefCountedBaseclang::ASTContext::Release() Line 54 C++

  45. libclang.dll!llvm::IntrusiveRefCntPtrInfoclang::ASTContext::release(clang::ASTContext * obj) Line 89 C++

  46. libclang.dll!llvm::IntrusiveRefCntPtrclang::ASTContext::release() Line 178 C++

  47. libclang.dll!llvm::IntrusiveRefCntPtrclang::ASTContext::~IntrusiveRefCntPtrclang::ASTContext() Line 148 C++

  48. libclang.dll!clang::ASTUnit::~ASTUnit() Line 274 C++

  49. libclang.dll!clang::ASTUnit::`scalar deleting destructor’(unsigned int) C++

  50. libclang.dll!clang_disposeTranslationUnit(CXTranslationUnitImpl * CTUnit) Line 2945 C++

To verify this, we made the following dirty fix that solved the problem for our limited usecase.

ASTUnit.cpp

static void unmapPreambleFile(const ASTUnit *AU, OnDiskData& D){
if (!D.PreambleFile.empty()) {
const ASTContext& Context = AU->getASTContext();
ASTReader& Reader = (ASTReader&)*Context.ExternalSource;
if (&Reader){
serialization::ModuleManager& ModuleMgr = Reader.getModuleManager();
ModuleMgr.removeModule(D.PreambleFile);
}
}
}

static void erasePreambleFile(const ASTUnit *AU) {
OnDiskData& D = getOnDiskData(AU);
unmapPreambleFile(AU, D);
D.CleanPreambleFile();
}

static void removeOnDiskEntry(const ASTUnit AU) {
// We require the mutex since we are modifying the structure of the
// DenseMap.
llvm::MutexGuard Guard(getOnDiskMutex());
OnDiskDataMap &M = getOnDiskDataMap();
OnDiskDataMap::iterator I = M.find(AU);
if (I != M.end()) {
OnDiskData pD = I->second;
unmapPreambleFile(AU, *pD);
pD->Cleanup();
delete pD;
M.erase(AU);
}
}

ModuleManager.cpp

void ModuleManager::removeModule(const std::string& FileName){
for (ModuleIterator M = Chain.begin(), MEnd = Chain.end(); M != MEnd; ++M) {
serialization::ModuleFile* pModuleFile = *M;
if (pModuleFile->FileName.compare(FileName)==0){
delete pModuleFile;
Chain.erase(M);
break;
}
}
}

2

Ultimately this boils down to the fact that you can unlink files you have open on non-Windows platforms and the fd will keep it alive. Maybe we should use FILE_SHARE_DELETE in llvm::sys::fs::openFileForRead(). Will that make our behavior consistent across platforms?