Load saved ast file crash libclang 3.2

Hi, all.

I am writing a tool to find references in C++ program based on libclang 3.2 python binding.
The parsed program include a lot of std libraries, so the parsing speed is slow, and I try to use TranslationUnit.save/load function to speed it up.

Use the following C++ code as example:

#include “std_all.h”

class Test
{
public:
void foo();
};

I can parse it and travese it with the cursor returned by TranslationUnit.from_source.
But when I load back through an ast file(5.2M) and try to travese it, i.e. walk the whole ast, the python program crashed in libclang.
Here’s the crash output:


python: /home/users/Downloads/llvm-3.2.src/tools/clang/include/clang/Basic/IdentifierTable.h:168: void clang::IdentifierInfo::RevertTokenIDToIdentifier(): Assertion `TokenID != tok::identifier && “Already at tok::identifier”’ failed.

Aborted (core dumped)

The following is the backtrace of the crash:

#0 0x0000003bc1632885 in raise () from /lib64/libc.so.6
#1 0x0000003bc1634065 in abort () from /lib64/libc.so.6
#2 0x0000003bc162b9fe in __assert_fail_base () from /lib64/libc.so.6
#3 0x0000003bc162bac0 in __assert_fail () from /lib64/libc.so.6
#4 0x00007ffe098da880 in clang::IdentifierInfo::RevertTokenIDToIdentifier() ()
from /usr/local/lib/libclang.so
#5 0x00007ffe098bac7b in clang::serialization::reader::ASTIdentifierLookupTrait::ReadData(std::pair<char const*, unsigned int> const&, unsigned char const*, unsigned int) () from /usr/local/lib/libclang.so
#6 0x00007ffe098e1d09 in clang::OnDiskChainedHashTableclang::serialization::reader::ASTIdentifierLookupTrait::iterator::operator*() const ()
from /usr/local/lib/libclang.so
#7 0x00007ffe098be540 in (anonymous namespace)::IdentifierLookupVisitor::visit(clang::serialization::ModuleFile&, void*) () from /usr/local/lib/libclang.so
#8 0x00007ffe099ac9eb in clang::serialization::ModuleManager::visit(bool ()(clang::serialization::ModuleFile&, void), void*) ()
from /usr/local/lib/libclang.so
#9 0x00007ffe098d40fe in clang::ASTReader::get(char const*, char const*) ()
from /usr/local/lib/libclang.so
#10 0x00007ffe098dceff in clang::ASTReader::get(llvm::StringRef) ()
from /usr/local/lib/libclang.so
#11 0x00007ffe0951c314 in clang::IdentifierTable::get(llvm::StringRef) ()
from /usr/local/lib/libclang.so
#12 0x00007ffe098d55a7 in clang::ASTReader::DecodeIdentifierInfo(unsigned int)
() from /usr/local/lib/libclang.so
#13 0x00007ffe098dcf67 in clang::ASTReader::GetIdentifierInfo(clang::serialization::ModuleFile&, llvm::SmallVector<unsigned long, 64u> const&, unsigned int&)
() from /usr/local/lib/libclang.so
#14 0x00007ffe098d5da3 in clang::ASTReader::ReadDeclarationName(clang::serialization::ModuleFile&, llvm::SmallVector<unsigned long, 64u> const&, unsigned int&)
() from /usr/local/lib/libclang.so
#15 0x00007ffe0990cbbc in clang::ASTDeclReader::VisitNamedDecl(clang::NamedDecl*) () from /usr/local/lib/libclang.so
#16 0x00007ffe099131c2 in clang::ASTDeclReader::VisitTemplateDecl(clang::TemplateDecl*) () from /usr/local/lib/libclang.so
#17 0x00007ffe099133ea in clang::ASTDeclReader::VisitRedeclarableTemplateDecl(clang::RedeclarableTemplateDecl*) () from /usr/local/lib/libclang.so
#18 0x00007ffe09913492 in clang::ASTDeclReader::VisitClassTemplateDecl(clang::ClassTemplateDecl*) () from /usr/local/lib/libclang.so
#19 0x00007ffe09924f40 in clang::DeclVisitor<clang::ASTDeclReader, void>::Visit(clang::Decl*) () from /usr/local/lib/libclang.so
#20 0x00007ffe0990c3e7 in clang::ASTDeclReader::Visit(clang::Decl*) ()
from /usr/local/lib/libclang.so
#21 0x00007ffe0991d848 in clang::ASTReader::ReadDeclRecord(unsigned int) ()
from /usr/local/lib/libclang.so
#22 0x00007ffe098d1277 in clang::ASTReader::GetDecl(unsigned int) ()
from /usr/local/lib/libclang.so
#23 0x00007ffe098dce57 in clang::ASTReader::GetLocalDecl(clang::serialization::ModuleFile&, unsigned int) () from /usr/local/lib/libclang.so
#24 0x00007ffe098d171e in (anonymous namespace)::FindExternalLexicalDeclsVisitor::visit(clang::serialization::ModuleFile&, bool, void*) ()
from /usr/local/lib/libclang.so
#25 0x00007ffe099acd52 in visitDepthFirst(clang::serialization::ModuleFile&, bool ()(clang::serialization::ModuleFile&, bool, void), void*, llvm::SmallPtrSet<clang::serialization::ModuleFile*, 4u>&) () from /usr/local/lib/libclang.so
#26 0x00007ffe099acde8 in clang::serialization::ModuleManager::visitDepthFirst(bool ()(clang::serialization::ModuleFile&, bool, void), void*) ()
from /usr/local/lib/libclang.so
#27 0x00007ffe098d17e8 in clang::ASTReader::FindExternalLexicalDecls(clang::DeclContext const*, bool ()(clang::Decl::Kind), llvm::SmallVectorImplclang::Decl*&) () from /usr/local/lib/libclang.so
#28 0x00007ffe09e6849a in clang::ExternalASTSource::FindExternalLexicalDecls(clang::DeclContext const
, llvm::SmallVectorImplclang::Decl*&) ()
from /usr/local/lib/libclang.so
#29 0x00007ffe09e666c2 in clang::DeclContext::LoadLexicalDeclsFromExternalStorage() const () from /usr/local/lib/libclang.so
#30 0x00007ffe09e66a88 in clang::DeclContext::decls_begin() const ()
from /usr/local/lib/libclang.so
#31 0x00007ffe0938216d in clang::cxcursor::CursorVisitor::VisitDeclContext(clang::DeclContext*) () from /usr/local/lib/libclang.so
#32 0x00007ffe093846bf in clang::cxcursor::CursorVisitor::VisitNamespaceDecl(clang::NamespaceDecl*) () from /usr/local/lib/libclang.so
#33 0x00007ffe093ac330 in clang::DeclVisitor<clang::cxcursor::CursorVisitor, bool>::Visit(clang::Decl*) () from /usr/local/lib/libclang.so
#34 0x00007ffe09381a05 in clang::cxcursor::CursorVisitor::VisitChildren(CXCursor) () from /usr/local/lib/libclang.so
#35 0x00007ffe0938ca66 in clang_visitChildren ()
from /usr/local/lib/libclang.so
#36 0x00007ffe0ba29dac in ffi_call_unix64 () from /usr/lib64/libffi.so.5
#37 0x00007ffe0ba29b34 in ffi_call () from /usr/lib64/libffi.so.5
#38 0x00007ffe0bc3d004 in _CallProc ()
from /usr/lib64/python2.6/lib-dynload/_ctypes.so
#39 0x00007ffe0bc36312 in ?? ()
from /usr/lib64/python2.6/lib-dynload/_ctypes.so
#40 0x0000003bd2e44043 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
#41 0x0000003bd2eddd84 in PyEval_EvalFrameEx ()

Note that for small program, e.g. a cpp file including only a simple function definition, loading the saved ast file will not crash. I think this crash will only happen for big source file.

Is this a bug in clang 3.2? Has it been fixed in latest code? Or anyone can explain this?

Hi, all.

I am writing a tool to find references in C++ program based on libclang 3.2 python binding.
The parsed program include a lot of std libraries, so the parsing speed is slow, and I try to use TranslationUnit.save/load function to speed it up.

Use the following C++ code as example:

#include “std_all.h”

class Test
{
public:
void foo();
};

I can parse it and travese it with the cursor returned by TranslationUnit.from_source.
But when I load back through an ast file(5.2M) and try to travese it, i.e. walk the whole ast, the python program crashed in libclang.
Here’s the crash output:


python: /home/users/Downloads/llvm-3.2.src/tools/clang/include/clang/Basic/IdentifierTable.h:168: void clang::IdentifierInfo::RevertTokenIDToIdentifier(): Assertion `TokenID != tok::identifier && “Already at tok::identifier”’ failed.

Aborted (core dumped)

The following is the backtrace of the crash:

#0 0x0000003bc1632885 in raise () from /lib64/libc.so.6
#1 0x0000003bc1634065 in abort () from /lib64/libc.so.6
#2 0x0000003bc162b9fe in __assert_fail_base () from /lib64/libc.so.6
#3 0x0000003bc162bac0 in __assert_fail () from /lib64/libc.so.6
#4 0x00007ffe098da880 in clang::IdentifierInfo::RevertTokenIDToIdentifier() ()
from /usr/local/lib/libclang.so
#5 0x00007ffe098bac7b in clang::serialization::reader::ASTIdentifierLookupTrait::ReadData(std::pair<char const*, unsigned int> const&, unsigned char const*, unsigned int) () from /usr/local/lib/libclang.so
#6 0x00007ffe098e1d09 in clang::OnDiskChainedHashTableclang::serialization::reader::ASTIdentifierLookupTrait::iterator::operator*() const ()
from /usr/local/lib/libclang.so
#7 0x00007ffe098be540 in (anonymous namespace)::IdentifierLookupVisitor::visit(clang::serialization::ModuleFile&, void*) () from /usr/local/lib/libclang.so
#8 0x00007ffe099ac9eb in clang::serialization::ModuleManager::visit(bool ()(clang::serialization::ModuleFile&, void), void*) ()
from /usr/local/lib/libclang.so
#9 0x00007ffe098d40fe in clang::ASTReader::get(char const*, char const*) ()
from /usr/local/lib/libclang.so
#10 0x00007ffe098dceff in clang::ASTReader::get(llvm::StringRef) ()
from /usr/local/lib/libclang.so
#11 0x00007ffe0951c314 in clang::IdentifierTable::get(llvm::StringRef) ()
from /usr/local/lib/libclang.so
#12 0x00007ffe098d55a7 in clang::ASTReader::DecodeIdentifierInfo(unsigned int)
() from /usr/local/lib/libclang.so
#13 0x00007ffe098dcf67 in clang::ASTReader::GetIdentifierInfo(clang::serialization::ModuleFile&, llvm::SmallVector<unsigned long, 64u> const&, unsigned int&)
() from /usr/local/lib/libclang.so
#14 0x00007ffe098d5da3 in clang::ASTReader::ReadDeclarationName(clang::serialization::ModuleFile&, llvm::SmallVector<unsigned long, 64u> const&, unsigned int&)
() from /usr/local/lib/libclang.so
#15 0x00007ffe0990cbbc in clang::ASTDeclReader::VisitNamedDecl(clang::NamedDecl*) () from /usr/local/lib/libclang.so
#16 0x00007ffe099131c2 in clang::ASTDeclReader::VisitTemplateDecl(clang::TemplateDecl*) () from /usr/local/lib/libclang.so
#17 0x00007ffe099133ea in clang::ASTDeclReader::VisitRedeclarableTemplateDecl(clang::RedeclarableTemplateDecl*) () from /usr/local/lib/libclang.so
#18 0x00007ffe09913492 in clang::ASTDeclReader::VisitClassTemplateDecl(clang::ClassTemplateDecl*) () from /usr/local/lib/libclang.so
#19 0x00007ffe09924f40 in clang::DeclVisitor<clang::ASTDeclReader, void>::Visit(clang::Decl*) () from /usr/local/lib/libclang.so
#20 0x00007ffe0990c3e7 in clang::ASTDeclReader::Visit(clang::Decl*) ()
from /usr/local/lib/libclang.so
#21 0x00007ffe0991d848 in clang::ASTReader::ReadDeclRecord(unsigned int) ()
from /usr/local/lib/libclang.so
#22 0x00007ffe098d1277 in clang::ASTReader::GetDecl(unsigned int) ()
from /usr/local/lib/libclang.so
#23 0x00007ffe098dce57 in clang::ASTReader::GetLocalDecl(clang::serialization::ModuleFile&, unsigned int) () from /usr/local/lib/libclang.so
#24 0x00007ffe098d171e in (anonymous namespace)::FindExternalLexicalDeclsVisitor::visit(clang::serialization::ModuleFile&, bool, void*) ()
from /usr/local/lib/libclang.so
#25 0x00007ffe099acd52 in visitDepthFirst(clang::serialization::ModuleFile&, bool ()(clang::serialization::ModuleFile&, bool, void), void*, llvm::SmallPtrSet<clang::serialization::ModuleFile*, 4u>&) () from /usr/local/lib/libclang.so
#26 0x00007ffe099acde8 in clang::serialization::ModuleManager::visitDepthFirst(bool ()(clang::serialization::ModuleFile&, bool, void), void*) ()
from /usr/local/lib/libclang.so
#27 0x00007ffe098d17e8 in clang::ASTReader::FindExternalLexicalDecls(clang::DeclContext const*, bool ()(clang::Decl::Kind), llvm::SmallVectorImplclang::Decl*&) () from /usr/local/lib/libclang.so
#28 0x00007ffe09e6849a in clang::ExternalASTSource::FindExternalLexicalDecls(clang::DeclContext const
, llvm::SmallVectorImplclang::Decl*&) ()
from /usr/local/lib/libclang.so
#29 0x00007ffe09e666c2 in clang::DeclContext::LoadLexicalDeclsFromExternalStorage() const () from /usr/local/lib/libclang.so
#30 0x00007ffe09e66a88 in clang::DeclContext::decls_begin() const ()
from /usr/local/lib/libclang.so
#31 0x00007ffe0938216d in clang::cxcursor::CursorVisitor::VisitDeclContext(clang::DeclContext*) () from /usr/local/lib/libclang.so
#32 0x00007ffe093846bf in clang::cxcursor::CursorVisitor::VisitNamespaceDecl(clang::NamespaceDecl*) () from /usr/local/lib/libclang.so
#33 0x00007ffe093ac330 in clang::DeclVisitor<clang::cxcursor::CursorVisitor, bool>::Visit(clang::Decl*) () from /usr/local/lib/libclang.so
#34 0x00007ffe09381a05 in clang::cxcursor::CursorVisitor::VisitChildren(CXCursor) () from /usr/local/lib/libclang.so
#35 0x00007ffe0938ca66 in clang_visitChildren ()
from /usr/local/lib/libclang.so
#36 0x00007ffe0ba29dac in ffi_call_unix64 () from /usr/lib64/libffi.so.5
#37 0x00007ffe0ba29b34 in ffi_call () from /usr/lib64/libffi.so.5
#38 0x00007ffe0bc3d004 in _CallProc ()
from /usr/lib64/python2.6/lib-dynload/_ctypes.so
#39 0x00007ffe0bc36312 in ?? ()
from /usr/lib64/python2.6/lib-dynload/_ctypes.so
#40 0x0000003bd2e44043 in PyObject_Call () from /usr/lib64/libpython2.6.so.1.0
#41 0x0000003bd2eddd84 in PyEval_EvalFrameEx ()

Note that for small program, e.g. a cpp file including only a simple function definition, loading the saved ast file will not crash. I think this crash will only happen for big source file.

Is this a bug in clang 3.2? Has it been fixed in latest code? Or anyone can explain this?

I think this has been fixed on trunk.