I’m having a bit of difficulty with the lexer. It’s been working dandy for me for the most part, but has some problems when lexing over macro expansions that are empty.
Defining this macro:
#ifndef MyMacro
#define MyMacro int g;
#endif
and lexing over this:
MyMacro int x;
is no problem. But defining the macro thusly:
#ifndef MyMacro
#define MyMacro
#endif
or:
#ifndef MyMacro
#define MyMacro /some comment/
#endif
causes a EXC_BAD_ACCESS here:
#0 clang::TokenLexer::isAtEnd at TokenLexer.h:122
#1 clang::TokenLexer::Lex at TokenLexer.cpp:279
#2 clang::Preprocessor::Lex at Preprocessor.h:262
#3 clang::Preprocessor::HandleMacroExpandedIdentifier at PPMacroExpansion.cpp:205
#4 clang::Preprocessor::HandleIdentifier at Preprocessor.cpp:532
#5 clang::Lexer::LexIdentifier at Lexer.cpp:534
#6 clang::Lexer::LexTokenInternal at Lexer.cpp:1314
#7 clang::Lexer::Lex at Lexer.h:141
(Actually, in the comment case, I’m not entirely sure if the comment gets expanded or simply removed by the preprocessor before expansion.)
This problem seems odd because the source file is getting parsed by clang correctly the first time around (for instance, I can run the clang driver and emit HTML, no problem). But when I make a new lexer (re-using the preprocessor from the original parsing), I get the above errors. Is there some lexer or preprocessor setting that I’m missing?
Thanks,
Emerson
I’m having a bit of difficulty with the lexer. It’s been working dandy for me for the most part, but has some problems when lexing over macro expansions that are empty.
Please paste the code you’re using to lex over this. The “clang -dump-tokens” option presumably works on this, right?
is no problem. But defining the macro thusly:
#ifndef MyMacro
#define MyMacro
#endif
or:
#ifndef MyMacro
#define MyMacro /some comment/
#endif
causes a EXC_BAD_ACCESS here:
#0 clang::TokenLexer::isAtEnd at TokenLexer.h:122
#1 clang::TokenLexer::Lex at TokenLexer.cpp:279
#2 clang::Preprocessor::Lex at Preprocessor.h:262
#3 clang::Preprocessor::HandleMacroExpandedIdentifier at PPMacroExpansion.cpp:205
#4 clang::Preprocessor::HandleIdentifier at Preprocessor.cpp:532
#5 clang::Lexer::LexIdentifier at Lexer.cpp:534
#6 clang::Lexer::LexTokenInternal at Lexer.cpp:1314
#7 clang::Lexer::Lex at Lexer.h:141
(Actually, in the comment case, I’m not entirely sure if the comment gets expanded or simply removed by the preprocessor before expansion.)
Are you sure you’re not lexing past EOF?
This problem seems odd because the source file is getting parsed by clang correctly the first time around (for instance, I can run the clang driver and emit HTML, no problem). But when I make a new lexer (re-using the preprocessor from the original parsing), I get the above errors. Is there some lexer or preprocessor setting that I’m missing?
when reusing a preprocessor like this, be aware that you’re starting out with the full t-u full of macro definitions already installed. This means that the macro is already defined, so the #ifndef block isn’t entered.
-Chris
I’m having a bit of difficulty with the lexer. It’s been working dandy for me for the most part, but has some problems when lexing over macro expansions that are empty.
Please paste the code you’re using to lex over this. The “clang -dump-tokens” option presumably works on this, right?
dump-tokens works. Here’s some crashing code:
Lexer* lexer = new Lexer(LocStart,*(provider->preprocessor()),startBuf,endBuf);
Token current;
do {
lexer->Lex(current);
}while(!current.is(tok::eof) && !current.is(tok::r_brace));
Works fine when the macro has something in it, crashes when it’s empty.
(Actually, in the comment case, I’m not entirely sure if the comment gets expanded or simply removed by the preprocessor before expansion.)
Are you sure you’re not lexing past EOF?
Pretty sure.
This problem seems odd because the source file is getting parsed by clang correctly the first time around (for instance, I can run the clang driver and emit HTML, no problem). But when I make a new lexer (re-using the preprocessor from the original parsing), I get the above errors. Is there some lexer or preprocessor setting that I’m missing?
when reusing a preprocessor like this, be aware that you’re starting out with the full t-u full of macro definitions already installed. This means that the macro is already defined, so the #ifndef block isn’t entered.
That makes sense.
Thanks,
e