Traversing the (macro-expanded) tokens that make up a Decl?

My problem is as follows: I have a FunctionDecl d for which I know that d->isConstexpr(), so somewhere in d->getSourceRange() the token "constexpr" should appear. Now, I want to know whether that "constexpr" is from the expansion of a specific object-like macro

   #define M constexpr

(whose use could be nested in some other macros, so it's important here to look into macro invocations in general).

I'm trying to traverse d->getSourceRange() with Lexer::MeasureTokenLength/SourceManager::getCharacterData to get at a token's content, then SourceLocation::getLocWithOffset to get at the next token. Which works more-or-less well. However:

* When d->getSourceRange() happens to start off within a macro expansion, I could fall off its end before reaching the declaration's end. This one is easy, I can use SourceManager::isAtEndOfImmediateMacroExpansion to climb back out of the macro expansion and continue traversal one level further out.

* But when the traversal reaches a macro invocation, I get the token of that invocation (multiple tokens in case of a function-like macro with its parentheses and arguments), without a clue that it is a macro and without a way to step into its expansion. Am I missing anything, or is there just no way to do that?

(I find at least two unanswered questions on stackoverflow that look like they ask the same question, <libclang - How to get tokens for a declaration using Clang C++ API? - Stack Overflow; and <http://stackoverflow.com/questions/36235445/clang-getting-preprocessed-characters-with-sourcelocation-sourcerange&gt;\.\)

You might try to make a lexer to Lex(token) for you until you've reached
the end of the range, rather than navigating by source locations. Though
actually I think we should make the AST record the location of the
`constexpr` keyword, so that applications don't have to jump through these
hoops.

-- James

    My problem is as follows: I have a FunctionDecl d for which I know
    that d->isConstexpr(), so somewhere in d->getSourceRange() the token
    "constexpr" should appear. Now, I want to know whether that
    "constexpr" is from the expansion of a specific object-like macro

      #define M constexpr

    (whose use could be nested in some other macros, so it's important
    here to look into macro invocations in general).

    I'm trying to traverse d->getSourceRange() with
    Lexer::MeasureTokenLength/SourceManager::getCharacterData to get at
    a token's content, then SourceLocation::getLocWithOffset to get at
    the next token. Which works more-or-less well. However:

    * When d->getSourceRange() happens to start off within a macro
    expansion, I could fall off its end before reaching the
    declaration's end. This one is easy, I can use
    SourceManager::isAtEndOfImmediateMacroExpansion to climb back out of
    the macro expansion and continue traversal one level further out.

    * But when the traversal reaches a macro invocation, I get the token
    of that invocation (multiple tokens in case of a function-like macro
    with its parentheses and arguments), without a clue that it is a
    macro and without a way to step into its expansion. Am I missing
    anything, or is there just no way to do that?

(Another issue to watch out for would be preprocessing directives, e.g., skipping over conditional groups when encountering an #else line.)

You might try to make a lexer to Lex(token) for you until you've reached
the end of the range, rather than navigating by source locations.

But I assume I'd need to re-lex all from the beginning, so that the lexer has an accurate idea of what macros are currently defined and whether we're in an #if group (so it wouldn't fail with an error if it encounters a---semingly unmatched---#endif line, if that actually is an issue).

Though actually I think we should make the AST record the location of
the `constexpr` keyword, so that applications don't have to jump through
these hoops.

Yeah, I guess I understand now why new "get SourceLocation of ..." functions keep appearing in include/clang/AST/ as needed (e.g., FunctionDecl::getExceptionSpecSourceRange). But would we want to track the locations of e.g. all specifiers in Decl instances? (Or would that be considered making such instances "too fat"?)

Anyway, my immediate need for the original issue is meanwhile gone again, so I at least won't pursue this further for now.