Keeping tokens in memory.

I need to keep all the raw tokens representing a file in memory.
But I could not understand where clang memorizes spelling and flags.
Just filling a vector of clang::Token does not work, this is what I am doing

void tokenize_whole_file(std::vector<clang::Token>& raw_tokens,
                         clang::FileID fileID) {

  clang::Lexer raw_lexer(fileID, source_manager(), lang_info());
  clang::Token current_token;



  while (raw_lexer.LexFromRawLexer(current_token),
         current_token.isNot(clang::tok::eof)) {
There is also a preprocessor correlated to the source manager and lang_info,
but later when I try to get the spelling or the flags of any token in the
raw_tokens vector it does not work.

I tried giving the Lexer a static duration, but it does not help.


This should work fine. The spelling is handled by the source location in the token. Without real code and a better description of what "doesn't work", it is hard to give you a better answer.