Adding comment callback for documentation tools

Hi,

Adding information in comment is useful for documentation purpose or for
adding information/metadata for others tools like formal proof or DSL
code generator...

Actually, there isn't (AFAIK) a facility to extract the comment and the
AST at the same time.

Since managing comment token in the parser is to complex (they can
appear everywhere) I propose that the lexer or the preprocessor use
callback to inform the client that he encoutered a comment.

For example, for doxygen like tools, the callback would check if the
comment is a documentation one, and if so, store it for later use. Once
the AST complete, it would walk the def and associate the corresponding
comment (or the other way around).

Does this seems good, any other idea? alternatively, the lexer/PP could
create a list of comment token.

The callback could be added to clang::PPCallbacks. It is the best place
, isn't it?

This modification would have zero cost for classical use since lexer
already has a condition keep comment. It would only had a conditionnal
in the PP when a comment token is consumed.

If this is ok, I will prepare a patch.

regards,

Cédric

Hi Cédric,

High bit: yes, I think this is a good idea.

While I’m not exactly sure the form the callback should take, this should not conflict with anything else in the system and if you want to experiment with using it to build new tools that would be very cool.

I agree that some sort of callback mechanism is the right approach, this is in line with the current clang design. It probably makes sense for this to be both a preprocessor callback and a parser (Action) callback. This would allow the parser to potentially add additional information about the current parsing state to the callback, or defer the comment Action callback in some cases (for example, until the end of statement).

Are you planning on writing this concurrently with a consumer of the callback?

Also, I think it might be reasonable to use this to replace the current -verify mechanism. I believe this would make it more robust and also somewhat easier to use. Thoughts?

  • Daniel

Hi

While I'm not exactly sure the form the callback should take, this should not conflict with anything else in the system and if you want to experiment with using it to build new tools that would be very cool.

I was thinking to just pass the token as argument, for my use case, it should contain all the needed information.

I agree that some sort of callback mechanism is the right approach, this is in line with the current clang design. It probably makes sense for this to be both a preprocessor callback and a parser (Action) callback. This would allow the parser to potentially add additional information about the current parsing state to the callback, or defer the comment Action callback in some cases (for example, until the end of statement).

I don't really see how to make it a parser action, since it can appear (nearly) anywhere. The only possibility I can see is to put a filter in getNextToken (I don't have the source here, but there must be a function like this) which call the action on a comment token and drop it. Then, I don't see any advantage over the PP callback. But perhaps I don't understand what you want to do?

Are you planning on writing this concurrently with a consumer of the callback?

Yes, I would like to write a small documentation tools, but I have no timeframe (lots of projects and so little times :)).

regards,

Cédric