libclang based doxygen-like tool

Hi:

I've written a small prototype of a doxygen-like tool based on libclang with the hope of eventually contributing it.

I've only just started using clang, so I was hoping I could share what I've got so far and get feedback from more experienced clang developers before moving to the next phase.

Currently, it generates an html page for each source file, a la doxygen, with intra- and inter-file links for variables, functions, and classes/structs. There are still some rough edges, but I wanted to make sure I was using the libclang API correctly before moving on to the documentation pages.

I'd be happy to post a tar here (I run FreeBSD), or send it directly to anyone who might be interested. Please let me know what's best.

thanks...
don

Posting tarballs to the mailing list is generally frowned upon, but I'd definitely like to see the code. Can you put it online somewhere? If not, add me to the list of people who would like a copy of the tarball...

David

I would very much like to see this as well. Thanks.

I would very much like to see this as well. Thanks.

I’ve sent copies you and the others who expressed an interest, and will look into hosting it somewhere.

thanks…
don

Isn't this exactly the sort of thing that should end up in the
clang-tools-extras repository?

What do others think? Chandler, this is your thing isn't it?

Cheers
Andy

P.S. Don, I'd also be interested in a copy too :o)

I've sent copies you and the others who expressed an interest,
and will look into hosting it somewhere.

Isn't this exactly the sort of thing that should end up in the
clang-tools-extras repository?

What do others think? Chandler, this is your thing isn't it?

Definitely, once the design has been discussed; my main concern would
be that the tool nicely fits the latest extensions to clang's comment
parsing, and has a clear strategy and maintenance story going forward.

Cheers,
/Manuel

Hi,
                                                                                 
libclang is really a great tool for writing your own documentation tool. I
never liked the concept of javadoc/doxygen of putting all the documentation
into the header/source files. In my opinion it bloats the headers and makes
them unclear. What I always wanted is a non-invasive documentation tool that
leaves my code clean. Fortunately libclang gives me the choice to do it my
way:

My libclang based tool extracts all the declarations I am interested to comment
and creates a doc-file with stubs:
  (1a) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/matrixtypes/symmetric/impl/symatrix.doc.txt
This can be converted to HTML:
  (1b) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/matrixtypes/symmetric/impl/symatrix.html
The tool also keeps these code snippets in sync with the header/source files.
E.g. it adds stubs new methods, modifies stubs if the header/source changes,...

For sure, you want to add some useful comments to the stubs, e.g.
  (2a) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/matrixtypes/general/impl/gematrix.doc.txt
and you get some nicer output:
  (2b) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/matrixtypes/general/impl/gematrix.html
Another example
  (2c) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/lapack/impl/sv.doc.txt
  (2d) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/lapack/impl/sv.html

Well, you still have to write the documenting text yourself (and I did not write
much so far) but you see that some reStructuredText can be used. Here a page with
tables, list, footnotes, links and tables with list:
  (3a) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/lapack/lapack.doc.txt
  (3b) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/lapack/lapack.html

Another nice feature is about creating tutorials. Assume you have written a
small example for your project. Then you want to write some general notes, show
the source code, comment the source code, tell people how to compile it and show
its output:
  (4a) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/examples/lapack-gelsy.doc.txt
When this doc-file gets converted to HTML it will include the example source file,
compile it and execute it:
  (4b) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/examples/lapack-gelsy.html

The certainly coolest feature that libclang gives to all of us is cross referencing
the source code, e.g.
  (5) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/examples/lapack-gelsy.cc.html
Note the words in the source code that are (dotted) underlined. For overloaded functions
you get red underlined word, e.g. 'cxxbla::ger':
  (6) http://www.mathematik.uni-ulm.de/~lehn/FLENS/flens/blas/level2/r.tcc.html#60

I was really surprised how easy libclang makes it to implement such tools!

Michael

However, it's easier to keep in-source documentation up to date.

Hi,

libclang is really a great tool for writing your own documentation tool. I
never liked the concept of javadoc/doxygen of putting all the documentation
into the header/source files. In my opinion it bloats the headers and makes
them unclear.

However, it's easier to keep in-source documentation up to date.

The first time I was using javadoc I had the same thought. I was thinking
that putting the documentation directly to the crime scene is the right thing
to do "... when I change the function I update the documentation next to it...".

But then I ended up with "... let's just fix the code quickly and update the
documentation later ...". And I ended up with documentation comments that were
not in sync with the code. So I learnt that no tool will do all the work for me.
But maybe I am the only one having this problem.

Anyway, from some point of view my documentation tool is doing the same thing as
javadoc/doxygen. Assume I have some file 'myfunc.h' with a function 'void dummy()'.
Then I my tool creates (or updates) some file 'myfunc.doc'. This file will contain
some documentation-stub, that looks like this:

*--[CODEREF]-----------*

17.08.2012, 21:18, “Michael Lehn” <michael.lehn@uni-ulm.de>:

Hi,

libclang is really a great tool for writing your own documentation tool. I
never liked the concept of javadoc/doxygen of putting all the documentation
into the header/source files. In my opinion it bloats the headers and makes
them unclear.

However, it’s easier to keep in-source documentation up to date.

The first time I was using javadoc I had the same thought. I was thinking
that putting the documentation directly to the crime scene is the right thing
to do “… when I change the function I update the documentation next to it…”.

But then I ended up with “… let’s just fix the code quickly and update the
documentation later …”. And I ended up with documentation comments that were
not in sync with the code. So I learnt that no tool will do all the work for me.
But maybe I am the only one having this problem.

Anyway, from some point of view my documentation tool is doing the same thing as
javadoc/doxygen. Assume I have some file ‘myfunc.h’ with a function ‘void dummy()’.
Then I my tool creates (or updates) some file ‘myfunc.doc’. This file will contain
some documentation-stub, that looks like this:

–[CODEREF]-----------

void dummy(); |

----------------------
[…USR]

The code snippet for ‘void dummy()’ is logically linked with the function’s
declaration through the USR generated by libclang. So whether I put my documentation
at the ‘myfunc.doc’ file beside its CODEREF location or put it at its original place
in ‘myfunc.h’ is logically equivalent. However, physically its separated clean and tidy.

At least for editors like VIM it is fairly easy to write a plugin such that you can jump
from the source-file to the doc-file forth and back. E.g. if your in ‘myfunc.h’ at the
‘dummy’ declaration you jump to the ‘myfunc.doc’ to corresponding CODEREF section. Or it
can create a new CODEREF snippet it doesn’t already exist.

If the signature of a function changes my tool detects that the doc-file contains
obsolete documentation. This is something javadoc/doxygen can not do.

Michael


cfe-dev mailing list
cfe-dev@cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

I must admit that the .doc file looks pretty weird (format-wise), especially the —[XXX]— parts, with | delimiters etc… but after looking at all the different things that can go within it, I suppose it makes (some) sense. I would still advise to make it more lightweight by removing the trailing | (and blanks), something as light as:

[CODE]
// some code example

Would be at least as simple to parse and be much easier to compose manually.

Otherwise, it does seem like a great tool. The debate for in-source vs out-of-source documentation has been raging for a while already and though I initially appreciated in-source documentation like you I have found that there are issues; my personal peeve is the amount of screen estate “lost”: when I am looking for a function, having only 3 signatures per screen because each is preceded by a large (mostly auto-generated and mind-numbing) doyxgen stub is extremely annoying.

Is this tool available somewhere ?

– Matthieu

Hope, I am mistaken, but Clang does not keep comments, isn’t it so? But it is impossible to imagine a documentation generation tool which skips comments…

2012/8/18 Slav <slavmfm@gmail.com>

As an alternative, more lightweight notation I already have

--- CODE -----------
// some code example

It does actually, and Dmitri Gribenko has been working on implementing a comment parser to recognize Doxygen-like comment (and hopefully other styles afterward), attach them to the entities they do comment and warn about improperly formatted doc comments.

– Matthieu

Do not worry, I do understand it is a work in progress.

I already have clang working correctly on two environments (Ubuntu server at home and Suse 11 at work) so I don’t think getting the tool to work on one or the other would be much of a problem. Not sure I’ll have much time before October to really check it out, but it does seem really handy.

Thanks for the links.

– Matthieu