Python bindings overhaul

I thought that the Python binding to libclang could use some love, so
I've been working on a number of enhancements to it.

You can find my work at [1]. Changes can be seen at [2]. Things are
still a work in progress and there are a number of outstanding issues
I'd like to address before I formally submit a patch. But, I think
things are ready for critique.

Major changes:

  * Enumerations split out into clang.cindex.enumerations module
  * Objects hold references to parent (i.e. worry-free memory management)
  * Support for Token API
  * More APIs supported around the board. Pretty much everything from
    libclang is supported except the high-level Indexing component.

Other improvements:

  * Python style improvements. pylint and pyflakes output is now
    tolerable and the code mostly follows Python best practices.
  * Improved documentation
  * Increased test coverage
  * Added test to ensure enumerations are up-to-date with Index.h (uses
Tokens API)
  * Introduced decorator to cache/lazy load properties. This should cut
down on C function calls.

Despite the major changes, API compatibility is mostly preserved. I did
break APIs. However, I actively tried to only break things that I
thought were internal to the module. If external consumers were using
the things that changed, IMO they deserved to break because they weren't
using things in an intended manner. This is my intent anyway. I haven't
thoroughly audited to ensure I stuck to this principle. If the lack of
changes to the existing tests is any indication, I think I did a good
job though.

That said, there are numerous API breaking changes I would like to make.
Here are some examples:

* Various APIs are using camel case instead of underscores for method
names, breaking the Python convention.
* Diagnostic exposes iterable structures as properties. IMO properties
  should return the same object every call. Currently, they return
  different instances. This is confusing, IMO.
* Code completion has a different style from the rest of the module.

If someone gives me a green light, I'd love to have a go at it and
produce a consistent API.

Gregory Szorc
gregory.szorc@gmail.com

[1] https://github.com/indygreg/clang/tree/python_features/bindings/python
[2] https://github.com/indygreg/clang/compare/master...python_features

The remaining content in this message is essentially my developer notes
and explains why I did what I did. If you will be commenting, please
read it first.

Hi Gregory!

Major changes:

* Enumerations split out into clang.cindex.enumerations module
* Objects hold references to parent (i.e. worry-free memory management)
* Support for Token API
* More APIs supported around the board. Pretty much everything from
   libclang is supported except the high-level Indexing component.

Other improvements:

* Python style improvements. pylint and pyflakes output is now
   tolerable and the code mostly follows Python best practices.
* Improved documentation
* Increased test coverage
* Added test to ensure enumerations are up-to-date with Index.h (uses
Tokens API)
* Introduced decorator to cache/lazy load properties. This should cut
down on C function calls.

This all sounds awesome. How about we start small here though? You've
mentioned that you have a bunch of cleanups for eye sores, doxygen
comments, etc that we could apply on our way to having the python
bindings get feature parity with libclang. Let's get some of those reviewed
separately and then tackle the larger tasks as we go?

Thanks again for all of this work, I haven't had time to look at your
github repository but hope to soon!

-eric

Hi Gregory,

just to give you some first feedback right now. I skimmed trough the git repository and I like the overall direction of the patches. They seem to improve the python bindings a lot. Unfortunately I did not have time to look into the exact details.

One comment:

I think the patch is very large. This is good to show the overall direction. However, to commit and review these changes I would very much
appreciate smaller patches. As I have seen in your git repository, you have all this already divided in small patches. Most of them seem to be easy to review.

I will give it another look (either today or tomorrow). I will also try your changes with clang_complete. If there are no big blockers (wait also for Erics review), I would propose to slowly commit the changes to svn. You could commit trivial fixes without pre-commit review and other changes could be posted in batches of five to get them reviewed. As they are split up in small portions, reviewing them should be fast.

Thanks again for your great work!

Cheers
Tobi

I just did this. For me they were a drop in replacement. Everything continues to work as expected. The completion time did not change.
clang_complete does not seem to be affected by these changes, but may obviously later take advantage of the new features.

Cheers
Tobi

Glad to hear it! Code complete doesn't have good test coverage, so I was a bit worried about that one.

I'll start sending some smaller patches that incrementally add libclang features. I think we can get a long way through the commits in my branch before we start hitting some larger, more controversial changes.

Gregory