libclang: Why do I have to dispose CXTUs in LIFO order for memory to be freed?

I have been trying to find out why the clang completer plugin in vim's
YouCompleteMe plugin makes vim take up more and more memory as I open
new buffers. On the github issue we eventually narrowed it down to a
simple program parsing 10 cpp files, then disposing their TUs.

When disposing them in the order that they were created, linux would
reuse the memory if the same files were parsed, but the RSS in ps
would never decrease, only increase when new files were parsed. I
eventually discovered that by disposing them in the opposite order, ie
LIFO order, the RSS stat would go up and down as expected.

For a completer plugin that needs to throw out the oldest TUs and
definitely not the newest ones, this is bad news. I would like to hear
your input on what it is in libclang that is causing this
fragmentation, and your advice for making it work.

The original discussion on github is here:

https://github.com/Valloric/YouCompleteMe/issues/184

And below is the (slightly modified) example program that demonstrates
the issue.

Harald Hvaal
harald.hvaal@gmail.com