Where should tooling logic live? I have no idea...

I have a specific question about one of your points that is much more meta
than the original thread:

       - It should live in a separate repository of Clang tools alongside
(but not a part of) the main Clang repository.

I think it's clear we will need some side repository that is home of
contributed tools that are maintained and released along with Clang, but
not necessarily of interest to everyone.

What isn't clear to me is how we should decide when a tool belongs there or
should actually come along with the core Clang checkout. My initial guess
at where this tool should live was actually in the core Clang repository,
so I suspect you have a different set of heuristics you're using. Could you
elaborate on them?

Some observations that have motivated my initial guesses:

1) It seems like 'clang-check' and maybe 'clang-fixit' (which should be the
Tooling analog to clang -fixit-always) are really good tools to put
directly in the Clang tree
2) It seems like ad-hoc tools like the one which removes redundant calls to
std::string::c_str() should not be in the main Clang tree.
3) I suspect that truly generic refactoring tools (let's say for example, a
'rename' tool) could very reasonably go either direction.
3.1) I suspect that this will be motivated by the fact that the generic
refactoring logic will be structured as a library, shared by lots of tools
4) The idea of a Clang service layer complicates this -- in that model
*all* of the tools should be libraries bundled into the server.

Combining those observations leaves me with no clear feeling for where
things should live.

I would personally like the tools to be plugins for the server. Even if they end up being statically linked, it’d still be nice to be able to build with only some enabled, say. And once you have that it’s more acceptable (in my mind anyway) to have some of them live out of tree.

Jordan

I completely agree, and that was the design I planned to follow.
(Specifically, both static linking and "pluggable" by checking out more
code into the same tree before building the server. I *do not* want dynamic
loading in a multithreaded system.)

But it does complicate things because it means:
- The architecture of the separate repo is a bit tricky and has to work to
support code sharing at the library level. This is good, but an added
complexity.
- We'll want to have some core functionality in the server that Always
Works. But what is that functionality? It's hard for me to (currently) draw
a clear line.

Regardless of where the line goes, even if for some reason we decide that
*all* of the tooling logic to ever get contributed to this open source
project, we still have to have the statically pluggable layout IMO, because
lots of users will want to drop their custom and special tooling logic in
that is highly specialized, inappropriate (or impossible) to open source,
etc.

I see the main Clang repository as being responsible for:

  1. The compiler proper
  2. IDE integration
  3. Clang service layer (when it happens)
  4. Infrastructure to build Clang-based tools and plugins
  5. Testing infrastructure for all of the above

To me, that’s the core of what Clang is. Things like the static analyzer, ObjC rewriter, ARC migrator, c_str() pruner, C++11 migrator, etc. are outside of that core and should eventually live in the side repository.

I worry a little bit about the side repository becoming broken. Obviously, there’s an additional testing burden to consider here, and I do worry (in the abstract) that we’ll be making trouble for ourselves by having to maintain a pile of standalone tools outside of the main repository. But, I think we can handle this.

The Clang service layer does complicate this. Allowing the server to load plugins would solve the issue for plugins that ride on the normal compile jobs. Perhaps we can make it not completely painful to turn a standalone tool into a plugin that can be loaded by the server.

  • Doug