[RFC] changing variable naming rules

Hi all,

To get wider visibility, build a broader consensus and address concerns on this topic, I’m again raising this as an RFC. This is a proposal to change the rule for variable names from CamelCase to camelBack really this time.

Background:

This has been proposed several times on this mailing list in the past. Most recent one was by Michael Platings in February this year [1], and there seems to be a general consensus that the status quo is not ideal.

In the previous RFC thread, I nominated lld [2] as a starter project for renaming and made a sweeping change to rename variables in a few commits. This renaming went well – even though it broke buildbots, I managed to unbreak them in a timely manner, and more importantly, it has been reported that several downstream repos have successfully migrated to the new naming scheme using a tool that I wrote to create sweeping changes. That being said, some claimed that the renaming attempt didn’t get enough attention, despite being discussed in a thread that has 100+ emails. So I’m raising this topic as a new thread.

I propose we do the same thing to another relatively small subproject, clang-tools-extras, to gain more experience, and then migrate the entire LLVM codebase to the new style. It seems technically doable, and even though it would cause a short-term pain, people seem to feel more comfortable with the new naming scheme than the current one. I also believe that the migration won’t be that painful.

Objectives:

  • Migrating the entire LLVM repo including subprojects to the new naming scheme without breaking them.
  • Many projects, especially LLVM and Clang have lots of out-of-tree downstream repos. We need to provide a tool to rebase such repos to a commit after a renaming.
  • The sweeping change shouldn’t break git blame.

What I learned from the lld’s naming scheme change:

  • There are many member variables in the LLVM/lld codebase that have the same name as accessors ignoring case (i.e. many classes define foo() as an accessor to a member variable Foo). Such variables would conflict with functions after renaming, so we had to rename accessors by prepending get.

  • A single large sweeping change seemed to work better than small incremental changes for downstream repos. Downstream repo maintainers rebased their trees to a commit just prior to the sweeping change, apply my tool to rename all variables in their trees, and then rebase the trees onto the sweeping change. Because the tool creates the same diffs for existing code, downstream maintainers basically only had to merge their diffs at the last step.

  • Even though my tool worked satisfactory, it couldn’t rewrite code that are excluded by #if, #ifdef and the like, because the clang-based tool doesn’t really see the code excluded by the preprocessor. That caused several buildbot breakages.

  • git 2.23 (released in August) added a new option --ignore-revs to git blame so that the command can take a list of commits that need to be ignored by blame. Developers can set a default ignore file (typically named .git-blame-ignore-revs) using git config so that blame automatically ignores commits listed in the file. As far as I tried, that command worked pretty well to ignore the sweeping change I made to lld, so the git blame issue seems a solved problem now.

Migration plan:

Given the above findings, I propose we migrate to the new coding style in the following steps.

  1. Change the codebase to eliminate name duplication between accessors and members. This can be done incrementally with as many commits as we want.
  2. Complete the tool and apply it to the entire LLVM tree. I’ll publish it at GitHub so that people can take a look and try it out.
  3. Setup buildbots so that they checkout my GitHub tree, build it and run its tests, to make sure that a sweeping change won’t break them. (I don’t know how to configure buildbots, but I presume this step is doable.)
  4. Give a heads-up and submit a sweeping change to clang-tools-extras, and make sure that that won’t break anything.
  5. Give a heads-up and submit a sweeping change to the entire LLVM.

I’d like to submit a sweeping change after LLVM migrates to GitHub to minimize confusion.

[1] http://lists.llvm.org/pipermail/llvm-dev/2019-February/130083.html
[2] https://github.com/llvm/llvm-project/tree/master/lld
[3] https://github.com/llvm/llvm-project/commit/3837f4273fcc40cc519035479aefe78e5cbd3055

Developers can set a default ignore file (typically named .git-blame-ignore-revs) using git config so that blame automatically ignores commits listed in the file.

To make this as painless as possible for all future generations of contributors, we should provide an in-tree script that will DTRT to set this up, and document it in the Getting Started page. My understanding is that cloning can’t set any config options like this automatically, so an easy one-time script is the next best thing.

–paulr

Good point. As far as I understand, after git migration we will make merge commits illegal, and we’ll have to have each developer run a few git config commands after checking out the LLVM repository. The git config for blame should be described that instruction. (Or it can be a script containing just a few lines of git config, I don’t have a strong preference.)

Hi Rui,

FWIW, I’m very +1 on this. I think you did a great job with the LLD migration, and I’m really happy you used that as a relatively small scale proof point to develop the methodology and see what issues shake out.

Thank you for driving this!

-Chris

Personally I think the benefit-cost ratio is quite low here, and changing the rules would bring little value to me.
Maybe I just don’t see that much benefits of changing the naming rules, having got used to the current naming scheme after a few years with LLVM.

I'm a little bit curious to hear more about the experiences from the changes in LLD.
I’m thinking about things like:

- ongoing reviews in Phabricator that suddenly needs major rebase

- merging of bugfixes to older release branches (I doubt that the script supports doing the reverse rewrite as well)

When it comes to git blame and the ignore sweeping change file:

- Will that work when I do blame inside github (I assume it would need to be configured to find the ignore file in some way)?
  Anyway, ignoring sweeping commits should probably not be the default in github, since sometimes you want to find the sweeping commit.

- There are several more tools that also has blame functionality (e.g. Gerrit Code Review that could be used by downstream forks).
  Just because there is a --ignore-revs option in newer git versions, it wouldn't be used by all my tools. So I'm not sure it can be considered as a solution.

Blame is also just one potential problem. I'd still end up with lots of (IMHO) "irrelevant" changes when doing "git diff". For example when trying to figure out what happened between LLVM 8.0.0 and LLVM 9.0.0 in some part of the code, by actually looking at code diffs and not just the commit log.

Kind Regards,
Björn

I do not support this. I feel the benefit is low, and the churn cost is high.

I’m not strongly opposed or anything, I just don’t believe this is worthwhile.

Philip

[just so i don't end up with "why haven't you spoken up"]

I do not support this. I feel the benefit is low, and the churn cost is high.

I'm not strongly opposed or anything, I just don't believe this is worthwhile.

Same thoughts.

Philip

Roman

[just so i don’t end up with “why haven’t you spoken up”]

Same.

I do not support this. I feel the benefit is low, and the churn cost is high.

I’m not strongly opposed or anything, I just don’t believe this is worthwhile.
Same thoughts.

Same.

[just so i don’t end up with “why haven’t you spoken up”]

Same.

I do not support this. I feel the benefit is low, and the churn cost is high.

I’m not strongly opposed or anything, I just don’t believe this is worthwhile.
Same thoughts.

Same.

Same.

What cost do you see here? Rui has done a significant amount of work to make this effectively zero cost.

The improvements are meaningful, and (as was discussed on the other threads) pretty much every large scale change in the LLVM world has been shot down with objections like “it is too much churn”.

This is a huge problem, because it leads to stagnation in the codebase and does not allow modernization. LLVM has always had the development philosophy of "trying to be the best”, even if it comes at a cost. The unwillingness to maintain a stable C++ API is one very significant aspect of this.

I don’t see how this case is any different.

-Chris

I'm a little bit curious to hear more about the experiences from the changes in LLD.
I’m thinking about things like:

[..]

- merging of bugfixes to older release branches (I doubt that the script supports doing the reverse rewrite as well)

As someone who does a lot of these merges, this is the aspect that
worries me the most.

How large is a typical patch to merge that is backported from the next major release to a previous dot release?

The diffs are usually fairly small, but there can be many. For 8.0.1
there were about 60 patches merged (git log
llvmorg-8.0.0..llvmorg-8.0.1 --oneline | wc -l). (For 9.0.0 I think
we're approaching 200 merges.)

Depending on when the rename happens, I guess it could affect merges
either for a major release or for a dot release, and it would mean a
lot of extra work unless there was some tool available to help. Maybe
if the rename happens after a dot release ships and before the next
major release process begins, it would not affect any merges at all,
but that's a pretty slim window, and downstream users might have other
release processes.

Hypothetically assume that we have an LLVM release X and want to rename variables in the next major release Y. In order to backport patches from Y to X.1 smoothly, I think we could first apply the automated renaming tool to X and then cherry-pick patches from Y. The obvious issue of doing this is, even though X.1 is a dot release, a very large number of lines would be changed from X, though.

The even bigger issue is that X.1 is supposed to be API and ABI
compatible with X.0.

LLVM has hundreds of downstream repos, including cases where LLVM is a part of some other project. There is no zero cost here.

Whether the changes result in meaningful improvements is debatable. They were motivated by arguments about code readability for people new to the project. While that’s important, getting acquainted with every new code base is a bit of a challenge. LLVM’s naming style is consistent and is easy to get used to (unless someone’s harboring a resentment for it).

“Too much churn” can be used as an excuse, but that doesn’t invalidate it as an argument. I think that in this case it is justified. Renaming variables is nothing like upgrading the code to use a C++14 feature, for example.

If there is a last-ditch effort to stop this, I’m joining it.

Ah, that is true. Thank you for pointing that out. I’m not pushing this idea too much, but in theory, if there’s a tool that does the reverse conversion, we might be able to backport patches from a newer tree to an old tree without breaking the API and ABI, but I can see that that can be tricky.

Krzysztof Parzyszek wrote:

LLVM’s naming style is consistent

Sorry, but none of the LLVM naming conventions are consistently used project-wide. There are lots of “historical” exceptions, and that is mainly because people got excited about “too much churn” which enshrines inconsistency.

Applying a tool project-wide will be a step forward in consistency, which I personally think is a good thing. I had the sad experience, early in my career, of maintaining a code base where you might see 3 different conventions used in the space of a dozen lines of code. LLVM isn’t that bad, but moving from one library to another can make it easy to forget that there is, if only in principle, a naming convention.

As a downstream maintainer, I’m well aware any change can cause trouble. People who do big-bang merges once a release (which we used to do) already have a lot to contend with; IMO the incremental pain is small in that case. Rui’s procedure for fixing up identifiers in a downstream repo did work for us, so I am comfortable saying the cost is annoying but not intolerable.

Re. the upstream release branches, certainly waiting until after 9.0 is final would be best.

–paulr

It’s hard to expect 100% consistency in a project with this many contributors, especially when they are not a part of the same organization. What matters to me is whether I have to apply a different convention to something I’ve seen/written somewhere else. This mostly applies to global objects, APIs. Whether a loop in some function uses capital or lowercase ‘i’ is not quite as relevant. The only persistent inconsistency that I keep seeing is the naming of functions: some historical APIs use UpperCamel, while the rest uses lowerCamel.

I think that a better way forward is to keep a closer eye on consistency in new code. The older code tends to be rewritten every now and then, and the inconsistencies could be addressed at that time. The global APIs could be renamed in one shot, but that would be a change that is nowhere near as invasive.

Yes, I suspect you’re personally the one who is hit the worst by this. I’m super sensitive to that, because I really really really want to make sure that releases are smooth and successful.

That said, are you aware of Rui’s work on lld? He developed a process in which people with significant out of tree patches were able to catch up, merge the mega patch and move forward without significant churn. I agree it is non-zero, but it is very low, and the cost seems worth the price of forward progress and avoiding perpetual stagnation.

-Chris