Hi cfe-dev,
My name is Louis Brandy and I work at Facebook. I’ve begun working on
getting clang modules setup in our C++ codebase. Mostly the purpose of
this email is just to introduce myself and let people know what we’re
doing and our motivation, but I’ve also brought a handful of newbie
questions. I’ve gotten the basic integrations into the build system and
have some core projects building modularly. To get this far, I hacked
together a highly unprincipled set of module maps for glibc, libstdc++.
I’m at the point, now, where I need “real” module maps for our std/system
headers.
First, is there any prior art re: glibc and libstdc++ module maps? I don’t
want to repeat any work that’s already been done, and my google-fu failed.
Richard, I remember in the past we talked and you sent me your glibc module
map and small patches. Any chance you could attach your latest ones here?
Looking back in my email, you said that you didn't have a libstdc++ module
map. Is that still the case?
Louis, I'm going to be setting up a LLVM/Clang buildbot (running linux)
that uses modules for building LLVM itself next week or so, so I'll
definitely keep you up to date.
Second, I’m interested in the workflow of actually incrementally adding
module maps to a large codebase. I do understand the need to start at the
bottom but I’m worried about proper coverage, and then prioritizing what
to do next. In particular, I find myself really wanting a “summary” of
what #includes did and did not magically become imports so I can use that
to make sure 1) I’ve not missed anything “below” and 2) to prioritize what
to do next (by e.g. aggregating over a build the most textually included
headers).
When I first went to investigate how much time is spent in which header, I
placed some DTrace probes inside of clang and aggregated the time spent
textually within a file across compiler invocations. See the thread "Some
DTrace probes for measuring per-file time." (
[cfe-dev] Some DTrace probes for measuring per-file time.)
The raw data that comes out of that DTrace script is a list of pairs
{"/path/to/file", total time spent in this file across all compiler
invocations}.
I then looked at the data in this Mathematica notebook:
https://drive.google.com/file/d/0B8v10qJ6EXRxTWpMTTBnaERQaVU/view?usp=sharing
Note that in that notebook (one of many) I removed the time spent after
parsing (basically, codegen time), so the pie chart at the end is a bit
deceptive.
I've attached two pie charts that include the time spent after parsing.
The first is a debug build (low optimization). The latter is a a release
build (for a release build, a much larger fraction of time is spent in
codegen).
If you don't have DTrace available so that you can directly measure the
time, you can probably get a decent idea based on the inclusion counts. One
easy way to do this is to tally up files mentioned by the -H option whose
output you can massage. There is also '.d' files, but I forget exactly what
we emit into them (we may emit header file names even if we didn't
textually touch the header, but only loaded its module).
Note that for measuring the time, you need to use a timestamp that is
virtualized CPU time. If you use real time then you will spuriously count
IO latency and other stuff, which will give wrong results (e.g. the total
sum of time will appear much larger than is possible).
I don’t think such a diagnostic/remark exists? I’ve not looked
too deeply, yet, at clang-modularize, so perhaps my answers lie over
there?
We have -Wauto-import which is sort of the opposite of this. Adding the
reverse warning "warn me when you included a header but didn't know about
it from a module map" could probably be done.
-- Sean Silva
