[RFC] Add a driver option to specify C library header directories

Would a C counterpart to -stdlib++-isystem be an acceptable addition to Clang? It would let us supply explicit libc header roots while the driver preserves its target-specific ordering relative to C++ and builtin headers.

Motivation

I maintain hermetic-llvm, a hermetic LLVM-based Bazel toolchain for cross-compilation. Its compiler, libc, Linux UAPI, and C++ library headers are separate build inputs, rather than a preassembled target sysroot. It is also the @llvm toolchain selected by LLVM’s own Bazel overlay’s hermetic-toolchain configuration.

We prefer supplying those headers explicitely to clang over a sysroot construction because creating a sysroot requires duplicating and aligning with driver expectations per target, and requires creating directory artifacts which are opaque to bazel and against bazel total expliciteness goal.

After long investigation, my understanding is that there doesn’t seem to be a way of achieving this today with full explicit driver include flags in a non tedious and brittle way. Like without using internal -Xclang flags and without duplicating ordering decision that belong in the driver.

This especially true given that the right include search diverge per target and environment, like for musl vs gnu for instance:

  • x86_64-linux-gnu = C++ library → Clang builtins → C library
  • x86_64-linux-musl = C++ library → C library → Clang builtins

This discussion is about providing C library headers only. Examples below assume that Clang builtin headers are provided by -resource-dir.

-stdlib++-isystem already expresses the C++ part of this requirement, and -resource-dir expresses the builtin-header part

We lack an equivalent for C library headers.

The current workaround

For GNU/Linux, we disable default standard-library header discovery and supply the libc and Linux UAPI header directories using private frontend options:

-nostdlibinc \
-stdlib++-isystem <some_path>/libcxx/include/c++/v1 \
-Xclang -internal-externc-isystem -Xclang <some_path>/libc/include \
-Xclang -internal-externc-isystem -Xclang <some_path>/linux-uapi/include

This places the C headers after Clang’s builtin headers, as the GNU/Linux driver normally does.

For musl, the Linux driver currently places C headers before builtin headers. Forwarding only the C paths through -Xpreprocessor also puts them before the C++ paths supplied by -stdlib++-isystem, which is incorrect.

To preserve the driver’s builtin-header handling, our current workaround therefore forwards both C++ and C header paths through private -Xpreprocessor arguments, with C++ first. This also means those C++ paths bypass -nostdinc++.

Another workaround is to disable builtin-header discovery and insert those headers ourselves. Either approach requires us to reproduce part of Clang’s include policy. I would much rather tell Clang where each set of headers lives and let the driver handle their ordering.

Proposed direction

A repeatable driver option, provisionally -cstdlib-isystem, would replace implicit C library header discovery with explicit roots, preserving their supplied order and the target’s existing builtin ordering and C-header classification. For example:

clang++ --target=x86_64-linux-musl \
  -stdlib++-isystem <some_path>/libcxx/include/c++/v1 \
  -cstdlib-isystem <some_path>/libc/include \
  -cstdlib-isystem /linux-uapi/include \
  -c example.cc

The same header arguments would work for GNU/Linux: the driver would choose the different builtin position. This would allow hermetic C header inputs without encoding that distinction in the build system. Library linking remains a separate concern.

This proposal would follow the driver’s policy, allowing that policy to evolve without corresponding changes in the build system.

Prototype checks

I have a prototype for GNU/Linux, musl Linux, and Darwin. Tests confirm that Clang uses the supplied C header directories instead of its defaults, while preserving their order relative to C++ and builtin headers. They also check #include_next and the flags that disable header searches.

That prototype follows -stdlib++-isystem’s explicit-root behavior by preserving supplied C roots under -nostdinc. It proposes -nostdlibinc as the cancellation switch; -nostdinc++ leaves C roots active. This is an intentional C adaptation: the C++ override survives -nostdlibinc, and that switch also suppresses default C++ discovery.

Feedback requested

  1. Would a flag for replacing Clang’s default C library header directories be useful and acceptable? Is there an existing direction I missed?
  2. Are the proposed ordering and interactions with -nostdinc, -nostdlibinc, and -nostdinc++ appropriate?
  3. This likely has implication to various Toolchains (as per driver language) and targets. Should this apply to all targets from the get go ?
  4. What would be the acceptable next steps ? I was thinking of doing a complete due dilligence of all per Toolchains change in driver behavior to account for all candidates for which to test non regression.

@MaskRay I’m cc’ing you given your recent involvment in the the CC toolchains ruleset of bazel.

A C counterpart to -stdlib++-isystem, driver-only (no cc1) option, sounds reasonable to me.

The name -cstdlib-isystem fits: -stdlib= selects the C++ library and --cstdlib= selects the C library (newlib/picolibc/llvm-libc). A plain -stdlib-isystem would read as the C++ one.

What it replaces. Please specify the exact set. After the resource directory, Linux::AddClangSystemIncludeArgs adds multilib variant directories, getStdlibIncludePath(), /usr/local/include, the GCC installation’s include directories, then either C_INCLUDE_DIRS or multiarch /usr/include/<triple>, /include, and /usr/include. I’d expect -cstdlib-isystem to replace all of them. It is essentially a command-line C_INCLUDE_DIRS, so the implementation can probably share that branch. One difference: C_INCLUDE_DIRS prefixes relative entries with the sysroot, but -cstdlib-isystem should take paths literally, like -isystem and -stdlib++-isystem, so that Bazel’s exec-root-relative paths work. Note that the branch returns early and skips musl’s trailing resource directory, so musl with C_INCLUDE_DIRS currently loses the builtin headers. Don’t copy that. On Darwin, -nostdlibinc also drops <sysroot>/System/Library/Frameworks and friends. I’d keep the framework paths, since they aren’t the C library, but please say so explicitly.

-nostd* interactions. -stdlib++-isystem ignores -nostdinc and -nostdlibinc. I’d suggest that -nostdinc and -nostdlibinc suppress -cstdlib-isystem, while -nostdinc++ and -nobuiltininc don’t.

Scope. About 37 toolchain files implement AddClangSystemIncludeArgs. I wouldn’t audit all of them up front. Implement Linux (glibc and musl) and Darwin with tests, and make other targets report err_drv_unsupported_opt_for_target, so other toolchains can opt in later. An error is better than silently ignoring the flag: a hermetic build would otherwise pick up host headers without noticing.

It would be great to have more documentation about all the various options mentioned here and in MaskRay’s reply. Some have precious little docs upstream, and it’s not always clear which are handled by which toolchain implementations (within clang).

1 Like

I’m generally in support of this RFC. This helps make it easier to specify a different C standard library which is useful for users, but also helpful for ourselves when testing behaviors between various C standard library implementations (which I don’t think we do a particularly stellar job of doing for Clang). Also, it’s always bothered me that the environment triple is what selects the C standard library, and this gets us a step closer to supporting -cstdlib=, so I like that.

I’d go one step further and ask that this be a hard requirement for all new user-facing driver options. The help text in the driver options file is necessarily encouraged to be terse. We should be documenting things more extensively in the user’s manual, particularly when the options have complex interactions between then like header search paths.