[RFC] Reliable compiler specification setting (at least include/lib dirs) through the process environment

Hello GCC and Clang devs!

Unlike the traditional approach of installing system libraries into one
central location like /usr/{lib,include}, the nix package manager [1]
installs each package into it's own prefix
(e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
each package is built in its own environment determined from its
explicitly listed dependencies, regardless of what else is installed on
the system. Because not all package build scripts properly respect
CFLAGS etc., we currently wrap the compiler [2] to respect custom
environment variables like NIX_CFLAGS_COMPILE, so during the build of a
package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".

Unfortunately, as you can see if you click through the link or look
through the git history, the wrapper is quite complex (frankly, hacky)
and has evolved mostly through trial and error. Moreover, it's known to
break things like response files [3] and is generally speaking a source
of frustration. I believe the situation would be much improved if gcc
and clang supported some form of "environment-specific" configuration,
either through environment variables or, if absolutely necessary,
command line flags that can be passed unconditionally (note, for
example, that we currently parse the cc command line to see if we're
going to do any linking before deciding to pass in linking-specific
options) and clearly have the semantics we want. Ideally we would be
able to specify something on the level of abstraction of "this directory
should be treated like you would normally treat /usr" and get
e.g. /include, /lib, frameworks on OS X, etc. handled properly.

Would patches aimed at achieving this be considered for inclusion
upstream? My current thought for a first step would be an environment
variable specifying a file with command line flags that are ignored by
the compiler driver in contexts where they are inapplicable or
overridden by other command line flags, but I'm definitely open to
guidance on how this should best be achieved from your perspective. I'm
happy to do the work needed to get this in place if there is interest,
please let me know!

Thanks,
Shea Levy

[1]: NixOS - NixOS Linux
[2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh
[3]: https://github.com/NixOS/nixpkgs/commit/a421e7bd4a28c69bded8b17888325e31554f61a1

What that suggests to me is options for having multiple sysroots (which
are treated like / not like /usr, but that's the existing concept for a
directory containing both header and library subdirectories, and you could
combine this with a Hurd-style configuration of the expected sysroot
subdirectories, i.e. no /usr inside the sysroot). That would however be
rather complex; both GCC and ld presume there is a single global sysroot
(modulo SYSROOT_SUFFIX_SPEC / SYSROOT_HEADERS_SUFFIX_SPEC that append to
it), as do the interfaces for other specs that pass down sysroot
information to cc1 etc. - and ld then interprets absolute paths in linker
scripts such as libc.so in a sysroot relative to that sysroot (so would
need to track which sysroot a particular linker script was found in to
know how to interpret absolute paths in it), and options such as -I=dir
for a sysroot-relative include don't have a clear meaning with multiple
sysroots.

I'm wary of adding environment variables as they tend to make debugging
hard when the same compiler behaves differently for different people for
no obvious reason.

You should not need to exclude linker options (as opposed to linker input
files) from the command line when not linking, or compiler options when
linking.

Hi Shea,

Shea Levy skribis:

Unlike the traditional approach of installing system libraries into one
central location like /usr/{lib,include}, the nix package manager [1]
installs each package into it's own prefix
(e.g. /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev). Moreover,
each package is built in its own environment determined from its
explicitly listed dependencies, regardless of what else is installed on
the system. Because not all package build scripts properly respect
CFLAGS etc., we currently wrap the compiler [2] to respect custom
environment variables like NIX_CFLAGS_COMPILE, so during the build of a
package that depends on zlib and Xlib might have NIX_CFLAGS_COMPILE set
to "-isystem /nix/store/bl0rz2xinsm9yslghd7n5vaba86zxknh-libX11-1.6.3-dev/include -isystem /nix/store/mn9kqag3d24v6q41x747zd7n5qnalch7-zlib-1.2.8-dev/include".

Unfortunately, as you can see if you click through the link or look
through the git history, the wrapper is quite complex (frankly, hacky)

[2]: https://github.com/NixOS/nixpkgs/blob/8cbdd9d0c290e294a9d783c8868e738db05c9ce2/pkgs/build-support/cc-wrapper/cc-wrapper.sh

Guix avoids the compiler wrapper altogether like this:

  • We use C_INCLUDE_PATH, LIBRARY_PATH, and friends:
    <gcc.scm\packages\gnu - guix.git - GNU Guix and GNU Guix System.

  • We have a simple linker wrapper aimed at adding -Wl,-rpath flags:
    <ld-wrapper.in\packages\gnu - guix.git - GNU Guix and GNU Guix System.
    The comment in that file explains why the other options considered
    were unsuitable.

  • We modify the built-in “lib” spec of GCC to add the necessary -L and
    -rpath flags:
    <gcc.scm\packages\gnu - guix.git - GNU Guix and GNU Guix System.

  • Likewise, we tell Clang where to find libc and friends:
    <http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/clang-libc-search-path.patch&gt;
    <llvm.scm\packages\gnu - guix.git - GNU Guix and GNU Guix System.

This is not too intrusive and more robust than wrapping everything.

I suppose GCC and Clang could facilitate this by providing configure
options to augment the “lib” spec, specify the location of libc alone,
or something along these lines.

Thoughts?

Ludo’.

Hey Ludo’,

Amazing, more than a decade of close working with these tools and I
never knew about C_INCLUDE_PATH et al! It looks like those will solve a
huge portion of the problem.

Will look at your gcc and clang patches as well, thank you!

~Shea

Ludovic Courtès <ludo@gnu.org> writes:

Hi Ludo’,

Your patches look good! My biggest concern is how the ld wrapper behaves
in the presence of response files. Have you tested that?

Thanks,
Shea

Ludovic Courtès <ludo@gnu.org> writes:

Hi!

Shea Levy <shea@shealevy.com> skribis:

Your patches look good! My biggest concern is how the ld wrapper behaves
in the presence of response files. Have you tested that?

It surely doesn’t (yet?).

However, GCC does not pass “@file” arguments when it invokes ‘ld’, and
the bug report you mentioned¹ talks about GHC invoking ‘gcc’, not ‘ld’,
so I guess it’s fine to ignore response files in the ld wrapper.

Ludo’.

¹ https://github.com/NixOS/nixpkgs/commit/a421e7bd4a28c69bded8b17888325e31554f61a1

GCC will pass response files to ld when response files were used in
the invocation of GCC.

-Nathan

Yes! I’m very interested in this as well!

One piece of information to add: it might be worth investigating something
more general like the configuration files used by ELLCC in controlling their cross-compilers.

This was somewhat recently discussed here:

http://lists.llvm.org/pipermail/llvm-dev/2015-July/087907.html

And looks like the most recent description of it is here:

http://ellcc.org/blog/?p=13246

This would be rather useful in Nix for creating cross-compilers as well.

AFAIK this isn’t immediately suited to your needs but I think that it’s the right direction to go
and it’d be ideal to support both use-cases if we are talking about upstream support.

~Will