Configuration files

[Finkel_Hal_J]

From: "Matthias Braun via cfe-dev" <cfe-dev@lists.llvm.org>
To: "Serge Pavlov" <sepavloff@gmail.com>
Cc: "Richard Smith" <richard@metafoo.co.uk>, "Clang Dev" <cfe-dev@lists.llvm.org>
Sent: Tuesday, September 27, 2016 1:45:09 PM
Subject: Re: [cfe-dev] Configuration files

Sorry for being late to the discussion,

I think automatic user defaults are a bad idea: Right now when you
invoke clang without any additional options you know that you are in
a well known state. Build systems and projects rely on this having
unknown compiler options active because the user put them into his
configuration file is a recipe for disaster IMO! The example below
"-std=c++14 -fcxx-exception" already illustrates a situation in
which settings are used that can break the compilation of many
projects!

- I'm fine with a commandline option triggering reading a config file
(@file)
- Using a different executable name/symlinks to trigger loading of
config files may be fine as well (build systems should not pick up
myarch-clang by accident)

I agree with both points.

- Automatically picking up options from user or system configuration
files is too dangerous IMO!

There are definitely good user-experience reasons to want to do this - It might be nice to have some global configuration file that causes Clang to default to using -march=native or use some particular include path (setting CPATH and friends often works, but can also be flaky). The problem is indeed build system interaction, dealing with incompatible options, etc. For many things our last-option-wins scheme will do the right thing, but it is not clear, for example, that -nostdinc would have the expected effect if the config file adds include paths.

-Hal

[jyknight]

I really don’t see what the purpose of this feature is, if it’s only going to work effectively identically to an “@args-file” option.

If that’s all you want, it doesn’t even need to be in the clang driver, a 5 line shell script that looks at $0 to choose a config file to read can work just as well.

But I don’t think that’s really what people want – I believe that any workable config-file scheme must interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target.

[spavloff]

This feature addresses at least two different use cases:

1. Config file as a way to override compiler defaults.

Compiler already has some default settings and it is hardly possible to find the settings that suit everybody. For instance, the warning -Wundefined-var-template is helpful for some users but annoys others. With config file a user could put new defaults there and thus customize the compiler. No changes to build script is required. Note, this scenario requires just default config files.

Setting config file via environment variable is a variant of this scenario, it allows modification of compiler options without touching build scripts.

Intel compiler implements both default config and environment variable as a way to modify compiler option and practice proved their convenience.

2. Config file as an “option macro”.

Some users, especially those using cross-compilation, want to have possibility to define an “option” that when applied specifies a set of other options. An example: switching from one target processor to another which requires change of sysroot, libraries, triple etc. By organizing these options into config files, work with several build variants becomes more convenient.

This scenario requires to specify particular config file, either by option --config or by encoding it in compiler executable like clang-conf.

Clang based ELLCC Compiler uses these kind of config files and they are proved to be convenient (http://lists.llvm.org/pipermail/cfe-dev/2016-September/050783.html).

[Joerg_Sonnenberger]

This feature addresses at least two different use cases:

1. Config file as a way to override compiler defaults.

Compiler already has some default settings and it is hardly possible to
find the settings that suit everybody. For instance, the warning
-Wundefined-var-template
is helpful for some users but annoys others. With config file a user could
put new defaults there and thus customize the compiler. No changes to build
script is required. Note, this scenario requires just default config files.

This is *exactly* why I am *strongly* against this. Yes, users do that.
Afterwards, they start to fill random bug reports against projects that
break, because the invocation includes -Werror. This is even worse than
setting a global CFLAGS or CXXFLAGS variable since the expansion of
those are at least generally visible. Hidden user-modifable sources of
flags are a support nightware.

Setting config file via environment variable is a variant of this scenario,
it allows modification of compiler options without touching build scripts.

See above. Same deal.

Intel compiler implements both default config and environment variable as a
way to modify compiler option and practice proved their convenience.

I dare to say that ICC tends to be used in a much more limited
environment. If you want to allow magic environmental variables in your
build script that are invisible for the casual observer, it is not
difficult to add. Well written build systems already support that
anyway.

2. Config file as an "option macro".

Some users, especially those using cross-compilation, want to have
possibility to define an "option" that when applied specifies a set of
other options. An example: switching from one target processor to another
which requires change of sysroot, libraries, triple etc. By organizing
these options into config files, work with several build variants becomes
more convenient.

This scenario requires to specify particular config file, either by option
--config or by encoding it in compiler executable like clang-conf.

Clang based ELLCC Compiler uses these kind of config files and they are
proved to be convenient (
http://lists.llvm.org/pipermail/cfe-dev/2016-September/050783.html).

There was a reasonable proposal when this came up in the past. If you
have an invocation like ${target}-clang, look for a ${target}.cfg option
file and parse that. The lookup should be limited to something derived
from the static configuration and -B options.

Joerg

PS: It is a PITA to reasonable respond to mails that aggregate feedback
to a complete thread.

[Joerg_Sonnenberger]

No, nothing like spec files. As I said, you want to be able to specify
that the arm-linux target uses --sysroot=/somewhere/arm-linux and that
the arm-netbsd target uses --sysroot/somewhere/arm-netbsd. That and
maybe some -B option to help finding the linker likely already covers a
good 90% of what people need for setting up cross-compilation, if the
automatic triple-based search is not enough.

Joerg

[Joerg_Sonnenberger]

> Sorry for being late to the discussion,
>
> I think automatic user defaults are a bad idea: Right now when you invoke
> clang without any additional options you know that you are in a well known
> state. Build systems and projects rely on this having unknown compiler
> options active because the user put them into his configuration file is a
> recipe for disaster IMO! The example below "-std=c++14 -fcxx-exception"
> already illustrates a situation in which settings are used that can break
> the compilation of many projects!
>
>
In fact invocation of clang without additional argument does not provide a
well known state. Actual state is represented by options passed from clang
driver to clang compiler, but these options depend on used OS, CPU. In
theory a system upgrade could cause compilation change. Hardly bare driver
arguments can represent compilation options.

This is frankly FUD. While a compiler update can sometimes introduce new
default warnings (or add them to -Wall), those are governed by pretty
strict rules. The far majority of the options, especially the per-OS
per-CPU configuration changes very rarely and few of those changes are
user-visible in the sense that they break valid input.

Default config file is just a way to influence the magic of option
calculation made by driver. Config files can be used by clang based SDK
suppliers to customize clang without need to modify compiler sources. If
end user plays with default config files, he must know what he does.

But that's the point. Users often don't know or forget. There are long
rants to be found by Google of people forcing -OMG -ffast-math for
everything because they don't understand the implications. I strongly
hope that any vendor shipping a customized default config puts a strong
"Do not change this file without fully understanding what you are doing"
comment in it. The important part is that others are the ones that pay
the price.

Joerg

[spavloff]

> > Sorry for being late to the discussion,
> >
> > I think automatic user defaults are a bad idea: Right now when you
invoke
> > clang without any additional options you know that you are in a well
known
> > state. Build systems and projects rely on this having unknown compiler
> > options active because the user put them into his configuration file
is a
> > recipe for disaster IMO! The example below "-std=c++14 -fcxx-exception"
> > already illustrates a situation in which settings are used that can
break
> > the compilation of many projects!
> >
> >
> In fact invocation of clang without additional argument does not provide
a
> well known state. Actual state is represented by options passed from
clang
> driver to clang compiler, but these options depend on used OS, CPU. In
> theory a system upgrade could cause compilation change. Hardly bare
driver
> arguments can represent compilation options.

This is frankly FUD. While a compiler update can sometimes introduce new
default warnings (or add them to -Wall), those are governed by pretty
strict rules. The far majority of the options, especially the per-OS
per-CPU configuration changes very rarely and few of those changes are
user-visible in the sense that they break valid input.

You are right, the mapping from driver options to compiler ones is
relatively stable now.
My statement was inspired by ICC option -xHost, which selects options
optimal
for the host processor. In any case the set of options passed from driver
to compiler is
more robust characteristic and it differs from user supplied options
substantially.

> Default config file is just a way to influence the magic of option
> calculation made by driver. Config files can be used by clang based SDK
> suppliers to customize clang without need to modify compiler sources. If
> end user plays with default config files, he must know what he does.

But that's the point. Users often don't know or forget. There are long
rants to be found by Google of people forcing -OMG -ffast-math for
everything because they don't understand the implications. I strongly
hope that any vendor shipping a customized default config puts a strong
"Do not change this file without fully understanding what you are doing"
comment in it. The important part is that others are the ones that pay
the price.

Absolutely agree!

Thanks,
--Serge

[spavloff]

> This feature addresses at least two different use cases:
>
> 1. Config file as a way to override compiler defaults.
>
> Compiler already has some default settings and it is hardly possible to
> find the settings that suit everybody. For instance, the warning
> -Wundefined-var-template
> is helpful for some users but annoys others. With config file a user
could
> put new defaults there and thus customize the compiler. No changes to
build
> script is required. Note, this scenario requires just default config
files.

This is *exactly* why I am *strongly* against this. Yes, users do that.
Afterwards, they start to fill random bug reports against projects that
break, because the invocation includes -Werror. This is even worse than
setting a global CFLAGS or CXXFLAGS variable since the expansion of
those are at least generally visible. Hidden user-modifable sources of
flags are a support nightware.

Users are of different categories. What you say is true for end users, who
just use
compiler as a tool. But there are other users, who develop clang based SDKs
for
these end users. They would use this feature with more care because support
problems
would be their problems. Otherwise such user may want to turn off unwanted
warnings by
modifying compiler sources, this is much worse case.

I dare to say that ICC tends to be used in a much more limited
environment. If you want to allow magic environmental variables in your
build script that are invisible for the casual observer, it is not
difficult to add. Well written build systems already support that
anyway.

Environment variable is just another way to specify configuration file, it
does not
need modification of build scripts. Convenient for experiments. If we
support config
file in any form, there is no additional danger to allow setting up
configuration via
environmental variables. It cannot add any option automatically.

Thanks,
--Serge

[Joerg_Sonnenberger]

2016-09-28 16:13 GMT+07:00 Joerg Sonnenberger via cfe-dev <
cfe-dev@lists.llvm.org>:

> > This feature addresses at least two different use cases:
> >
> > 1. Config file as a way to override compiler defaults.
> >
> > Compiler already has some default settings and it is hardly possible to
> > find the settings that suit everybody. For instance, the warning
> > -Wundefined-var-template
> > is helpful for some users but annoys others. With config file a user
> could
> > put new defaults there and thus customize the compiler. No changes to
> build
> > script is required. Note, this scenario requires just default config
> files.
>
> This is *exactly* why I am *strongly* against this. Yes, users do that.
> Afterwards, they start to fill random bug reports against projects that
> break, because the invocation includes -Werror. This is even worse than
> setting a global CFLAGS or CXXFLAGS variable since the expansion of
> those are at least generally visible. Hidden user-modifable sources of
> flags are a support nightware.
>
>
Users are of different categories. What you say is true for end users, who
just use
compiler as a tool. But there are other users, who develop clang based SDKs
for
these end users. They would use this feature with more care because support
problems
would be their problems. Otherwise such user may want to turn off unwanted
warnings by
modifying compiler sources, this is much worse case.

That's why I am not completely opposed to providing target-specific
configuration files.

> I dare to say that ICC tends to be used in a much more limited
> environment. If you want to allow magic environmental variables in your
> build script that are invisible for the casual observer, it is not
> difficult to add. Well written build systems already support that
> anyway.
>

Environment variable is just another way to specify configuration file, it
does not
need modification of build scripts. Convenient for experiments. If we
support config
file in any form, there is no additional danger to allow setting up
configuration via
environmental variables. It cannot add any option automatically.

If the configuration file is (normally) provided via the command line
that gives others at least a chance of finding out what is going wrong.
That's quite different from environment variables, that are often
invisible until you explicitly look for them. History of UNIX is full of
problems with environment variables.

Joerg

[rnk]

My main concern with configuration files is that they should interact with regular command line flags in a predictable and maintainable way. Your approach of extending response files seems like it’s going in the right direction.

I’m not excited about adding something like ELLC’s YAML format to Clang (http://ellcc.org/viewvc/svn/ellcc/trunk/libecc/config/ppc64el-linux?view=markup) because then we will have to document and maintain the semantics of things like multiple configuration files, in what order they override each other, and how they interact with flags. Our command line flag logic is already too complicated. We have one public interface, and it’s flags, and I’m hesitant to add a new one.

[spavloff]

The proposal has been evolved due to feedback. Below is a summary of updated version and feedback from various sources.

Problems

The two main problems this feature addresses are:

1. Ability to modify default compiler settings.

As an example, the warning -Wundefined-var-template can be helpful for people doing module-enabled builds (https://llvm.org/bugs/show_bug.cgi?id=24425), but it makes headache for others, who even come with ideas to turn this warning off by default (http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20160808/167354.html).

To cope with this and similar problem clang need a mechanism that effectively change default settings hardcoded in compiler sources.

2. Support of cross builds.

Doing cross builds requires to specify lots of options that set target, backend options, include and library directories etc. These options could be placed into config files and then be specified with single option or even without such. This feature could make cross compilation with clang more convenient.

Solution

Configuration file groups related options together and allow to specify them in more simple and less error prone way than just listing them somewhere in build scripts. According to the two problems to solve, there are two use patterns of configuration files:

• Default config file is applied without need to specify anything in compiler invocation. It can be read from predefined location or be specified by environmental variable. It allows to override compiler defaults.

• Named config file is specified explicitly by a user, either by command line option --config, or by using special executable file names, such as armv7l-clang. Such configuration may be thought as a “macro” that designates an option set and is expanded when compiler is called.

Only one config file is used, if it is specified explicitly, default config file is not searched. Config file may contain comments (# style), references to other config files using syntax @file.

Config file are searched in the directory where clang executable resides. User may specify any config file using option --config followed by full path of the file. Alternatively the path to config file may be specified in environmental variable CLANGCFG.

Existing experience

Configuration file is not a new concept, it has been used in various form in many compilers including clang.

1. Intel compiler supports default configuration files (https://software.intel.com/en-us/node/522780), the file is searched for in the directory where executable resides or can be specified by setting environmental variable. The feature is pretty convenient in practice.

2. Clang allows to specify target by using special executable names, like armv7l-clang. Only target name can be specified by this way, which is not sufficient to tune compiler for a new target. Obvious question arise - why not extend this mechanism for other target specific options too.

3. Clang allows modifying options using environmental variable CCC_OVERRIDE_OPTIONS. This is undocumented way and is looks more like a hack for internal use. Its existence however indicates that such functionality is useful.

4. Cross compilation environment ELLCC (http://ellcc.org), based on clang, supports advanced config files in YAML format. It has proven to be very handy to support a wide variety of targets.

5. GCC has support of spec files (https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html), which can be taken from default location or be specified by command line option --spec. Spec files are more powerful tool than just config files as they not only set options but also program driver logic. Using custom spec files allows GCC to be tuned to particular target without need to specify lots of options in command line.

6. Visual Studio supports custom project templates (https://msdn.microsoft.com/en-us/library/xkh1wxd8.aspx), which may specify any compiler flags as default setting. Although it can be considered as feature of build system rather than compiler, it allows to override default compiler settings.

7. Keil C compilers allow to specify compiler options in environmental variables, ARMCOMPILER6_CLANGOPT (http://www.keil.com/support/man/docs/armclang_intro/armclang_intro_chr1374139991387.htm) or ARMCC5_CCOPT (http://infocenter.arm.com/help/topic/com.arm.doc.dui0592e/pge1399635613963.html) depending on compiler version.

Concerns

Somewhat random set of concerns extracted from discussions with answers to them.

1. With default config files the options specified for clang in command line do not represent definite state anymore, because some options may come from the config file. A user that prepares bug report can forget to specify content of the used default config file, it can make difficult to reproduce failure conditions.

Answer: Actual set of compiler options is provided in error dump files, in output of clang -v and clang -###, these invocations also report used config file. If a project wants to disable some warnings by default, add it to the Makefile.

2. It is not a business of compiler to adjust options. What flags to pass to the compiler should be configured in a project’s build system.

Answer: A project may include dozens of components each come with own build systems, which are tuned differently. Retargeting such build system can be time consuming and error prone. On the other hand, setting compiler and target specific options is not a business of a component, ideally it may be unaware of using in cross compilation environment. Also, not all component build systems are flexible enough. As a result some users do not want modify their build files, they would prefer customized compiler.

Config files brings more power and convenience to the compiler.

3. Clang already has @file for bundling up compiler flags.

Answer: Config file make this feature a bit more convenient by providing a way to choose proper option set and making files containing options more user friendly by allowing comments and nested @file. Also @file cannot solve the problem of overriding compiler defaults.

4. The idea of having the compiler search for config files in random places on the filesystem seems scary.

Answer: Default config files are searched only in the directory where compiler executable file resides. If compiler is installed from package, this file is placed in system directories and should not be changed by a simple user. It is assumed that such file is a part of SDK or some package prepared by experienced person, it is not changed frequently. The same applies to target-specific config files. A user may specify full path to config file, in this case the file may be anywhere but this explicit action.

Most of elaborated environments allows search for libraries, packages, classes etc in some predefined places. Running simple command gcc file.c make compiler search set of various places, and the set may change depending on version and OS variant.

5. There is already a lot of magic in the driver to figure out how to invoke the tools, this feature makes driver more complex.

Answer: Orchestrating tools made by clang driver is already a complex task and hardly it can be made simple. But it can be made more structured, config file may be a step in this direction making existing target selection in clang more powerful and useful.

6. This is too primitive solution:

• A good solution must be able to deduce target from arguments (-m32).

• We should also probably automatically silence unused options from config files. For example, if my cross-compile toolchain passes some extra linker arguments with -Wl,-foo, then I don’t want every single line in my build to complain that -Wl is unused in conjunction with -c.

• It should provide default mapping from targets to sysroots and associated paths; we currently hardcode a lot of this into the compiler itself.

• Any workable config-file scheme must interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target.

• Maybe it’d be better to create a regular, .ini/.conf-style configuration file.

Answer: This is a simple solution, that solves a good deal of problems. It does not require big changes in documentation and substantial maintenance efforts. It can be extended in future by allowing more complex constructs in config files.

[David_Chisnall3]

It would be nice if a proposed solution resulted in a net decrease in total complexity for the driver, with more of the code moved into default config files.

David

[spavloff]

Hi all,

Implementation is presented in https://reviews.llvm.org/D24933. It is limited to the case in which user explicitly specifies config file, either by option --config or by using mangled compiler name, like armv7l-clang. Its main goal is to simplify use of clang when command line contains many options that are the same for many invocations. The other use case is default config file, which is applied automatically. This case is not presented in this implementation to simplify review process.

To move forward additional feedback is necessary, mainly to decide whether this feature is worth implementing. Config files may be convenient in the cases in cross-compilation. So far, there is a positive opinion about using config files in clang based cross compilation environment: http://lists.llvm.org/pipermail/cfe-dev/2016-September/050783.html.

Some express concern that this is not a useful feature, the main points are listed below.

1. Config files vs @file

There is an opinion that the functionality provided by config files is already available through the directive @file. Both config file and @file specify a set of command line options, but config file has some advantages:

• It searches well-known places for files, while @file must search only the current directory,
• It may include other config file, they are resolved relative to including file, making it possible to move set of related files across file system. Nested @file searches only current directory. Coupled with above note it means that using @file directive requires all files be specified by absolute paths, if we want to reference the same files during build process.
• Config files may contain comments,
• Options from config file do not produce warning ‘unused option’ even if they are indeed unused.

Ability to contain comments is not only a convenience for maintenance. It can be useful for extending functionality. We can assume, for instance, that if the first line is a comment of some special form, such as #!yaml or # format = xml or some other, then it specifies format of the config file. This would allow extending functionality of config files in future without breaking compatibility.

2. More elaborated solutions

Many are of opinion that config file in this form is too primitive solution, for instance:

• A good solution must be able to deduce target from arguments (-m32),
• Any workable config-file scheme must interact with the various target specification command-line arguments, and allow for setting options dependent upon the actually-selected target,
• Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set of options, it cannot solve these problems. However it can be extended in future so that, for instance, the following construct could be allowed (particular syntax does not matter):

…
#if (argv.contains(“-m32”))
-target i686-unknown-linux-gnu
#else
-targer x86_64-unknown-linux-gnu
#endif
…

This syntax would require to support if/else/endif control construct, and builtin function argv.contains. Ability to include an option conditionally is likely to be sufficient to solve large part of the problems mentioned above and can indeed reduce complexity of the driver.

Speaking about ways to make clang driver more flexible and less complex, we need to mention an approach implemented in gcc as spec files (https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html). A config file, even in the extended form, is only a way to specify a set of options for the driver. A spec file is a set of instructions for the driver how to handle a particular input file. These two solution have different ecological niches and can coexist. A regular user may prepare a set of config files for their needs, these files would capture user-specific information, such as where various components are placed in the user environment. A spec file is more compiler specific, if it needs user information, it would get it from options or environmental variables. As a part of spec file interpretation, the driver must build command lines for compiler, linker and other tools, functionality of config file may be useful in this case too.

In comparison with the more elaborated solutions mentioned above, the proposed implementation of config files is much simpler and if it is useful, it can be a first step toward more powerful and convenient driver.

[Finkel_Hal_J]

I chatted with Richard about this in person a couple of weeks ago, and he seems happy with the general direction here. One comment he had was that it seems like @file processing is a restricted case of config-file processing. As a result, it might make sense to replace the current @file-processing logic with the config-file processing logic (just with a bunch of features turned off). This is appealing, but I think that we really need to be careful here. We don’t want to create a system where people are encouraged to replicate complex driver logic in the config files in order to independently figure out which features might be enabled. Of course, we already do this, but the logic ends up in wrapper scripts (and is just as likely to be broken, if not more so). However, if we’re going to create a solution here, we should do so in a way that does not encourage the confg-file writer to duplicate driver logic. In any case, I definitely consider this follow-up work. Thanks again, Hal

[spavloff]

Hi Hal,

Thank you very much for your interest and help!

I chatted with Richard about this in person a couple of weeks ago, and he
seems happy with the general direction here. One comment he had was that it
seems like `@file` processing is a restricted case of config-file
processing. As a result, it might make sense to replace the current
`@file`-processing logic with the config-file processing logic (just with a
bunch of features turned off).

The main issue of using `@file` for loading configuration is compatibility.
The construct `@file` has been used for ages and there must be lots of
build scripts using it, the change of semantics must not break them. It
looks like the features that may be added to the file format in `@file`
construct without disturbing existing functionality are:
- comments,
- splitting long lines with trailing '\',
- suppressing warnings on unused options.
The remaining feature is resolving names in nested `@file` constructs, it
must keep current behavior (resolve names relative to current directory).
This makes nesting constructs `@file` inconvenient it not useless, but it
does not look like a big loss.

Configuration encoded in compiler file name, as in `armv7l-clang` is not
related to `@file`, so this functionality implemented in
https://reviews.llvm.org/D24933 remains unchanged. Explicit specification
of config file in command line should be made by `@file` option, there are
several issues as compared to the proposed `--config` option:
- config file must be specified by its full path otherwise it won't be
found if the current directory is changed by build script. This limitation
should not be a big problem if options are specified in build scripts, but
for manual invocations in command line it is inconvenient.
- command line may contain several `@file` constructs, there may be options
before `@file`. If we want to distinguish config file from others (for
instance, we want to suppress unused option warnings only in such files),
we need to assume that only the first option may load it.

So if we are ready to sacrifice some convenience, config files can be
implemented on top of `@file` construct. If there are no objections I'll
update the patch.

2. More elaborated solutions

Many are of opinion that config file in this form is too primitive
solution, for instance:
- A good solution must be able to deduce target from arguments (-m32),
- Any workable config-file scheme *must* interact with the various target
specification command-line arguments, and allow for setting options
dependent upon the actually-selected target,
- Config file must reduce complexity of clang driver.

The presented implementation of config files is only a way to specify set
of options, it cannot solve these problems. However it can be extended in
future so that, for instance, the following construct could be allowed
(particular syntax does not matter):

    …
    #if (argv.contains("-m32"))
    -target i686-unknown-linux-gnu
    #else
    -targer x86_64-unknown-linux-gnu
    #endif
    …

This syntax would require to support `if/else/endif` control construct,
and builtin function `argv.contains`. Ability to include an option
conditionally is likely to be sufficient to solve large part of the
problems mentioned above and can indeed reduce complexity of the driver.

This is appealing, but I think that we really need to be careful here. We
don't want to create a system where people are encouraged to replicate
complex driver logic in the config files in order to independently figure
out which features might be enabled. Of course, we already do this, but the
logic ends up in wrapper scripts (and is just as likely to be broken, if
not more so). However, if we're going to create a solution here, we should
do so in a way that does not encourage the confg-file writer to duplicate
driver logic.

In any case, I definitely consider this follow-up work.

This was a bad example. To obtain more powerful solution clang need
something like gcc spec files. This is completely different topic.

Thanks,
--Serge

[Finkel_Hal_J]

Right. Can’t you just make it configurable? If parsing an @file, do the old thing, otherwise, do the new thing? Maybe I’m missing something, but I think you’re taking this the wrong way. I don’t recommend giving up features in this sense. I think the question is: can we replace the current @file processing with the new config-file logic (just with some of the new features turned off)? If that can’t be done reasonably, then I think that’s okay too. We just should understand (and document) why. -Hal

[Finkel_Hal_J]

To elaborate slightly, we currently support @file in Clang by calling llvm::ExpandResponseFiles in Clang’s driver. Since this is general LLVM functionality, and the config-file paring is more Clang-specific, maybe this is not worth changing. However, what if we processed @file like config files, but in a mode which was backward compatible with existing @file semantics? -Hal

[spavloff]

Thanks,
--Serge

It looks like the features that may be added to the file format in `@file`
construct without disturbing existing functionality are:

- comments,
- splitting long lines with trailing '\',
- suppressing warnings on unused options.
The remaining feature is resolving names in nested `@file` constructs, it
must keep current behavior (resolve names relative to current directory).
This makes nesting constructs `@file` inconvenient it not useless, but it
does not look like a big loss.

Can't you just make it configurable? If parsing an `@file`, do the old
thing, otherwise, do the new thing?

Configuration encoded in compiler file name, as in `armv7l-clang` is not
related to `@file`, so this functionality implemented in
https://reviews.llvm.org/D24933 remains unchanged. Explicit specification
of config file in command line should be made by `@file` option, there are
several issues as compared to the proposed `--config` option:
- config file must be specified by its full path otherwise it won't be
found if the current directory is changed by build script. This limitation
should not be a big problem if options are specified in build scripts, but
for manual invocations in command line it is inconvenient.
- command line may contain several `@file` constructs, there may be
options before `@file`. If we want to distinguish config file from others
(for instance, we want to suppress unused option warnings only in such
files), we need to assume that only the first option may load it.

So if we are ready to sacrifice some convenience, config files can be
implemented on top of `@file` construct. If there are no objections I'll
update the patch.

Maybe I'm missing something, but I think you're taking this the wrong way.
I don't recommend giving up features in this sense. I think the question
is: can we replace the current `@file` processing with the new config-file
logic (just with some of the new features turned off)?

If that can't be done reasonably, then I think that's okay too. We just
should understand (and document) why.

I understood the question as "can we load config files with @file not with
new option", there were such opinions in this thread above. If the question
is only about enhancements to processing of `@file`, the reply is yes. In
the LLVM part of this patch config file parsing in implemented on top of
`@file` processing (https://reviews.llvm.org/D24926,
function `cl::tokenizeConfigFile`). It seems to me that comments and using
'\' to glue lines may be enabled in `@file` as well, as an extension. Other
features may be enabled for config files only but clang must somehow know
that the file contains configuration.

Thanks,
--Serge

[Finkel_Hal_J]

Sounds good. I’ll look at the patches. -Hal

[jyknight]

I'd really like to at least have a *design* for how this can eventually
incorporate target-specific options before moving forward with adding a
--config option, even if the initial commit won't implement the full design.

I don't believe hand-wavy "maybe we'll add syntax that looks kinda like a
comment so older versions will ignore it" is good enough there.

I'd like to again keep in mind the use-case I mentioned a while ago.
Approximately every linux distro configures GCC to set their default target
cpu levels. E.g., Debian seems to set the following:
- On x86, the default CPU should be i686.
- But on x86-64, the default CPU isn't changed (aka it's left as "x86-64").
- For ppc32/ppc64, the default CPU should be power7 but tune for power8.
- For ppc64le, the default CPU should be power8.
- On ARM (hf), armv7-a should be the default cpu, vfpv3-d16 the default
fpu, and it should default to thumb mode.
etc...

Note that those defaults are different on different releases of the distro.

The way you do this with GCC is via options passed to the configure script:
--with-arch-32= --with-arch-64= --with-fpu= --with-mode= etc. which turn
into values used in the target-specific OPTION_DEFAULT_SPECS macro. Since
GCC only builds for one target at a time (or two, if you count 32/64
separately), and you're expected to need to build a new gcc any time you
want to cross-compile, that's sufficient.

Clang is intrinsically a cross-compiler, so gcc's solution isn't good
enough for clang (nor is clang's current behavior enough). So, what's the
plan to actually solve this?

1. There needs to a way to be able to configure the defaults for all
supported architectures of the platform (either in a single config, or in
multiple that clang knows how to select).

2. Those platform defaults should, somehow, avoid interfering with the use
of clang to cross-compile TO a different platform, and be easy to use FROM
a different host platform. (find default config via sysroot, maybe?)

3. How do we recommend users select a target?
Do we need to look for the proper defaults when the user specifies
"--target $TARGET" to clang?
Or maybe we favor of the "$TARGET-clang" symlink method?
Or maybe "--target" is a low-level feature not recommended for end-users,
and we should steer people to using something like "-arch", to select a
named architecture within the current platform configuration, like apple
does with darwin-based platforms now?

Again, I'm not saying the first patch needs to implement everything, but I
do think a design should exist on how this is going to work.