[RFC] Adding a default file location to config file support

I’d like to propose an extension to the config file support. The current behaviour of searching for the config file based off of the command being specified works almost the way we need. The limitation with the current behaviour is that if the user specifies the --config option, clang only reads that file. We have a need on z/OS for an installation to have a “base” config file that is always read in by clang even if the user specifies --config or uses the command name to find the config file. The options specfied in this base config file would be ones that would be needed by any compilation. An example of this for z/OS is the data sets containing the equivalent of libc & libc++. This base config file would be an optional file in the installation directory of clang (eg. install-dir/etc/clang.cfg)

I am proposing adding an extra step before reading the config file that would look for and read this base config file. The option gather process will become:

  1. look for the base config file (install-dir/etc/clang.cfg). If found, read the options.
  2. look for the config file using --config & command name. If found, read the options
  3. get options from the command line.

I’d also like to provide an environment variable so that the base config file doesn’t have to be located in the installation dir. The current thought is to make this environment variable a search path containing one or more directories. I’m proposing the name of the environment variable to be CLANG_CFG_PATH.

I had considered changing clang so multiple --config options can be specified and each file would be read. This would also include the --config option not overriding the command name based file. The only problem with this is it makes a significant change in behaviour that could impact existing usage of the compiler.

This extension seems like something that can be useful on other platforms and would like feedback on adding this for all platforms.

Why do you need to have a “base” configuration for z/OS, can you instead put the defaults you need in the clang driver?

I don’t really like having default config files, because I think it’s confusing for users and it just adds one more variable that developers need to consider when troubleshooting issues. I do think supporting multiple --config options would be useful.

An example of this for z/OS is the data sets containing the equivalent of libc & libc++. This base config file would be an optional file in the installation directory of clang (eg. install-dir/etc/clang.cfg)

Even in this case, it seems to me that users might want to override the base configuration; e.g., to use a modified or alternate set of libc and libc++ files. How would the user bypass the base configuration in that case?

The ability to specify multiple --config options would presumably allow a base configuration file to be added on a target dependent basis; the driver could add such options based on target. I like this approach better.

How would the base configuration file work when cross compiling?

Base config file facility may be implemented by adding @base_config_file at the beginning of relevant config files. This solution however is not convenient enough, because base_config_file must be specified either by absolute path, or relative to the specified config file. It is because the body of config file is processed by the same procedure that is used to expand response files. It substantially limits config file mechanism.

If location of included config files could be specified more flexibly, it would make config files more useful. I think implementation of environment variable like CLANG_CFG_PATH is a good idea.

The location of the libc/libc++ libraries is done via options. The base config file would have the normal location for the system. The user can also specify those options on the command line. Those options use the last one wins rule so the user option will override the one in the base config file.

I don’t think this base config file would really come into play with cross compiling. The way I would envision cross compiling working is the base config file, if needed, would provide options needed for native or any cross compilation. The user would then use the --config option or use the target dependent config file to specify a config file that provides all of the options needed for cross compilation.

The defaults can’t be put into the clang driver. The problem is that the location of the system libraries can be different for each machine. There is one usual location but that location can be changed by the sys admin while installing the OS. The location is not stored in some well known location so the driver can’t just look it up.

The multiple config file option doesn’t actually solve the problem we have on z/OS. I missed this in the original post, but the multiple config files works fine for one or more --config options and up to one target dependent file. We need this base config file to be read even if the user provides a target dependent config file.

We can actually do both. We can support multiple config files on top of this extension to add the base config file. I’d like to proceed with this proposal as multiple config files doesn’t quite handle what we need and then tackle multiple config files as a separate feature.

Who do you think will be using this feature for z/OS? Will it be OS maintainers building the OS or will it be end users building their own applications?

The people creating these base config file will be the sys admins of customers installing the compiler on the their systems. I can also see anyone who downloads and builds clang for their own use possibly creating the base config files too.

A base config file would need to be created at the time of installing the compiler if the system libraries are not in the usual location. So in general anyone installing the compiler could need to create one of these base config files.

I think something that would be more useful for you and the rest of the community is to have clang find cfg files based on the target. I suggested something like that in this thread: Re-land -stdlib=libc++ support in clang-cl - #2 by tobiashieta

1 Like

This seems at odd with your goal of using the base config file to specify the locations of the (presumably z/OS specific) libc and libc++ files. When cross compiling, one often needs to specify a target dependent set of such files.

My assumptions may not match yours. I’ve been assuming that, for the z/OS scenario, the base config file would specify options like -isysroot or -isystem, but perhaps you have something else in mind.

I don’t think we are too far off from each other. I think the difference is I’m limiting cross-compiling to just the compilation step, not linking. The base config file would still be read in during the link step so we could easily make it work for linking during cross-compilation.

Your assumption about setting options like -isysroot and -isystem aren’t far off from the options we are adding. The -isysroot doesn’t work for us since on z/OS we have two different file hierarchies to work with. For example, the linker is /bin/ld and the C headers are in /usr/include. The default for -isysroot works for these. However, all of the shared libraries are in data sets (eg. SYS1.SCEELIB). A sys admin can choose another name for SYS1. That is the information we need to pass as an option in the base config file.

I read over your reference post. I think I agree with that. I may not fully follow what you are saying as what you have in the post sounds like what the documentation already states.

My understanding is that your idea is (or would be) handled by the existing config file processing that uses the command name & target to determine which config file name to look for.

We could merge this base config file idea if we implemented the idea of multiple config files and some kind of include next mechanism. The include next mechanism would be needed to handle the case where the user wants to create their own target config file which augments the already existing target config file. The multiple config files won’t be simple because they support the @ to include additional files. We will end up needing to deal with duplicate inclusions and probably some other issues that we have with header files. At that point we should be able to have a solution that encompasses everything into a single design.

In the meantime, I’d like to continue on this path.

My concern has to do with use of a base configuration file to specify target dependent options. I think that would be fine if the base configuration file supported conditional target dependent directives such that options could be conditionally added based on the selected target. From a design perspective, I don’t think an unconditionally included base configuration file makes sense without such support. I think something like the following would be reasonable (this of course depends on the ability to identify the target before the base configuration file is included).

!if target(zos-*)
-L'//SYS1.SCEELIB'
!endif

I think there was some misunderstanding regarding the suggestion of having the driver add target dependent options as a solution for this issue. If multiple --config options were supported, then the driver, when invoked for a z/OS target, could add something like --config zos.cfg. The compiler installation or administrator could then edit the zos.cfg file as needed.

This sounds kind of similar to the hardcoded support in Clang for reading a special file in /etc on Gentoo to find the GCC installation path. See Generic_GCC::GCCInstallationDetector::ScanGentooGccConfig.

Note that this is entirely distinct from the config-file mechanism. This mechanism allows end-users on Gentoo to configure which version of libstdc++/etc to use (if multiple are installed), and for Clang to find the correct one.

I don’t know if adding new hardcoded support to read a custom file to find the correct path on zOS is actually a good idea, but maybe worth considering if the generic config-file stuff doesn’t seem to be meeting requirements.

1 Like

I think a base config file still makes sense. To support multiple targets, I do agree that multiple config files with an include next mechanism is a better solution. We would encourage people to use the target specific config file names along with the CLANG_CONFIG_DIR and the include next mechanism to add target specific options to the compilation. I can start an RFC to hash out the multiple config file support and how it incorporates this base config file.

The base config file would still play a useful role to options that are target independent.

Thanks for pointing that support out. I took a quick look at the code. I was hoping that we might be able to combine these into a basis for multiple config support but it looks like the config file this code is scanning is used by other tools and specific to that environment.

The requirement for z/OS is similar to this in that in both cases the information in the config needs to be used even if the user specifies a config file. The difference is that on z/OS we aren’t restrained by a pre-existing file format and could leverage the existing option processing more.

That sounds good.

The base config file would still play a useful role to options that are target independent.

I don’t disagree in principle, but it isn’t clear to me that much motivation for a target independent base config file exists; the options that you have expressed interest in are target dependent.

Yeah the rationale I used to present this is target dependent. However, I can see a case (and we do have some) where someone will want to add some options that should be applied to all compilations regardless of target.

However, I can see a case (and we do have some) where someone will want to add some options that should be applied to all compilations regardless of target.

I can certainly see use cases as well. Enumerating the use cases you have in mind would be helpful to build motivation for support of a base config file.

One example would be if a site had some coding standards and wanted to use some -W options to enforce them.