--gcc-toolchain (GCC_INSTALL_PREFIX) and --prefix(-B)

If --gcc-toolchain is specified, its value overrides the cmake variable GCC_INSTALL_PREFIX.
When the value is non-empty: the value is appended to the --prefix list and is used to detect GCC installations.
The GCC installation is used to provide include directories/library directories and some startup files (e.g. crtbegin).

Problem 1.

–prefix(-B) does more than --gcc-toolchain: clang::driver::Driver::GetProgramPath basically searches for $prefix/$triple-$file and $prefix$file,
where $prefix is taken from the list of --prefix(-B). --gcc-toolchain does not participate in the search.
The result is that ‘ld’ and ‘as’ may come from the system (more precisely, sysroot):

cd clang/test/Driver

Make sure Inputs/opensuse_42.2_aarch64_tree/usr/bin/ld exists.

clang -target aarch64-suse-linux --gcc-toolchain=Inputs/opensuse_42.2_aarch64_tree/usr ‘-###’ gcc-toolchain.cpp -v

I have ld in my /usr/local/bin and it takes precedence over /usr/bin/ld

“/usr/local/bin/ld” “-EL” “–eh-frame-hdr” “-m” “aarch64linux” “-dynamic-linker” “/lib/ld-linux-aarch64.so.1” “-o” “a.out” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crt1.o” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crti.o” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtbegin.o” “-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8” “-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64” “-L/lib/aarch64-linux-gnu” “-L/lib/…/lib64” “-L/usr/lib/aarch64-linux-gnu” “-L/usr/lib/…/lib64” “-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…” “-L/tmp/RelA/bin/…/lib” “-L/lib” “-L/usr/lib” “/tmp/gcc-toolchain-f87f08.o” “-lgcc” “–as-needed” “-lgcc_s” “–no-as-needed” “-lc” “-lgcc” “–as-needed” “-lgcc_s” “–no-as-needed” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtend.o” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crtn.o”

The -L and crt* files are indeed from Inputs/opensuse_42.2_aarch64_tree/usr, but ld (and as if -fno-integrated-as) is from the system.
On many Linux distributions you can normally assume that the system ld and as only support the host architecture.
This means --gcc-toolchain can only be used to specify a GCC installation with the same architecture.
–prefix can make as and ld paths correct, but: if another --prefix is needed, why do we use --gcc-toolchain?

I have sent a patch to document the current state: https://reviews.llvm.org/D97902

Problem 2.

Non-empty --gcc-toolchain has one nicer property: it suppresses GCC installation detection in the sysroot.
Now let me alter the command a bit:

“/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…/…/aarch64-linux-gnu/bin/ld” “-EL” “–eh-frame-hdr” “-m” “aarch64linux” “-dynamic-linker” “/lib/ld-linux-aarch64.so.1” “-o” “a.out” “/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…
/…/aarch64-linux-gnu/lib/crt1.o” “/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…/…/aarch64-linux-gnu/lib/crti.o” “/usr/lib/gcc-cross/aarch64-linux-gnu/10/crtbegin.o” “-L/usr/lib/gcc-cross/aarch64-linux-gnu/10” “-L/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…/aarch64-linux-gnu” “-L/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…/…/lib64” “-L/lib/aarch64-linux-gnu” “-L/lib/…/lib64” “-L/usr/lib/aarch64-linux-gnu” “-L/usr/lib/…/lib64” “-L/usr/lib/aarch64-linux-gnu/…/…/lib64” “-L/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…/…/aarch64-linux-gnu/lib” “-L/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…” “-L/tmp/RelA/bin/…/lib” “-L/lib” “-L/usr/lib” “/tmp/gcc-toolchain-b491cd.o” “-lgcc” “–as-needed” “-lgcc_s” “–no-as-needed” “-lc” “-lgcc” “–as-needed” “-lgcc_s” “–no-as-needed” “/usr/lib/gcc-cross/aarch64-linux-gnu/10/crtend.o” “/usr/lib/gcc-cross/aarch64-linux-gnu/10/…/…/…/…/aarch64-linux-gnu/lib/crtn.o”

I have installed an aarch64 cross gcc. Because its version is larger than the version of the GCC installation under --prefix, the system gcc-cross/aarch64-linux-gnu takes precedence.
This behavior looks a bit unfortunate. Should we let the first --prefix win and drop future --prefix and default system installations?

[1]:

The logic is around https://github.com/llvm/llvm-project/blob/main/clang/lib/Driver/ToolChains/Gnu.cpp#L1910

Prefixes = --prefix/-B list (only the directory subset is effective)
StringRef GCCToolchainDir = --gcc-toolchain= or CMake variable GCC_INSTALL_PREFIX
if (GCCToolchainDir != “”) {
Prefixes.push_back(std::string(GCCToolchainDir));
} else {
if (!D.SysRoot.empty()) {
Prefixes.push_back(D.SysRoot);
// Add D.SysRoot+"/usr" to Prefixes. Some distributions add more directories.
AddDefaultGCCPrefixes(TargetTriple, Prefixes, D.SysRoot);
}

// D.InstalledDir is the directory of the clang executable, e.g. /usr/bin
Prefixes.push_back(D.InstalledDir + “/…”);

if (D.SysRoot.empty())
AddDefaultGCCPrefixes(TargetTriple, Prefixes, D.SysRoot);
}

// Gentoo / ChromeOS specific logic.
// I will move this block in https://reviews.llvm.org/D97894
if (GCCToolchainDir == “” || GCCToolchainDir == D.SysRoot + “/usr”) {

}

// Loop over the various components which exist and select the best GCC
// installation available. GCC installs are ranked by version number.
Version = GCCVersion::Parse(“0.0.0”);
for (const std::string &Prefix : Prefixes) {
auto &VFS = D.getVFS();
if (!VFS.exists(Prefix))
continue;

// CandidateLibDirs is a subset of {/lib64, /lib32, /lib}.
for (StringRef Suffix : CandidateLibDirs) {
const std::string LibDir = Prefix + Suffix.str();
if (!VFS.exists(LibDir))
continue;
bool GCCDirExists = VFS.exists(LibDir + “/gcc”);
bool GCCCrossDirExists = VFS.exists(LibDir + “/gcc-cross”);

// Precise match. Detect $Prefix/lib/$–target
ScanLibDirForGCCTriple(TargetTriple, Args, LibDir, TargetTriple.str(),
false, GCCDirExists, GCCCrossDirExists);
// Usually empty.
for (StringRef Candidate : ExtraTripleAliases) // Try these first.
ScanLibDirForGCCTriple(TargetTriple, Args, LibDir, Candidate, false,
GCCDirExists, GCCCrossDirExists);
// CandidateTripleAliases is a set with “x86_64-linux-gnu”, “x86_64-unknown-linux-gnu”, …
// This loop detects directories like $Prefix/lib/x86_64-linux-gnu.
for (StringRef Candidate : CandidateTripleAliases)
ScanLibDirForGCCTriple(TargetTriple, Args, LibDir, Candidate, false,
GCCDirExists, GCCCrossDirExists);
}
for (StringRef Suffix : CandidateBiarchLibDirs) {
const std::string LibDir = Prefix + Suffix.str();
if (!VFS.exists(LibDir))
continue;
bool GCCDirExists = VFS.exists(LibDir + “/gcc”);
bool GCCCrossDirExists = VFS.exists(LibDir + “/gcc-cross”);
for (StringRef Candidate : CandidateBiarchTripleAliases)
ScanLibDirForGCCTriple(TargetTriple, Args, LibDir, Candidate, true,
GCCDirExists, GCCCrossDirExists);
}
}

If --gcc-toolchain is specified, its value overrides the cmake variable GCC_INSTALL_PREFIX.
When the value is non-empty: the value is appended to the --prefix list and is used to detect GCC installations.
The GCC installation is used to provide include directories/library directories and some startup files (e.g. crtbegin).

Problem 1.

–prefix(-B) does more than --gcc-toolchain: clang::driver::Driver::GetProgramPath basically searches for $prefix/$triple-$file and $prefix$file,
where $prefix is taken from the list of --prefix(-B). --gcc-toolchain does not participate in the search.
The result is that ‘ld’ and ‘as’ may come from the system (more precisely, sysroot):

cd clang/test/Driver

Make sure Inputs/opensuse_42.2_aarch64_tree/usr/bin/ld exists.

clang -target aarch64-suse-linux --gcc-toolchain=Inputs/opensuse_42.2_aarch64_tree/usr ‘-###’ gcc-toolchain.cpp -v

I have ld in my /usr/local/bin and it takes precedence over /usr/bin/ld

“/usr/local/bin/ld” “-EL” “–eh-frame-hdr” “-m” “aarch64linux” “-dynamic-linker” “/lib/ld-linux-aarch64.so.1” “-o” “a.out” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crt1.o” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crti.o” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtbegin.o” “-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8” “-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64” “-L/lib/aarch64-linux-gnu” “-L/lib/…/lib64” “-L/usr/lib/aarch64-linux-gnu” “-L/usr/lib/…/lib64” “-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…” “-L/tmp/RelA/bin/…/lib” “-L/lib” “-L/usr/lib” “/tmp/gcc-toolchain-f87f08.o” “-lgcc” “–as-needed” “-lgcc_s” “–no-as-needed” “-lc” “-lgcc” “–as-needed” “-lgcc_s” “–no-as-needed” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtend.o” “Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crtn.o”

The -L and crt* files are indeed from Inputs/opensuse_42.2_aarch64_tree/usr, but ld (and as if -fno-integrated-as) is from the system.
On many Linux distributions you can normally assume that the system ld and as only support the host architecture.
This means --gcc-toolchain can only be used to specify a GCC installation with the same architecture.

… and specifying a GCC installation with the same architecture (e.g., to use a newer libstdc++) is something that people do:
$ clang++ -fuse-ld=ld --gcc-toolchain=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as <<<’#include <stdio.h>’$’\n’‘int main(void) { printf(“Hello, world!\n”); }’ && echo OKAY
OKAY
… but it does seem ld is coming from --gcc-toolchain for me anyway:
$ ./build/bootstrap/stage1/build/bin/clang++ -fuse-ld=ld --gcc-toolchain=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as <<<’#include <stdio.h>’$’\n’‘int main(void) { printf(“Hello, world!\n”); }’ -### 2>&1 | grep /ld
“/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…/…/bin/ld” “–hash-style=gnu” “–eh-frame-hdr” “-m” “elf64lppc” “-dynamic-linker” “/lib64/ld64.so.2” “-o” “a.out” “/lib/…/lib64/crt1.o” “/lib/…/lib64/crti.o” “/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtbegin.o” “-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7” “-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…/…/lib64” “-L/lib/powerpc64le-linux-gnu” “-L/lib/…/lib64” “-L/usr/lib/powerpc64le-linux-gnu” “-L/usr/lib/…/lib64” “-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…” “-L/data1/hstong.local/wybuild/build/bootstrap/stage1/build/bin/…/lib” “-L/lib” “-L/usr/lib” “/tmp/–8f8c6a.o” “-lstdc++” “-lm” “-lgcc_s” “-lgcc” “-lc” “-lgcc_s” “-lgcc” “/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtend.o” “/lib/…/lib64/crtn.o”

–prefix can make as and ld paths correct, but: if another --prefix is needed, why do we use --gcc-toolchain?

Well, I’m not sure that another --prefix is needed. Also, --prefix doesn’t help to pick up the GCC header and library paths and --sysroot overrides too much:
$ clang++ -fuse-ld=ld --sysroot=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as <<<’#include <stdio.h>’$’\n’‘int main(void) { printf(“Hello, world!\n”); }’
:1:10: fatal error: ‘stdio.h’ file not found
#include <stdio.h>
^~~~~~~~~
1 error generated.

Seems that it might do the library paths, but not the header paths:
$ clang++ -fuse-ld=ld --prefix=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as <<<’#include <stdio.h>’$’\n’‘int main(void) { printf(“Hello, world!\n”); }’ -###
clang version 13.0.0
Target: powerpc64le-unknown-linux-gnu
Thread model: posix
InstalledDir: /data1/hstong.local/wybuild/./build/bootstrap/stage1/build/bin
“/data1/hstong.local/wybuild/build/bootstrap/stage1/build/bin/clang-12” “-cc1” “-triple” “powerpc64le-unknown-linux-gnu” “-S” “-disable-free” “-main-file-name” “-” “-mrelocation-model” “static” “-mframe-pointer=all” “-fmath-errno” “-fno-rounding-math” “-no-integrated-as” “-mconstructor-aliases” “-munwind-tables” “-target-cpu” “ppc64le” “-mfloat-abi” “hard” “-target-abi” “elfv2” “-fno-split-dwarf-inlining” “-debugger-tuning=gdb” “-fcoverage-compilation-dir=/data1/hstong.local/wybuild” “-resource-dir” “/data1/hstong.local/wybuild/build/bootstrap/stage1/build/lib/clang/13.0.0” “-internal-isystem” “/data1/hstong.local/wybuild/build/bootstrap/stage1/build/lib/clang/13.0.0/include/ppc_wrappers” “-internal-isystem” “/usr/local/include” “-internal-isystem” “/data1/hstong.local/wybuild/build/bootstrap/stage1/build/lib/clang/13.0.0/include” “-internal-externc-isystem” “/include” “-internal-externc-isystem” “/usr/include” “-fno-dwarf-directory-asm” “-fdebug-compilation-dir=/data1/hstong.local/wybuild” “-ferror-limit” “19” “-fno-signed-char” “-fgnuc-version=4.2.1” “-fcolor-diagnostics” “-o” “/tmp/–dbf5fb.s” “-x” “c” “-”
“/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…/…/bin/as” “-a64” “-mppc64” “-mlittle-endian” “-mpower8” “-o” “/tmp/–d53b10.o” “/tmp/–dbf5fb.s”
“/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…/…/bin/ld” “–hash-style=gnu” “–eh-frame-hdr” “-m” “elf64lppc” “-dynamic-linker” “/lib64/ld64.so.2” “-o” “a.out” “/lib/…/lib64/crt1.o” “/lib/…/lib64/crti.o” “/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtbegin.o” “-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7” “-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…/…/lib64” “-L/lib/powerpc64le-linux-gnu” “-L/lib/…/lib64” “-L/usr/lib/powerpc64le-linux-gnu” “-L/usr/lib/…/lib64” “-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…” “-L/data1/hstong.local/wybuild/build/bootstrap/stage1/build/bin/…/lib” “-L/lib” “-L/usr/lib” “/tmp/–d53b10.o” “-lstdc++” “-lm” “-lgcc_s” “-lgcc” “-lc” “-lgcc_s” “-lgcc” “/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtend.o” “/lib/…/lib64/crtn.o”

Gak. Needed -xc++.

If --gcc-toolchain is specified, its value overrides the cmake variable
GCC_INSTALL_PREFIX.
When the value is non-empty: the value is appended to the --prefix list
and is used to detect GCC installations.
The GCC installation is used to provide include directories/library
directories and some startup files (e.g. crtbegin).

Problem 1.

--prefix(-B) does more than --gcc-toolchain:
clang::driver::Driver::GetProgramPath basically searches for
$prefix/$triple-$file and $prefix$file,
where $prefix is taken from the list of --prefix(-B). --gcc-toolchain does
not participate in the search. <have attached a summary of the algorithm at
the bottom>
The result is that 'ld' and 'as' may come from the system (more precisely,
sysroot):

cd clang/test/Driver
# Make sure Inputs/opensuse_42.2_aarch64_tree/usr/bin/ld exists.
clang -target aarch64-suse-linux
--gcc-toolchain=Inputs/opensuse_42.2_aarch64_tree/usr '-###'
gcc-toolchain.cpp -v

   # I have ld in my /usr/local/bin and it takes precedence over
/usr/bin/ld
"/usr/local/bin/ld" "-EL" "--eh-frame-hdr" "-m" "aarch64linux"
"-dynamic-linker" "/lib/ld-linux-aarch64.so.1" "-o" "a.out"
"Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crt1.o"
"Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crti.o"
"Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtbegin.o"
"-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8"
"-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64"
"-L/lib/aarch64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/aarch64-linux-gnu"
"-L/usr/lib/../lib64"
"-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../.."
"-L/tmp/RelA/bin/../lib" "-L/lib" "-L/usr/lib"
"/tmp/gcc-toolchain-f87f08.o" "-lgcc" "--as-needed" "-lgcc_s"
"--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed"
"Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtend.o"
"Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crtn.o"

The -L and crt* files are indeed from
Inputs/opensuse_42.2_aarch64_tree/usr, but ld (and as if
-fno-integrated-as) is from the system.
On many Linux distributions you can normally assume that the system ld and
as only support the host architecture.
This means --gcc-toolchain can only be used to specify a GCC installation
with the same architecture.

... and specifying a GCC installation with the same architecture (e.g., to
use a newer libstdc++) is something that people do:
$ clang++ -fuse-ld=ld --gcc-toolchain=/opt/rh/devtoolset-7/root -xc -
-fno-integrated-as <<<'#include <stdio.h>'$'\n''int main(void) {
printf("Hello, world!\n"); }' && echo OKAY
OKAY
... but it does seem ld is coming from --gcc-toolchain for me anyway:
$ ./build/bootstrap/stage1/build/bin/clang++ -fuse-ld=ld
--gcc-toolchain=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as
<<<'#include <stdio.h>'$'\n''int main(void) { printf("Hello, world!\n"); }'
-### 2>&1 | grep /ld
"/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../../../bin/ld"
"--hash-style=gnu" "--eh-frame-hdr" "-m" "elf64lppc" "-dynamic-linker"
"/lib64/ld64.so.2" "-o" "a.out" "/lib/../lib64/crt1.o"
"/lib/../lib64/crti.o"
"/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtbegin.o"
"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7"
"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../../../lib64"
"-L/lib/powerpc64le-linux-gnu" "-L/lib/../lib64"
"-L/usr/lib/powerpc64le-linux-gnu" "-L/usr/lib/../lib64"
"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../.."
"-L/data1/hstong.local/wybuild/build/bootstrap/stage1/build/bin/../lib"
"-L/lib" "-L/usr/lib" "/tmp/--8f8c6a.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc"
"-lc" "-lgcc_s" "-lgcc"
"/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtend.o"
"/lib/../lib64/crtn.o"

'ld' path is computed by clang/lib/Driver/Driver.cpp Driver::GetProgramPath.
Did you set PATH to the devtoolset-7 bin directory?

If --gcc-toolchain and multiple -B are specified, the search order is:

-B -B -B --gcc-toolchain

Currently the GCC installation with the largest version wins.
Do you think we should stop search after one -B provides a GCC installation?

--prefix can make as and ld paths correct, but: if another --prefix is
needed, why do we use --gcc-toolchain?

Well, I'm not sure that another --prefix is needed. Also, --prefix doesn't
help to pick up the GCC header and library paths and --sysroot overrides
too much:
$ clang++ -fuse-ld=ld --sysroot=/opt/rh/devtoolset-7/root -xc -
-fno-integrated-as <<<'#include <stdio.h>'$'\n''int main(void) {
printf("Hello, world!\n"); }'
<stdin>:1:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
        ^~~~~~~~~
1 error generated.

(The example used --sysroot instead of --prefix)

OK, your other reply clarified that --prefix does pick up include/library directories.

If --gcc-toolchain is specified, its value overrides the cmake variable
GCC_INSTALL_PREFIX.
When the value is non-empty: the value is appended to the --prefix list
and is used to detect GCC installations.
The GCC installation is used to provide include directories/library
directories and some startup files (e.g. crtbegin).

Problem 1.

–prefix(-B) does more than --gcc-toolchain:
clang::driver::Driver::GetProgramPath basically searches for
$prefix/$triple-$file and $prefix$file,
where $prefix is taken from the list of --prefix(-B). --gcc-toolchain does
not participate in the search.
The result is that ‘ld’ and ‘as’ may come from the system (more precisely,
sysroot):

cd clang/test/Driver

Make sure Inputs/opensuse_42.2_aarch64_tree/usr/bin/ld exists.

clang -target aarch64-suse-linux
–gcc-toolchain=Inputs/opensuse_42.2_aarch64_tree/usr ‘-###’
gcc-toolchain.cpp -v

I have ld in my /usr/local/bin and it takes precedence over

/usr/bin/ld
“/usr/local/bin/ld” “-EL” “–eh-frame-hdr” “-m” “aarch64linux”
“-dynamic-linker” “/lib/ld-linux-aarch64.so.1” “-o” “a.out”
“Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crt1.o”
“Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crti.o”
“Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtbegin.o”
“-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8”
“-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64”
“-L/lib/aarch64-linux-gnu” “-L/lib/…/lib64” “-L/usr/lib/aarch64-linux-gnu”
“-L/usr/lib/…/lib64”
“-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…”
“-L/tmp/RelA/bin/…/lib” “-L/lib” “-L/usr/lib”
“/tmp/gcc-toolchain-f87f08.o” “-lgcc” “–as-needed” “-lgcc_s”
“–no-as-needed” “-lc” “-lgcc” “–as-needed” “-lgcc_s” “–no-as-needed”
“Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtend.o”
“Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/…/…/…/…/lib64/crtn.o”

The -L and crt* files are indeed from
Inputs/opensuse_42.2_aarch64_tree/usr, but ld (and as if
-fno-integrated-as) is from the system.
On many Linux distributions you can normally assume that the system ld and
as only support the host architecture.
This means --gcc-toolchain can only be used to specify a GCC installation
with the same architecture.

… and specifying a GCC installation with the same architecture (e.g., to
use a newer libstdc++) is something that people do:
$ clang++ -fuse-ld=ld --gcc-toolchain=/opt/rh/devtoolset-7/root -xc -
-fno-integrated-as <<<’#include <stdio.h>’$’\n’‘int main(void) {
printf(“Hello, world!\n”); }’ && echo OKAY
OKAY
… but it does seem ld is coming from --gcc-toolchain for me anyway:
$ ./build/bootstrap/stage1/build/bin/clang++ -fuse-ld=ld
–gcc-toolchain=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as
<<<’#include <stdio.h>’$’\n’‘int main(void) { printf(“Hello, world!\n”); }’
-### 2>&1 | grep /ld
“/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…/…/bin/ld”
“–hash-style=gnu” “–eh-frame-hdr” “-m” “elf64lppc” “-dynamic-linker”
“/lib64/ld64.so.2” “-o” “a.out” “/lib/…/lib64/crt1.o”
“/lib/…/lib64/crti.o”
“/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtbegin.o”
“-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7”
“-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…/…/lib64”
“-L/lib/powerpc64le-linux-gnu” “-L/lib/…/lib64”
“-L/usr/lib/powerpc64le-linux-gnu” “-L/usr/lib/…/lib64”
“-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/…/…/…”
“-L/data1/hstong.local/wybuild/build/bootstrap/stage1/build/bin/…/lib”
“-L/lib” “-L/usr/lib” “/tmp/–8f8c6a.o” “-lstdc++” “-lm” “-lgcc_s” “-lgcc”
“-lc” “-lgcc_s” “-lgcc”
“/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtend.o”
“/lib/…/lib64/crtn.o”

‘ld’ path is computed by clang/lib/Driver/Driver.cpp Driver::GetProgramPath.
Did you set PATH to the devtoolset-7 bin directory?

No, I did not.

If --gcc-toolchain and multiple -B are specified, the search order is:

-B -B -B --gcc-toolchain

Currently the GCC installation with the largest version wins.
Do you think we should stop search after one -B provides a GCC installation?

Well, the ordering where --gcc-toolchain follows all of the -B seems okay in the context of looking for ld or as. For changing “largest version wins”, it might make some sense to order --gcc-toolchain first for the resolution of the GCC installation (which allows “largest version wins” to continue with -B). I suppose it means people using builds preconfigured with a GCC install prefix will be somewhat inconvenienced when they don’t want to use the preconfigured path; however, I think this retains most functionality (there is a way to limit the search for GCC installations but also a way to compose search paths for the latest GCC version).

–prefix can make as and ld paths correct, but: if another --prefix is
needed, why do we use --gcc-toolchain?

Well, I’m not sure that another --prefix is needed. Also, --prefix doesn’t
help to pick up the GCC header and library paths and --sysroot overrides
too much:
$ clang++ -fuse-ld=ld --sysroot=/opt/rh/devtoolset-7/root -xc -
-fno-integrated-as <<<’#include <stdio.h>’$’\n’‘int main(void) {
printf(“Hello, world!\n”); }’
:1:10: fatal error: ‘stdio.h’ file not found
#include <stdio.h>
^~~~~~~~~
1 error generated.

(The example used --sysroot instead of --prefix)

Yes.

OK, your other reply clarified that --prefix does pick up include/library directories.

Yes.

>
>> If --gcc-toolchain is specified, its value overrides the cmake variable
>> GCC_INSTALL_PREFIX.
>> When the value is non-empty: the value is appended to the --prefix list
>> and is used to detect GCC installations.
>> The GCC installation is used to provide include directories/library
>> directories and some startup files (e.g. crtbegin).
>>
>> Problem 1.
>>
>> --prefix(-B) does more than --gcc-toolchain:
>> clang::driver::Driver::GetProgramPath basically searches for
>> $prefix/$triple-$file and $prefix$file,
>> where $prefix is taken from the list of --prefix(-B). --gcc-toolchain does
>> not participate in the search. <have attached a summary of the algorithm at
>> the bottom>
>> The result is that 'ld' and 'as' may come from the system (more precisely,
>> sysroot):
>>
>> cd clang/test/Driver
>> # Make sure Inputs/opensuse_42.2_aarch64_tree/usr/bin/ld exists.
>> clang -target aarch64-suse-linux
>> --gcc-toolchain=Inputs/opensuse_42.2_aarch64_tree/usr '-###'
>> gcc-toolchain.cpp -v
>>
>> # I have ld in my /usr/local/bin and it takes precedence over
>> /usr/bin/ld
>> "/usr/local/bin/ld" "-EL" "--eh-frame-hdr" "-m" "aarch64linux"
>> "-dynamic-linker" "/lib/ld-linux-aarch64.so.1" "-o" "a.out"
>> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crt1.o"
>> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crti.o"
>> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtbegin.o"
>> "-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8"
>> "-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64"
>> "-L/lib/aarch64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/aarch64-linux-gnu"
>> "-L/usr/lib/../lib64"
>> "-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../.."
>> "-L/tmp/RelA/bin/../lib" "-L/lib" "-L/usr/lib"
>> "/tmp/gcc-toolchain-f87f08.o" "-lgcc" "--as-needed" "-lgcc_s"
>> "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed"
>> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtend.o"
>> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crtn.o"
>>
>> The -L and crt* files are indeed from
>> Inputs/opensuse_42.2_aarch64_tree/usr, but ld (and as if
>> -fno-integrated-as) is from the system.
>> On many Linux distributions you can normally assume that the system ld and
>> as only support the host architecture.
>> This means --gcc-toolchain can only be used to specify a GCC installation
>> with the same architecture.
>>
>... and specifying a GCC installation with the same architecture (e.g., to
>use a newer libstdc++) is something that people do:
>$ clang++ -fuse-ld=ld --gcc-toolchain=/opt/rh/devtoolset-7/root -xc -
>-fno-integrated-as <<<'#include <stdio.h>'$'\n''int main(void) {
>printf("Hello, world!\n"); }' && echo OKAY
>OKAY
>... but it does seem ld is coming from --gcc-toolchain for me anyway:
>$ ./build/bootstrap/stage1/build/bin/clang++ -fuse-ld=ld
>--gcc-toolchain=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as
><<<'#include <stdio.h>'$'\n''int main(void) { printf("Hello, world!\n"); }'
>-### 2>&1 | grep /ld
> "/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../../../bin/ld"
>"--hash-style=gnu" "--eh-frame-hdr" "-m" "elf64lppc" "-dynamic-linker"
>"/lib64/ld64.so.2" "-o" "a.out" "/lib/../lib64/crt1.o"
>"/lib/../lib64/crti.o"
>"/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtbegin.o"
>"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7"
>"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../../../lib64"
>"-L/lib/powerpc64le-linux-gnu" "-L/lib/../lib64"
>"-L/usr/lib/powerpc64le-linux-gnu" "-L/usr/lib/../lib64"
>"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../.."
>"-L/data1/hstong.local/wybuild/build/bootstrap/stage1/build/bin/../lib"
>"-L/lib" "-L/usr/lib" "/tmp/--8f8c6a.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc"
>"-lc" "-lgcc_s" "-lgcc"
>"/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtend.o"
>"/lib/../lib64/crtn.o"

'ld' path is computed by clang/lib/Driver/Driver.cpp Driver::GetProgramPath.
Did you set PATH to the devtoolset-7 bin directory?

No, I did not.

Do you mind debugging clang '-###' a bit: set a breakpoint on
clang/lib/Driver/Driver.cpp `GetProgramPath` and report how the 'ld'
under --gcc-toolchain is selected?

If --gcc-toolchain and multiple -B are specified, the search order is:

-B -B -B --gcc-toolchain

Currently the GCC installation with the largest version wins.
Do you think we should stop search after one -B provides a GCC installation?

Well, the ordering where --gcc-toolchain follows all of the -B seems okay in the context of looking for ld or as. For changing "largest version wins", it might make some sense to order --gcc-toolchain first for the resolution of the GCC installation (which allows "largest version wins" to continue with -B). I suppose it means people using builds preconfigured with a GCC install prefix will be somewhat inconvenienced when they don't want to use the preconfigured path; however, I think this retains most functionality (there is a way to limit the search for GCC installations but also a way to compose search paths for the latest GCC version).

If cmake GCC_INSTALL_PREFIX was specified configuring llvm-project, I
think it makes sense that explicit options (-B, --gcc-toolchain) take
precedence over GCC_INSTALL_PREFIX.
When --gcc-toolchain is specified: yes, probably it should take
precedence over --prefix. I wonder whether it should suppress --prefix
for GCC installation.

I played with gcc -B a bit: looks like it picks the first --prefix
where files can be found. So I am going to send a patch making the
"largest version wins" local to one -B (i.e. if multiple versions are
detected, pick the largest).

In the case of Chrome OS, we currently are using all 3 options:
“-B”, “–prefix” and “–gcc-toolchain”. “–gcc-toolchain” is only used when the clang binary is not inside /usr/bin e.g. installed as /opt/llvm/bin/clang.

I think the reason is the layout of gcc and binutils for cross compilation. Example for aarch64:

binutils installation: /usr/libexec/gcc/aarch64-cros-linux-gnu → This is path for “–prefix” (–prefix=’/usr/libexec/gcc/aarch64-cros-linux-gnu/aarch64-cros-linux-gnu-’) and “-B” (-B’/usr/libexec/gcc/aarch64-cros-linux-gnu/’)
gcc binaries: /usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/gcc-bin/10.2.0/
gcc libraries: /usr/lib/gcc/aarch64-cros-linux-gnu/10.2.0/
libc headers + libs: /usr/aarch64-cros-linux-gnu/usr/{include,lib64}, /usr/aarch64-cros-linux-gnu/lib64
sysroot: /build/device : Also has libc headers + libraries when building for the device, substitutes libc files from /usr/aarch64-cros-linux-gnu (–sysroot=/build/device).

The final command line looks like:
/opt/llvm/bin/clang --prefix=’/usr/libexec/gcc/aarch64-cros-linux-gnu/aarch64-cros-linux-gnu-’ -B’/usr/libexec/gcc/aarch64-cros-linux-gnu --gcc-toolchain=’/usr’ --sysroot=/build/device .

My understanding is without passing “–gcc-toolchain=/usr”, clang (when not installed in /usr/bin) is not able to find libgcc from /usr/lib/gcc/aarch64-cros-linux-gnu/10.2.0/. i.e. -B/–prefix options were insufficient for finding gcc library paths.

Thanks,
Manoj

In the case of Chrome OS, we currently are using all 3 options:
"-B", "--prefix" and "--gcc-toolchain". "--gcc-toolchain" is only used when the clang binary is not inside /usr/bin e.g. installed as /opt/llvm/bin/clang.

I think the reason is the layout of gcc and binutils for cross compilation. Example for aarch64:

binutils installation: /usr/libexec/gcc/aarch64-cros-linux-gnu -> This is path for "--prefix" (--prefix='/usr/libexec/gcc/aarch64-cros-linux-gnu/aarch64-cros-linux-gnu-') and "-B" (-B'/usr/libexec/gcc/aarch64-cros-linux-gnu/')
gcc binaries: /usr/x86_64-pc-linux-gnu/aarch64-cros-linux-gnu/gcc-bin/10.2.0/
gcc libraries: /usr/lib/gcc/aarch64-cros-linux-gnu/10.2.0/
libc headers + libs: /usr/aarch64-cros-linux-gnu/usr/{include,lib64}, /usr/aarch64-cros-linux-gnu/lib64
sysroot: /build/device : Also has libc headers + libraries when building for the device, substitutes libc files from /usr/aarch64-cros-linux-gnu (--sysroot=/build/device).

The final command line looks like:
/opt/llvm/bin/clang --prefix='/usr/libexec/gcc/aarch64-cros-linux-gnu/aarch64-cros-linux-gnu-' -B'/usr/libexec/gcc/aarch64-cros-linux-gnu --gcc-toolchain='/usr' --sysroot=/build/device .

My understanding is without passing "--gcc-toolchain=/usr", clang (when not installed in /usr/bin) is not able to find libgcc from /usr/lib/gcc/aarch64-cros-linux-gnu/10.2.0/. i.e. -B/--prefix options were insufficient for finding gcc library paths.

Thanks,
Manoj

--prefix is an alias for -B. I've added some documentation for
existing behaviors in ⚙ D97902 [docs] Improve documentation of -B and --gcc-toolchain

For ChromeOS, I think -B/usr/libexec/gcc/aarch64-cros-linux-gnu may
not be needed. Having -B...aarch64-cros-linux-gnu-
(--prefix=.../aarch64-cros-linux-gnu-) is sufficient.

In Clang, we detect GCC installations under -B.
However, in GCC, -B is not used to detect installations. Executable
files and startup files are found via $prefix$file. $prefix/include is
added as an include directory.

--gcc-toolchain currently suppresses GCC detection under sysroot. It
probably makes sense to suppress detection under -B as well.
Unfortunately, some platforms may have relied on the GCC detection
behavior with -B (perhaps Android).

>
>>
>> >
>> >> If --gcc-toolchain is specified, its value overrides the cmake variable
>> >> GCC_INSTALL_PREFIX.
>> >> When the value is non-empty: the value is appended to the --prefix list
>> >> and is used to detect GCC installations.
>> >> The GCC installation is used to provide include directories/library
>> >> directories and some startup files (e.g. crtbegin).
>> >>
>> >> Problem 1.
>> >>
>> >> --prefix(-B) does more than --gcc-toolchain:
>> >> clang::driver::Driver::GetProgramPath basically searches for
>> >> $prefix/$triple-$file and $prefix$file,
>> >> where $prefix is taken from the list of --prefix(-B). --gcc-toolchain does
>> >> not participate in the search. <have attached a summary of the algorithm at
>> >> the bottom>
>> >> The result is that 'ld' and 'as' may come from the system (more precisely,
>> >> sysroot):
>> >>
>> >> cd clang/test/Driver
>> >> # Make sure Inputs/opensuse_42.2_aarch64_tree/usr/bin/ld exists.
>> >> clang -target aarch64-suse-linux
>> >> --gcc-toolchain=Inputs/opensuse_42.2_aarch64_tree/usr '-###'
>> >> gcc-toolchain.cpp -v
>> >>
>> >> # I have ld in my /usr/local/bin and it takes precedence over
>> >> /usr/bin/ld
>> >> "/usr/local/bin/ld" "-EL" "--eh-frame-hdr" "-m" "aarch64linux"
>> >> "-dynamic-linker" "/lib/ld-linux-aarch64.so.1" "-o" "a.out"
>> >> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crt1.o"
>> >> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crti.o"
>> >> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtbegin.o"
>> >> "-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8"
>> >> "-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64"
>> >> "-L/lib/aarch64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/aarch64-linux-gnu"
>> >> "-L/usr/lib/../lib64"
>> >> "-LInputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../.."
>> >> "-L/tmp/RelA/bin/../lib" "-L/lib" "-L/usr/lib"
>> >> "/tmp/gcc-toolchain-f87f08.o" "-lgcc" "--as-needed" "-lgcc_s"
>> >> "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed"
>> >> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/crtend.o"
>> >> "Inputs/opensuse_42.2_aarch64_tree/usr/lib64/gcc/aarch64-suse-linux/4.8/../../../../lib64/crtn.o"
>> >>
>> >> The -L and crt* files are indeed from
>> >> Inputs/opensuse_42.2_aarch64_tree/usr, but ld (and as if
>> >> -fno-integrated-as) is from the system.
>> >> On many Linux distributions you can normally assume that the system ld and
>> >> as only support the host architecture.
>> >> This means --gcc-toolchain can only be used to specify a GCC installation
>> >> with the same architecture.
>> >>
>> >... and specifying a GCC installation with the same architecture (e.g., to
>> >use a newer libstdc++) is something that people do:
>> >$ clang++ -fuse-ld=ld --gcc-toolchain=/opt/rh/devtoolset-7/root -xc -
>> >-fno-integrated-as <<<'#include <stdio.h>'$'\n''int main(void) {
>> >printf("Hello, world!\n"); }' && echo OKAY
>> >OKAY
>> >... but it does seem ld is coming from --gcc-toolchain for me anyway:
>> >$ ./build/bootstrap/stage1/build/bin/clang++ -fuse-ld=ld
>> >--gcc-toolchain=/opt/rh/devtoolset-7/root -xc - -fno-integrated-as
>> ><<<'#include <stdio.h>'$'\n''int main(void) { printf("Hello, world!\n"); }'
>> >-### 2>&1 | grep /ld
>> > "/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../../../bin/ld"
>> >"--hash-style=gnu" "--eh-frame-hdr" "-m" "elf64lppc" "-dynamic-linker"
>> >"/lib64/ld64.so.2" "-o" "a.out" "/lib/../lib64/crt1.o"
>> >"/lib/../lib64/crti.o"
>> >"/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtbegin.o"
>> >"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7"
>> >"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../../../lib64"
>> >"-L/lib/powerpc64le-linux-gnu" "-L/lib/../lib64"
>> >"-L/usr/lib/powerpc64le-linux-gnu" "-L/usr/lib/../lib64"
>> >"-L/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/../../.."
>> >"-L/data1/hstong.local/wybuild/build/bootstrap/stage1/build/bin/../lib"
>> >"-L/lib" "-L/usr/lib" "/tmp/--8f8c6a.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc"
>> >"-lc" "-lgcc_s" "-lgcc"
>> >"/opt/rh/devtoolset-7/root/lib/gcc/ppc64le-redhat-linux/7/crtend.o"
>> >"/lib/../lib64/crtn.o"
>>
>> 'ld' path is computed by clang/lib/Driver/Driver.cpp Driver::GetProgramPath.
>> Did you set PATH to the devtoolset-7 bin directory?
>
> No, I did not.

Do you mind debugging clang '-###' a bit: set a breakpoint on
clang/lib/Driver/Driver.cpp `GetProgramPath` and report how the 'ld'
under --gcc-toolchain is selected?

OK, I have found the answer. ⚙ D34848 Driver: Don't mix system tools with devtoolset tools on RHEL
opt/rh/devtoolset is special cased...