Android target triples contain the target API level as part of the environment, e.g. aarch64-none-linux-android21. This can cause issues with the per-target runtime directory layout, because Clang only searches for the exact target triple you’re building with right now. For example, if I build compiler-rt with the triple aarch64-none-linux-android21 and the per-target runtime directory layout, and then code using my toolchain wants to use a target triple with a newer API level, Clang will fail to find my runtime libraries because of the target triple mismatch, even though the libraries in my toolchain will work perfectly fine for the target triple being used for building.
I’m currently working around this by just not using the per-target runtime directory layout, but that has its own issues; in particular, you might want to legitimately ship multiple versions of the runtimes for different API levels (e.g. API level 29 adds ELF TLS, which can be used by some runtimes). I can think of two potential ways to make this work better:
Add a mechanism to the runtimes build to produce symlinked target directories, e.g. you build for aarch64-none-linux-android21 but then you symlink that to aarch64-none-linux-android22, aarch64-none-linux-android23, etc. I’m not a big fan of this because (a) it clutters up your build directory, (b) you’ll have to add a new symlink for each new version you want to support (or just add a ton of versions preemptively, but that makes (a) even worse), and (c) symlinks aren’t supported in all environments, e.g. Windows has historically had issues (I know things are a lot better there now).
Teach Clang about version numbers in target triples when it’s searching for its runtime directories. E.g. if the target triple is aarch64-none-linux-android23 and there’s no directory for that target, search for aarch64-none-linux-android22, aarch64-none-linux-android21, etc. until you find a directory or exhaust all options. (I’m not sure whether it’d make sense to also search for the target without any version.) This seems cleaner from a build perspective, but there’s the cost in Clang of the extra directory searches, plus the added complexity in the driver code (not a lot IMO, but still something).
I’d appreciate feedback on what people prefer (and of course, if there’s other target triples which commonly have version numbers in them, it’d be great to come up with something that works across the board). CCing @petrhosek since you’ve done all the work for the runtimes build and the Clang driver changes for it, @MaskRay since I’ve seen some diffs from you lately related to Clang’s runtime directory search, and @srhines since I’m thinking about this in the context of Android (I’d also CC Dan Albert and Ryan Prichard, but I don’t think they’re on this Discourse).
I don’t know if this would help your use case, but one thing I proposed on a different topic was to add a --runtime-target option that would allow you to specify the specific runtime target triple to use. This would be another way to get the correct runtime directory without add a lot of directory search overhead to the driver. You could also add a CMake option to set the default if you wanted to avoid passing the option for each compiler invocation.
If that doesn’t work, then I agree with @petrhosek that the second option is better.
Collin Baker’s ⚙ D115049 Fall back on Android triple w/o API level for runtimes search made the driver fall back to the lib without a version number, for the reasons you describe. If you want to do per-version fallback (ie try 23, 22, 21, …) before the version-less name, that’d be fine, I suppose.
In ⚙ D121761 [draft] runtimes doc wip I started to document the runtimes build a bit, but didn’t get terribly far. It touches on this topic a bit. (In case you’d like to take that over and write some docs…)
Thanks everyone for the thoughts and the links; they’re super helpful as a reference I’ll take a shot at implementing the version search then.
This is a good idea for a lot of cases, but since there’s multiple targets at play here (e.g. different Android architectures), we’d need to either encode a bunch of info about the toolchain into the builds using that toolchain, or add some logic to the toolchain to select an appropriate default --runtime-target based on the --target, and at that point just doing the version search is likely simpler.
I’ll agree with @petrhosek that I’d prefer option 2. In addition to the extra directory visual complexity/bloat from option 1, the NDK doesn’t ship symlinks for Windows releases, so we’d have N directories/copies of all these runtime libraries installed on dev machines, which seems less than ideal. I think the backwards target level search is something that makes it more obvious what is actually happening anyways (from an abstraction POV). Thanks for taking the time to work on this, and for consulting with other knowledgeable folks.
IMHO Clang driver can recognize the version number in android23 and use that to provide driver option defaults, but I’d hope that clang --target=aarch64-none-linux-android23 does not fall back to directories named aarch64-none-linux-android22. At least for non-Android targets, I think such a behavior would be surprising to users.
So I guess I’ll favor an explicit option like --runtime-target.
Could you elaborate on what you find confusing? The idea I had is that libraries for lower API levels should be compatible, so we should use the runtime libraries for the newest API level we can find instead that’s <= the requested API level of failing entirely. That’s for Android specifically though, whereas you said non-Android, so I’m curious what other platforms you were thinking about.
--runtime-target would be painful here because you’d end up hardcoding a bunch of information about your toolchain in the build system and have to keep the two in sync. E.g. today I ship a toolchain with the runtime libraries built for API 21, and the users of the toolchain can use any newer API level. In the future maybe I add runtime libraries built for API 29 (to take advantage of ELF TLS) as well, and now I want any clients building for API 29 or newer to transparently use the API 29 libraries, and any clients building for older API levels to continue using the API 21 libraries. I don’t see a good way to accomplish that with --runtime-target (good as in doesn’t involve a bunch of hard-coding).
libandroid.so [exposes only the APIs present in KitKat]
libc.so [exposes only the APIs present in KitKat]
…
The directory is just named 19, not arm-linux-androideabi19.
There is a separate API level directory for every currently supported API between 19 and 32, except for 20 and 25. The NDK’s build system code remaps those API levels: 20->19, 25->24, but Clang can’t find the sysroot libraries if I target API 20 or 25.
The NDK deduplicates some of the per-API-level libraries using linker scripts. e.g. The libc++.so and libc++.a files are both linker scripts. For API 19, they are:
No fundamental objection, just my general concern of having more magic in the runtime library search rules. My viewpoint is that for FreeBSD, and for an imaginary Linux distribution which encodes version into the target triple, we can expect that they don’t want mix-and-match: if I specify --target to target version X, it does not necessarily mean I expect a silent fallback to a runtime library of version X-1.
In clang/test/Driver/Inputs/basic_android_ndk_tree (rprichard’s “FWIW: the Android API level is also used to lookup sysroot libraries”), sysroot/usr/lib/aarch64-linux-android/21/ is in the library search paths. For consistency, you’d need adding more logic there.
I did want to recommend “Add a mechanism to the runtimes build to produce symlinked target directories”, but I see I would be the lone one here. I myself have concern with having more CMake complexity, so I can see that this choice won’t make people so happy.
if I specify --target to target version X, it does not necessarily mean I expect a silent fallback to a runtime library of version X-1.
FWIW this is exactly the behavior we want in Chromium: We build a single clang package (including clang’s runtime libs) at some NDK version N and we want to be able to use that package to build Chromium binaries for systems with version >= N (where the static build-time version of the triple of the built Chromium is >= N).
How do you build the runtimes directory without the API level in it? IIRC in the CMake runtimes build, the directory name is the same as the target triple you’re building with, so to create a runtimes directory for an Android target without the API level in it, you’d also need to perform the build with a target triple without the API level in it, which seems undesirable (and is the primary reason I’m interested in the previous version searching behavior).
Another idea I had after re-reading this thread: Could we put some kind of file in the runtime directory that the clang driver can look at to determine if the runtime is compatible with the triple? Using the directory path name to communicate whether or not a runtime is compatible with a triple just seems very fragile and inflexible.
Hmm, interesting … did you have ideas on how this could work for e.g. the Android versioned triples? I agree that raw directory names aren’t ideal, so a better alternative would be interesting.
I was thinking something similar to the current multi-arch search rountines (e.g. for Android ARM) that check for the presence of certain files in the candidate directory.
So for android you could install a file called abi_version or something that had information in it that could be used by the driver to auto-detect if a runtime directory was compatible with the current compiler.
I’ll take a shot at implementing the version search then.