[RFC] Multilib

Clang Frontend
1

Happy new year folks!

I’d love to get your views on a new approach to multilib. This is a follow up to LLVM Embedded Toolchains Working Group sync up - #19 by voltur01

The problem: for embedded development there are a lot of potential targets to build for, and a toolchain provider should provide a library for each one. It’s typically not obvious to a user of the toolchain which library they should use. A simplified example is if a user wants to build for Cortex-M4 then it may not be obvious whether they should use an armv7m or armv7em library.

The solution is called multilib. The user specifies -mcpu=cortex-m4 and the multilib system chooses an appropriate library.

Clang already has Multilib and MultilibSet. However that system is too restrictive to be used in the development of LLVM Embedded Toolchain for Arm because:
a) It appears to be designed for the set of available library variants to be hard-coded in clang.
b) It can’t express backwards compatibility.

An example of backwards compatibility:
If I’m building my application with -mfloat-abi=softfp then I can use a library built with either -mfloat-abi=softfp or -mfloat-abi=soft. The ABI is the same in both cases but the former requires an FPU. softfp is backwards compatible with soft.
However if I’m building my application with -mfloat-abi=soft then a multilib system must assume that an FPU isn’t available and must only link with a library that was also built with -mfloat-abi=soft.
This directional compatibility can’t be expressed with Clang’s existing Multilib class.

A solution that was previously proposed was to introduce a DSL and this RFC explores that idea in detail. The idea is to introduce a new file that Clang ToolChains can recognise and parse each time clang runs: multilib.yaml.

multilib.yaml expresses both which libraries are available and how to establish their compatibility.

The compatibility model starts with the idea that each library variant requires a set of attributes e.g. armv7m support, hardware floating point support. We can infer the set of available attributes from the command line arguments provided to clang e.g. -mfloat-abi=hard implies hardware floating point support. As long as the set of attributes on a library is a subset of the attributes inferred from the command line arguments then the library is deemed compatible. If multiple libraries are compatible then the last in the list wins. However the file format could be extended in future to facilitate more sophisticated scoring for better optimisation.

The compatibility detection relies on string matching against command line arguments. However --target=arm-none-eabi -march=armv7em -mfloat-abi=hard and --target=thumbv7em-none-eabihf are equivalent so the arguments must be normalised. Clang already normalises arguments before passing them to clang -cc1 so that functionality is reused.

The proposed format is explained in more detail in the prototype multilib.yaml: ⚙ Changeset View

The C++ code is not intended to be production quality but I’m very interested to get thoughts on the interface it exposes to the user.

2

This is way out of my area of expertise. I just want to note that AWS is not the first one to note that it is hard to write YAML files by hand. They offer their customers the CDK. It is a tool for writing your YAML files, e.g., in Python: What is the AWS CDK? - AWS Cloud Development Kit (AWS CDK) v2

3

While the existing Multilib and MultilibSet implementation doesn’t provide all the functionality you need, there’s a significant overlap. Rather than introducing a second parallel implementation, would it be possible to improve and extend the existing one to support all the functionality you need (updating the existing usages as needed)? The ability to serialize and deserialize the configuration to and from a YAML file could then be added as an additional functionality (akin to how this is done in VFS for example).

Regarding the backwards compatibility, an aspect that is important for our use cases is a support for fallback. To give you a concrete example, in our toolchain we include multilib variants of C++ libraries compiled with -fno-exceptions. This is a pure optimization, if the -fno-exceptions variant of the library exists, compiler should use it, but if it doesn’t exist, it’ll fallback to the base variant. Practically, this means that instead of using -L<base>/noexcept, driver would use -L<base>/noexcept -L<base>, so we aren’t completely swapping out the sysroot, instead we’re layering search directories from the more specialized to the more generic. It’d be great if this fallback behavior could be expressed using the new configuration mechanism.

4

Hi Petr,

I agree there is significant overlap with MultilibSet and yes I could replace that instead of having the two implementations exist side by side. This would be a breaking change to MultilibSet rather than extending it and indeed existing usages would need updating.

For LLVM Embedded Toolchain for Arm we’ll likely have a similar requirement for -fno-exceptions library variants. How I anticipated we would do this:

variants:
- path: yes/exceptions
  args: [...]
  attrs: [...]
- path: no/exceptions
  args: [-fno-exceptions, ...]
  attrs: [no-exceptions, ...]
arguments:
- regex: -fno-exceptions
  matchAttrs: [no-exceptions]

If the user doesn’t specify -fno-exceptions then only yes/exceptions will match.
If the user does specify -fno-exceptions then both yes/exceptions and no/exceptions match. The rule is that last wins so no/exceptions is picked.

(The “last wins” rule works fine as long as long as you can order your optimisations along one dimension. I anticipate this won’t always be the case so in future the system could be extended with some way to score optional attributes.)

The way I’ve been using the new implementation is to change the sysroot. However I think you’re suggesting that all library variants will share a single include directory and have multiple lib directories. I vaguely remember being told that this was discussed and some people have the requirement that each variant has its own include directory since header files may be different between variants. However I think the multilib selection code could be used in different ways for different ToolChain classes. Instead of the multilib code selecting a particular variant, it could provide a list of all matching variants. One ToolChain class (e.g. BareMetal) could use the last matching variant as a --sysroot argument while another ToolChain (e.g. Fuchsia) could use all the variants as -L arguments. How does that sound?

5

One ToolChain class (e.g. BareMetal) could use the last matching variant as a --sysroot argument while another ToolChain (e.g. Fuchsia) could use all the variants as -L arguments. How does that sound?

To summarise our discussion earlier in the LLVM Embedded Toolchains Working Group sync up, I will not do this. Instead I will update the prototype to not use --sysroot but instead use -isystem and -L for all matching variants. This achieves the layering effect you want and it should work regardless of whether each variant has its own include directory or if they share a common one, or something in between.

6

I’ve created a new stack of changes taking into account the feedback I received on ⚙ D140959 RFC: Multilib prototype. Unlike that change which was strictly a prototype, the new changes should be suitable for detailed review and hopefully approval.

The patches I’m most interested to get feedback on are Multilib YAML parsing and Add -print-multi-selection-flags argument because they define a new stable APIs for Clang, so mistakes will be costly to fix later.

I will update the prototype to not use --sysroot but instead use -isystem and -L for all matching variants.

I haven’t done this but I am working on adding further patches to the stack to achieve the layering effect in the BareMetal ToolChain.

1 Like
7

I am working on adding further patches to the stack to achieve the layering effect in the BareMetal ToolChain

… and the patches are ready for review:

8

I put a design doc up for review at ⚙ D143587 [Docs] Multilib design. Hopefully that makes this whole thing more accessible to people who aren’t already familiar with the design space.

I’ll be on holiday next week but I’m hoping to start getting the patches landed the week after that unless a major issue is identified. The changes have had a good number of eyes on them so it feels like the time is approaching to understand how well it works in practice, and use those insights to update the design.

9

Thanks to @petrhosek, & @MaskRay for your reviews. One patch landed, 7 to go! I know that reviewing this code is quite time consuming so I don’t want to take it for granted. I think I’ve addressed all the comments on the reviews so how do you want to proceed? Do you want me to wait for you to accept the changes or are you satisfied with the changes at a high level and happy to leave it to someone else to give a final OK for the finer details?
The LLVM Embedded Toolchains Working Group sync up is this Thursday so we can discuss then if not before.

10

First of all, thank you for all the effort you put into this. I think the changes to the driver implementation are ready to go in, especially since those changes should be largely invisible to users. I’d like to discuss the new multilib.yaml configuration format since that part is visible to users.

In comparison to GCC, the proposed configuration format offers more flexibility. In the case of GCC, the supported multilibs are configured only at build time. Having runtime configurability for multilibs is a very nice idea, but we should be mindful of the unnecessary complexity.

In the proposed design, you map flags to labels using regex-based matching (the FlagMap section) and then define variants as a combination of labels (the Variants section). That’s fundamentally consistent with the GCC implementation which maps flag combinations to a chosen set of variants, but uses option parsing instead.

I think the naming could be clearer. In particular the term “flags” is used both in the Variants and FlagMap sections, but these are not actually flags, rather they’re arbitrary strings that are used for matching input flags to variants. Perhaps “label” or “tag” would be a better name?

I am concerned about regular expressions as the default mechanism for matching flags since regular expressions, while powerful, also tend to be error prone.

To use a concrete example from the review, to match the target Armv8.5 or newer, you’d have to do the following in GCC:

MULTILIB_MATCHES += target=armv8.5-none-eabi=target=armv8.6-none-eabi
MULTILIB_MATCHES += target=armv8.5-none-eabi=target=armv8.7-none-eabi
MULTILIB_MATCHES += target=armv8.5-none-eabi=target=armv8.8-none-eabi
...

In the proposed design, you could instead do:

- Regex: target=armv8\.([6-9]|[1-9][0-9]+)-none-eabi
  MatchFlags: [target=armv8.5-none-eabi]

This is an example where the higher expressive power of regular expressions helps with complex rules.

The downside is that regular expressions require escaping which makes simple cases more difficult. For example, if I wanted to match the -fc++-abi=itanium flag, I have to use:

- Regex: fc\+\+-abi=itanium
  MatchFlags: [itanium-abi]

This may be non-obvious and I’m worried that this will lead to unexpected and difficult to debug issues. We’ve already seen similar issues with Sanitizer special case list files which also use regular expressions (they also treat * as .* straddling the line between glob patterns and regular expressions).

Perhaps starting with a simpler, even if more verbose, version resembling the GCC implementation would be better? If it turns out to be a problem in practice, we could introduce the support for regular expressions (or glob patterns) later.

The other aspect I’m concerned about is defining variants, specifically around composition. When defining variants, you have to specify a directory name, a set of “flags” and the output of -print-multi-lib which is very flexible but also potentially error prone and might result in a lot of duplication.

In the driver, this is handled by the MultilibBuilder and MultilibSetBuilder classes which aid in constructing hierarchies of variants that we use in drivers like Gnu. To give a concrete example:

auto ArchV7A = MultilibBuilder("/armv7-a").flag("+march=armv7-a");
auto ArchV7M = MultilibBuilder("/armv7-m").flag("+march=armv7-m");
auto Hard = MultilibBuilder("/hard").flag("+mfloat-abi=hard");
auto SoftFp = MultilibBuilder("/soft").flag("+mfloat-abi=softfp");
auto Soft = MultilibBuilder("").flag("+mfloat-abi=soft");

MultilibSet Multilibs =
    .Either(ArchV7A, ArchV7M)
    .Either(Hard, SoftFp, Soft)
    .makeMultilibSet();

This code would generate the following variants:

armv7-a/hard;@march=armv7-a@mfloat-abi=hard
armv7-a/soft;@march=armv7-a@mfloat-abi=softfp
armv7-a;@march=armv7-a@mfloat-abi=soft
armv7-m/hard;@march=armv7-m@mfloat-abi=hard
armv7-m/soft;@march=armv7-m@mfloat-abi=softfp
armv7-m;@march=armv7-m@mfloat-abi=soft

This is quite powerful and becomes increasingly more important as the number of potential combinations grows. I’m wondering if we could make the configuration file more closely resemble the usage of those classes and allow automatic composition rather than requiring users to specify all potential combinations manually?

To make my suggestions more concrete, here’s an idea for an alternative format:

Options:
  '':
    - mfloat-abi=soft
  '/armv7-a':
    - march=armv7-a
  '/armv7-m':
    - march=armv7-m
  '/hard':
    - mfloat-abi=hard
  '/soft':
    - mfloat-abi=softfp

Variants:
  Either: ['/armv7-a', '/armv7-m']
  Either: ['', '/soft', '/hard']

To handle different spelling of flags, we could do something like:

Options:
  '/armv8.5':
    - target=armv8.5-none-eabi: # used for -print-multi-lib
    - target=armv8.[6-7]-none-eabi # other matches
  '/thumbv7m':
    - target=thumbv7m-none-eabi:
    - target=thumbv[7-9]*
    - mfpu=fpv4-sp-d16
  '/thumbv6m':
    - target=thumbv6m-none-eabi:
    - target=thumbv6m-*

I’m fine landing the proposed format as is (and marking it as experimental) to get more practical experience, but I also want to make sure we explore other alternatives since this is a pretty significant new feature that will likely get adopted by a lot of users, especially in the embedded space. I would also be interested in hearing from others.

11

Thanks very much for your careful thoughts.

  1. Yes I’m happy to change the naming. “Flags” comes from the existing multilib system but if it no longer fits then we should change it.
    I will create a new patch to switch to “Tags”.
    (However for the remainder of this message I’ll stick with the “Flags” terminology just to make it easier to refer to the existing proposal).

  2. I’ve realised that the example in ⚙ D143587 [Docs] Multilib design misses out that you can do Flags: fc++abi=itanium in the Variants list, you don’t need to use FlagMap. I apologise for not making this clearer and I’ve updated the patch to make it more apparent.
    FlagMap is specifically intended for more complex cases that you want either negation or to map many possible Clang-generated flags onto another flag. Given that specific intention, I think the additional power of a regular expression is appropriate. Nevertheless I’m open to switching to the more verbose option, particularly if we start to see problems like those in sanitizer files.

  3. I think we can approach composition from a number of directions. For LLVM Embedded Toolchain for Arm my intention has been to generate both the multilib.yaml and the libraries themselves from within its build system. But I can also see the appeal of expressing how the combinations are generated within the YAML itself. Since YAML is so flexible I think we can explore many ideas side by side just by using different keys for different options, and narrow down the options before removing the “experimental” status.

  4. What do you say to landing the patches on Friday, unless we hear differently from others? I’ve updated the docs patch in various places to make the “experimental” more obvious.

12

I will create a new patch to switch to “Tags”.

Done: ⚙ D145567 [Driver] Rename multilib flags to tags

13

I’ve gone through the patches and set approved from the Arm side with the consideration that we will iterate on these in tree, including the experimental interface for the DSL.

I’ve asked that we give some time for any further comments or objections from the US time zone. If you do plan to make any more comments, or just need some more time to review please let us know? Either here or on one or more of the patches.

Thanks for the suggestion about an alternative syntax. It looks like that could work for a lot of cases. Where I’m less sure is where there needs to be a multilib variant based on some CPU feature that is inferred by one or more +feature1+feature2 . In the current DSL we process the flags into tags to make that simpler. This still could be done for your suggestion, although it may need some additional complexity to make it work.

14

Does this mean that you can do the following?

Variants:
  - Dir: itanium
    Tags: [fc++abi=itanium]
    PrintOptions: [-fc++abi=itanium]

If that’s the case then I think the original name “flags” is a better fit than “tags”. I apologize about going back on forth on this. It might be also worth including this example in the documentation.

15

I’d like to take another look but this week is going to be especially busy for me so I might need a bit more time if that’s possible.

16

Thanks Petr for adding your reviews quickly. I’ve responded to all of them except the ones on D142933 which I’ll get to tomorrow.

Yes you can do this:

Variants:
  - Dir: itanium
    Flags: [fc++abi=itanium]
    PrintOptions: [-fc++abi=itanium]

I’ve renamed back to Flags which I agree works better.

17

Thanks, I’ll try to do another pass over your changes tomorrow.

Do you think it’d be possible to derive PrintOptions automatically (at least for the simple cases such as this one) to avoid the duplication?

I’ve also noticed that a lot of the complexity in the new implementation is due to -print-multi-lib support (such as needing the PrintOptions field) which makes me wonder whether there’s a reason why this is needed? Is the goal to support existing tools that consume the output of -print-multi-lib? If not, could we provide a different option with a simpler output that doesn’t require all this complexity?

18

It’s tempting but I think the cost will be an extra wrinkle in the API. In practice I expect most multilib.yaml files to be generated rather than hand-written, in which case the duplication is less of an issue and keeping the API simpler will be preferable. This prediction may prove incorrect but the best test of that is getting people more people actively using the system. Therefore this question is one I’d like to explore after landing the current stack of patches.

Yes the picolibc build system does interrogate -print-multi-lib output. Also I feel we need to continue to provide -print-multi-lib functionality because it’s an API that clang shares with gcc.
I share your desire to keep things simple and fast. My latest update to ⚙ D142905 [Driver] Change multilib selection algorithm improved speed at the expense of increasing complexity slightly so I’ll take another look to see if we can have speed without compromising on simplicity.

19

I think we have to pick our poison:

  1. Have the callback function to enable lazily calculating the print options.
  2. Add plumbing to many functions so we can pass a NeedPrintOptions parameter to MultilibBuilder.makeMultilib() to tell it whether or not to calculate the print options at that time.
  3. Eat the 2 microsecond cost of calculating the print options every time you invoke clang.

Right now we have option 1 and that particular poison doesn’t taste too bad to me.

I’ve now responded to your comments on D142933. Over to you!

20

Sorry I’m late to the thread but AFAIU, Multilib::flag() can express this specific case already:

auto SoftFP = MultilibBuilder(“/soft”,“”,“”,0)
.flag(“+mfloat-abi=softfp”)
.flag(“+mfloat-abi=soft”);
auto HardFP = MultilibBuilder(“/hard”,“”,“”,1)
.flag(“+mfloat-abi=hard”)
.flag(“+mfloat-abi=softfp”)
.flag(“-mfloat-abi=soft”);
MultilibSet Multilibs = MultilibSet().Either(SoftFp, HardFp);

If you build with -mfloat-abi=hard, you should get the /hard multilib.
If you build with -mfloat-abi=softfp, both are valid and the /hard multilib is selected because it has higher priority.
If you build with -mfloat-abi=soft, you should get the /soft multilib.