Shipping custom libc++ on MacOS

Runtimes C++
1

Hi Ken,

I’m in the same situation as you in conda package manager.
Note that with libcxx 12, https://reviews.llvm.org/D91517 was merged where
codecvt with char8_t is added in the middle of a structure instead of at the end.

This means that you have to be careful that all applications link to the custom libc++.

For libcxx developers,

While we are discussing how to avoid ODR violations, is it possible to move the two lines at
https://github.com/llvm/llvm-project/blob/llvmorg-12.0.1/libcxx/src/locale.cpp#L210-L211

to the bottom to avoid segfaults resulting from this?

Thanks,

Isuru

2

Generally speaking, we make the assumption that there’s a single copy of libc++ in use inside a final linked image. Or if you want to use libc++ as an implementation detail and link it statically, ensure it doesn’t export any symbols and has absolutely no ABI surface outside of the executable where it’s being used. Chrome does that.

Anything else is very brittle, and I’m reluctant to add workarounds that make it look like it works, because you may run into other issues that we won’t be able to patch so easily.

Louis

3

I had thought libc++ was more immune to this kind of problem given Apple’s usual support to deploy on older systems.

If the libc++ structures have changed and are now incompatible with previous versions of libc++, would that now mean that an Apple build system using the new libc++ structures can no longer target a deployment target with an older libc++ (any system prior to the change)?

Ken

4

Thanks Louis for the clarification. I’ll try to make sure we are using only one version.

Ken, it’s incompatible, but this is an implementation detail and will not be a problem if you are using only one libc++.
Problems happen when mixing two different versions of libc++.

Isuru

5

Hi Ken,

I’m in the same situation as you in conda package manager.
Note that with libcxx 12, https://reviews.llvm.org/D91517 was merged where
codecvt with char8_t is added in the middle of a structure instead of at the end.

This means that you have to be careful that all applications link to the custom libc++.

For libcxx developers,

While we are discussing how to avoid ODR violations, is it possible to move the two lines at
https://github.com/llvm/llvm-project/blob/llvmorg-12.0.1/libcxx/src/locale.cpp#L210-L211

to the bottom to avoid segfaults resulting from this?

Generally speaking, we make the assumption that there’s a single copy of libc++ in use inside a final linked image. Or if you want to use libc++ as an implementation detail and link it statically, ensure it doesn’t export any symbols and has absolutely no ABI surface outside of the executable where it’s being used. Chrome does that.

Anything else is very brittle, and I’m reluctant to add workarounds that make it look like it works, because you may run into other issues that we won’t be able to patch so easily.

Louis

I had thought libc++ was more immune to this kind of problem given Apple’s usual support to deploy on older systems.

It can’t really be immune to those problems since they are byproducts of how linking works at a pretty fundamental level. Back-deploying to older platforms works nicely as long as you use the libc++ shipped with the system (we make sure of that), or that you “embed” libc++ into your application by linking it as a static archive and disregard the system libc++ altogether (Chrome does that). When you do something in-between, that’s when things start failing.

If the libc++ structures have changed and are now incompatible with previous versions of libc++, would that now mean that an Apple build system using the new libc++ structures can no longer target a deployment target with an older libc++ (any system prior to the change)?

There was no ABI breaking change to the libc++ structures in https://reviews.llvm.org/D91517. We still support the same back-deployment targets as before.

Louis

6

Thanks for taking a moment to respond. I always appreciate the feedback and knowledge I receive here from you and others, and TBH I am usually reluctant to ask many questions, recognizing the rarefied air around this place. Despite that, I will humbly ask:

Isuru notes that the facet structure in libc++ used to be ABCD, and now as of D91517 it is ABCFD, with a new set of fields in the middle. He notes crashes when mixing binaries built against before and after D91517 versions of libc++, which is fair enough and easy to understand.

However, If you build against libc++ headers with the ABCFD version of that facet structure, and link either statically or dynamically (it would seem to not matter which) to a libc++ that was built with an ABCFD facet structure, then your application will be passing around ABCFD facet objects.

If you then deploy onto a system with the older ABCD facet structure in libc++, and you try to pass that ABCFD facet object to any system framework or library to do something with, will you not be in ODR-violation territory?

(If this is a CS-101 level question that is just obvious to everyone, I will take it over to StackOverflow :> )

Ken

7

Hi,

I came back to this from [libc++] Remove _LIBCPP_DISABLE_AVAILABILITY macro by ldionne · Pull Request #112952 · llvm/llvm-project · GitHub and noticed that this was left unanswered. I think this was around the time of the Discourse migration so that may be why this was lost.

Either way, libc++'s goal is always to be 100% ABI compatible, except in tiny corner cases where we evaluate that it won’t impact anyone. However, I don’t see how the changes in D91517 could break the ABI. I think the issue reported by @isuruf is slightly different.

Isuru notes that the facet structure in libc++ used to be ABCD, and now as of D91517 it is ABCFD, with a new set of fields in the middle.

Yes, but the word “structure” here is used a bit incorrectly. There is an array internal to the built library (.a or .dylib) that used to contain elements in order ABCD, and now it contains elements in order ABCFD. But that’s an implementation detail of the built library, and it doesn’t affect anything on its ABI boundary. That internal array gets passed around as a black box by the headers.

He notes crashes when mixing binaries built against before and after D91517 versions of libc++, which is fair enough and easy to understand.

Specifically, I think the problem that was reported is that if a program uses both an old libc++.a and a new libc++.a at the same time, then you may get crashes. In other words, running into issues with this requires having multiple copies of libc++ in the same program, which is indeed an ODR violation unless you’re very careful.

What’s happening is basically that the new libc++.a is creating an array in order ABCFD, and then some other part of the program is processing it using a function taken from the old libc++.a that expects it as ABCD. Or something along those lines. But if you had exactly one copy of the library in your program (as should be done), there would be no problem.

In particular, there is no backwards compatibility problem in the general sense: if you compiled your program against old headers and ran against an old libc++.dylib, it works since the headers are passing the ABCD array as a black box and never reaching into it. If you now run the program against a new libc++.dylib (e.g. you update your system and get a new version of the dylib but you don’t rebuild your program), the code instantiated from the headers are still passing an array around as a black box, except it now happens to be laid out as ABCFD. There is no problem as long as the headers don’t look into that “black box”, which AFAICT they don’t.

1 Like