RFC: Nullability qualifiers

Clang Frontend
1

Nullability Qualifiers
We propose the addition of a new set of type qualifiers, spelled __nullable, __nonnull, and __null_unspecified, to Clang. These are collectively known as nullability qualifiers and may be written anywhere any other type qualifier may be written (such as const) on any type subject to the following restrictions:

__nonnull is already used by glibc headers. Can you please use a different name so that we don't have the __block situation all over again? And, hopefully, do that before everybody using the beta has already changed their code to use __nonnull and thus making it too late to change it?

As this seems to be a general problem (Apple introducing language extensions that work fine on OS X/iOS but breaking glibc), maybe it would make sense to not enable any language extensions when Clang can detect it's a (g)libc header? Or would it maybe be possible that you use _Foo instead __foo for your language extensions? The C standard reserves both for the compiler and libc, however, glibc only seems to use __foo. So, maybe change this to _Nonnull and _Nullable?

Thanks.

PS: Please CC me as I'm not on the list!

2

Nullability Qualifiers
We propose the addition of a new set of type qualifiers, spelled __nullable, __nonnull, and __null_unspecified, to Clang. These are collectively known as nullability qualifiers and may be written anywhere any other type qualifier may be written (such as const) on any type subject to the following restrictions:

__nonnull is already used by glibc headers. Can you please use a different name so that we don't have the __block situation all over again? And, hopefully, do that before everybody using the beta has already changed their code to use __nonnull and thus making it too late to change it?

I hadn't noticed this, and none of the initial feedback on this feature (3-4 months ago) mentioned the collision. We made some other changes based on feedback here, but at this point, it's far too late to change the spelling of __nonnull: Apple Clang has been shipping with this feature since Xcode 6.3.

As this seems to be a general problem (Apple introducing language extensions that work fine on OS X/iOS but breaking glibc), maybe it would make sense to not enable any language extensions when Clang can detect it's a (g)libc header? Or would it maybe be possible that you use _Foo instead __foo for your language extensions? The C standard reserves both for the compiler and libc, however, glibc only seems to use __foo. So, maybe change this to _Nonnull and _Nullable?

It's a general problem when you have different compiler and library vendors all working in the same space. Some vendor will conflict with another. It's more fun when a vendor conflicts with itself, eg, libstdc++ 4.2's __is_pod class template vs. GCC > 4.2's __is_pod type trait intrinsic, but it happens even when everything is developed in the open over years: C++11 lambda syntax is barely distinguishable from C array designated initializers and Objective-C message sends, both of which predates lambdas by a decade. With very few exceptions, we can deal with this gracefully in Clang.

  - Doug

3

I hadn't noticed this, and none of the initial feedback on this feature (3-4 months ago) mentioned the collision. We made some other changes based on feedback here, but at this point, it's far too late to change the spelling of __nonnull: Apple Clang has been shipping with this feature since Xcode 6.3.

Well, Xcode 6.3 is still kind of new and I don't know how widespread the use of __nonnull in OS X / iOS code is, but I guess not much, since this has only been announced at WWDC as a feature of Xcode 7. However, I'm not really sure it's a good idea to keep it like that just because there might already be some use of it.

How about this: Maybe we should change the name,and *in addition*, Clang on OS X / iOS keeps accepting __nonnull? This will not break any existing code and new code can use a name that has no conflicts. If this does not work for you, what would work for you and solve the problem? Since this is not only an ObjC feature, but a C feature, like it is right now, C code using it is no longer portable.

Also, since this is not the first time this happens, can you guys maybe add to your release guidelines to test if there are conflicts on Linux and/or other major OSes? Just grepping through glibc headers would already have prevented it and would have taken maybe a minute. Maybe as a general rule of thumb, if you use something that's reserved by the standard, check if there's anything else that can also claim it according to those reservation rules and check if it already uses that name.

It's a general problem when you have different compiler and library vendors all working in the same space. Some vendor will conflict with another. It's more fun when a vendor conflicts with itself, eg, libstdc++ 4.2's __is_pod class template vs. GCC > 4.2's __is_pod type trait intrinsic, but it happens even when everything is developed in the open over years: C++11 lambda syntax is barely distinguishable from C array designated initializers and Objective-C message sends, both of which predates lambdas by a decade. With very few exceptions, we can deal with this gracefully in Clang.

Yes, I totally agree here. The standard was written under the assumption that the compiler vendor is also the library vendor. It would totally make sense to change this in the next version of C, e.g. specify _Foo is for the compiler and __foo for the libc. However, this will take quite a long while, if it ever succeeds, and won't give us a solution for now :).

4

I hadn't noticed this, and none of the initial feedback on this feature (3-4 months ago) mentioned the collision. We made some other changes based on feedback here, but at this point, it's far too late to change the spelling of __nonnull: Apple Clang has been shipping with this feature since Xcode 6.3.

Well, Xcode 6.3 is still kind of new and I don't know how widespread the use of __nonnull in OS X / iOS code is, but I guess not much, since this has only been announced at WWDC as a feature of Xcode 7. However, I'm not really sure it's a good idea to keep it like that just because there might already be some use of it.

The use of this feature is too widespread in iOS/OS X to change the spelling there. The feature was in the Xcode 6.3 release notes, the subject of a Swift blog entry, and has been adopted in the wild. It is too late.

How about this: Maybe we should change the name,and *in addition*, Clang on OS X / iOS keeps accepting __nonnull? This will not break any existing code and new code can use a name that has no conflicts. If this does not work for you, what would work for you and solve the problem? Since this is not only an ObjC feature, but a C feature, like it is right now, C code using it is no longer portable.

You're jumping to the conclusion that this is an unresolvable conflict and that it's the language feature that must change. We might have overlapping features, or an easy way to teach Clang to account for the differences. Creating a terminology schism has costs, too, if both uses are driving toward the same goal.

What breaks due to this conflict?

Also, since this is not the first time this happens, can you guys maybe add to your release guidelines to test if there are conflicts on Linux and/or other major OSes? Just grepping through glibc headers would already have prevented it and would have taken maybe a minute. Maybe as a general rule of thumb, if you use something that's reserved by the standard, check if there's anything else that can also claim it according to those reservation rules and check if it already uses that name.

Your tone here is not conducive to a productive discussion. We don't vilify vendors nor patronize it's developers here in the Clang community. Keep it civil and keep it technical.

And, for the record, this language feature has been tested on Linux and this did not manifest as an observable problem there.

It's a general problem when you have different compiler and library vendors all working in the same space. Some vendor will conflict with another. It's more fun when a vendor conflicts with itself, eg, libstdc++ 4.2's __is_pod class template vs. GCC > 4.2's __is_pod type trait intrinsic, but it happens even when everything is developed in the open over years: C++11 lambda syntax is barely distinguishable from C array designated initializers and Objective-C message sends, both of which predates lambdas by a decade. With very few exceptions, we can deal with this gracefully in Clang.

Yes, I totally agree here. The standard was written under the assumption that the compiler vendor is also the library vendor. It would totally make sense to change this in the next version of C, e.g. specify _Foo is for the compiler and __foo for the libc. However, this will take quite a long while, if it ever succeeds, and won't give us a solution for now :).

You missed my point completely.

Conflicts over syntax happen, even under the most idealized situation of well-established languages and slow-moving standards committees. We don't point fingers, we dive into the technical problem and come up with a solution. And most of the time, it's worked out quite well in Clang.

A quick search shows that glibc is defining __nonnull as a function-like macro, so this conflict is trivial to resolve. Is there something more I've missed that causes an actual problem?

  - Doug

5

The use of this feature is too widespread in iOS/OS X to change the spelling there. The feature was in the Xcode 6.3 release notes, the subject of a Swift blog entry, and has been adopted in the wild. It is too late.

Which is why I suggested to have __nonnull as an alias to whatever the new name would be on OS X / iOS. Can you please elaborate why that would not be a solution? All existing code will keep working.

You're jumping to the conclusion that this is an unresolvable conflict and that it's the language feature that must change. We might have overlapping features, or an easy way to teach Clang to account for the differences. Creating a terminology schism has costs, too, if both uses are driving toward the same goal.

I don't see where this is a problem of overlapping features. It's an issue of using a name that is already used.

What breaks due to this conflict?

Well, there are two cases we have to look at: The case where Clang supports nullability and the case where it doesn't.

Let's start with the one that has nullability:
glibc defines __nonnull in sys/cdef.h. This will change the behavior of __nonnull to that that glibc wants. If you now want to use code that uses the __nonnull from Clang, things will not compile.

Now for the case where you don't have nullability:
You add __nonnull to your code to have the new nullability feature on OS X / iOS. Because this is not available in upstream Clang yet, you add code like:

#if !__has_feature(nullability)
# define __nonnull
# define __nullable
#endif

Now, __nonnull is already defined and thus this won't work. You might think that it's a good idea to just #undef __nonnull. However, this will break the glibc headers.

Your tone here is not conducive to a productive discussion. We don't vilify vendors nor patronize it's developers here in the Clang community. Keep it civil and keep it technical.

I don't see how my tone here is not productive - I was merely suggesting what to do to prevent this in the future, as this is not the first time of this is happening and I'm sensing a pattern here. These were suggestions to make sure this won't happen again. The whole __block thing was a real problem for quite a long while.

And, for the record, this language feature has been tested on Linux and this did not manifest as an observable problem there.

Could you share what exactly you tested? For me, even the case where your Clang doesn't support nullability fails, so I didn't even bother trying further after that, as that was already a complete blocker to using __nonnull.

You missed my point completely.

Conflicts over syntax happen, even under the most idealized situation of well-established languages and slow-moving standards committees. We don't point fingers, we dive into the technical problem and come up with a solution. And most of the time, it's worked out quite well in Clang.

A quick search shows that glibc is defining __nonnull as a function-like macro, so this conflict is trivial to resolve. Is there something more I've missed that causes an actual problem?

How is this trivially fixed other than changing it in glibc to have a different name? Could you please elaborate? The only other way I can think of is not making __nonnull available for system headers, which, honestly, strikes me as a bad idea as that means system headers can't define libc functions a __nonnull, which might be quite useful.

6

> The use of this feature is too widespread in iOS/OS X to change the
spelling there. The feature was in the Xcode 6.3 release notes, the subject
of a Swift blog entry, and has been adopted in the wild. It is too late.

Which is why I suggested to have __nonnull as an alias to whatever the new
name would be on OS X / iOS. Can you please elaborate why that would not be
a solution? All existing code will keep working.

> You're jumping to the conclusion that this is an unresolvable conflict
and that it's the language feature that must change. We might have
overlapping features, or an easy way to teach Clang to account for the
differences. Creating a terminology schism has costs, too, if both uses are
driving toward the same goal.

I don't see where this is a problem of overlapping features. It's an issue
of using a name that is already used.

> What breaks due to this conflict?

Well, there are two cases we have to look at: The case where Clang
supports nullability and the case where it doesn't.

Let's start with the one that has nullability:
glibc defines __nonnull in sys/cdef.h. This will change the behavior of
__nonnull to that that glibc wants. If you now want to use code that uses
the __nonnull from Clang, things will not compile.

__nonnull is defined as a function-like macro, so any use that is not
followed by a ( will work fine. That makes the portability problem
significantly less severe.

Now for the case where you don't have nullability:
You add __nonnull to your code to have the new nullability feature on OS X
/ iOS. Because this is not available in upstream Clang yet, you add code
like:

#if !__has_feature(nullability)
# define __nonnull
# define __nullable
#endif

Now, __nonnull is already defined and thus this won't work. You might
think that it's a good idea to just #undef __nonnull. However, this will
break the glibc headers.

That seems like an odd way to deal with that portability problem. The
conventional way is:

#if __has_feature(nullability)
#define NONNULL __nonnull
#else
#define NONNULL
#endif

... which again works without problems so long as you don't follow NONNULL
with a left parenthesis.

7

__nonnull is defined as a function-like macro, so any use that is not followed by a ( will work fine. That makes the portability problem significantly less severe.

Hm, indeed, weird. I got some errors about not giving it enough arguments, however, I can't reproduce what I did there.

That seems like an odd way to deal with that portability problem. The conventional way is:

#if __has_feature(nullability)
#define NONNULL __nonnull
#else
#define NONNULL
#endif

... which again works without problems so long as you don't follow NONNULL with a left parenthesis.

I disagree about this being odd. As a counter example, restrict was added in C99. #define restrict was a very common thing to make sure it still compiles as C89.

As for using a hack with a define, this is what I did for now. Still, it's a hack and makes all declarations look ugly and weird. Wouldn't it be much nicer to have a keyword that you can just define to nothing for backwards compatibility, like with restrict? (That does not mean we can't keep __nonnull in addition to another name on OS X / iOS, so it would not break any existing code).

I wonder if it makes sense to bring up this issue with the glibc maintainers as well?

8

> __nonnull is defined as a function-like macro, so any use that is not
followed by a ( will work fine. That makes the portability problem
significantly less severe.

Hm, indeed, weird. I got some errors about not giving it enough arguments,
however, I can't reproduce what I did there.

> That seems like an odd way to deal with that portability problem. The
conventional way is:
>

> #if __has_feature(nullability)
> #define NONNULL __nonnull
> #else
> #define NONNULL
> #endif
>
> ... which again works without problems so long as you don't follow
NONNULL with a left parenthesis.

I disagree about this being odd. As a counter example, restrict was added
in C99. #define restrict was a very common thing to make sure it still
compiles as C89.

Well, 'restrict' was not a reserved identifier in C89. __nonnull is a
reserved identifier in all C-family languages. But you're right, some
people might be #defining __nonnull themselves, and if they do, their code
will be non-portable.

As for using a hack with a define, this is what I did for now. Still, it's
a hack and makes all declarations look ugly and weird. Wouldn't it be much
nicer to have a keyword that you can just define to nothing for backwards
compatibility, like with restrict? (That does not mean we can't keep
__nonnull in addition to another name on OS X / iOS, so it would not break
any existing code).

I don't think that using two separate reserved identifiers for this keyword
is really much nicer than using one. If we could use a 'prettier' keyword
for this, like "nonnull", that might be nicer, but we really don't want
Clang to introduce its own non-conforming language mode like that. But you
can create this state of affairs yourself with a #define.

9

Well, 'restrict' was not a reserved identifier in C89. __nonnull is a reserved identifier in all C-family languages. But you're right, some people might be #defining __nonnull themselves, and if they do, their code will be non-portable.

Well yes, that's true…

I don't think that using two separate reserved identifiers for this keyword is really much nicer than using one. If we could use a 'prettier' keyword for this, like "nonnull", that might be nicer, but we really don't want Clang to introduce its own non-conforming language mode like that. But you can create this state of affairs yourself with a #define.

That's why I said only for OS X / iOS.

#define nonnull __nonnull will not work, as this *also* introduces nonnull for use within ObjC method declarations and as a property attribute.