[RFC] _Optional: a type qualifier to indicate pointer nullability

Clang Frontend
1

Abstract: This paper proposes a new type qualifier for the purpose of adding pointer nullability information to C programs. Its goal is to provide value not only for static analysis and documentation, but also for compilers which report errors based only on existing type-compatibility rules. The syntax and semantics are designed to be as familiar (to C programmers) and ergonomic as possible. In contrast, existing solutions are incompatible, confusing, error-prone, and intrusive.

Philosophical underpinning

The single most important (and redeeming) feature of C is its simplicity. It should be (relatively) quick to learn every aspect of the language, (relatively) easy to create a compiler for it, and the language’s semantics should follow (more-or-less) directly from its syntax.

People criticise C’s syntax, but I consider it the foundation of the language. Any experienced C programmer has already acquired the mindset necessary to read and write code using it. Aside from the need to minimize incompatibilities, the syntactic aberrations introduced by C++ can be ignored.

“Pythonic” is sometimes used as an adjective to praise code for its use of Python-specific language idioms. I believe that an equivalent word “scenic” could be used to describe C language idioms, meaning that they conform to a mode of expression characteristic of C. I’ve tried to keep that in mind when evaluating syntax and semantics.

Inspiration from Python

For the past twenty years, I’ve mostly been coding in C. I had always considered C to be a strongly-typed language: it allows implicit conversions between void * pointers and other pointer types, and between enum and integer types, but those aren’t serious shortcomings so long as the programmer is aware of them.

Recently, I switched to a team that writes code in a mixture of languages (including C++, Python, and Javascript). Writing code in languages that are dynamically-typed but with statically checked type annotations was a revelation to me. Our project uses MyPy and Typescript for static type checking.

The main thing that I grew to appreciate was the strong distinction that MyPy makes between values that can be None and values that cannot. Such values are annotated as Optional[int], for example. Any attempt to pass an Optional value to a function that isn’t annotated to accept None is faulted, as is any attempt to do unguarded operations on Optional values (i.e. without first checking for the value being None).

Problem statement

In contrast to Python, C’s type system makes no distinction between pointer values that can be null, and those that cannot. Effectively, any pointer in a C program can be null, which leads to repetitive, longwinded and unverifiable parameter descriptions such as “Non-null pointer to…” or “Address of X … (must not be null)”.

Such invariants are not usually documented within a function except by assertions, which clutter the source code and are ineffective without testing. Some programmers even write tests to verify that assertions fail when null is passed to a function, although the same stimulus would provoke undefined behaviour in release builds. The amount of time and effort that could be saved if such misuse were instead caught at compile time is huge.

Isn’t this a solved problem?

Given that the issue of undefined behaviour caused by null pointer dereferences has been present in C since its inception, many solutions have already been attempted.

C99 extended the syntax for function arguments to allow static within [], which requires the passed array to be at least a specified size:

void *my_memcpy(char dest[static 1], const char src[static 1], size_t len);

void test(void)
{
  char *dest = NULL, *src = NULL;
  my_memcpy(NULL, NULL, 10); // warning: argument 1 to 'char[static 1]' is null where non-null expected
  my_memcpy(dest, src, 10); // no compiler warning
}

This trick may generate a warning in cases where a null pointer constant is specified directly as a function argument — but not for any other source of null such as a failed call to malloc. It’s not a general-purpose solution anyway because arrays of type void are illegal, which makes this syntax unusable for declaring functions such as memcpy.

A GNU compiler extension (also supported by Clang and the ARM compiler) allows function arguments to be marked as not supposed to be null:

void *my_memcpy(void *dest, const void *src, size_t len) __attribute__((nonnull (1, 2)));

void test(void)
{
  char *dest = NULL, *src = NULL;
  my_memcpy(NULL, NULL, 10); // warning: argument 1 null where non-null expected
  my_memcpy(dest, src, 10); // no compiler warning
}

I find the __attribute__ syntax intrusive and verbose. It is also error-prone because attributes only apply to function declarations as a whole: it’s easy to accidentally specify wrong argument indices, because the arguments themselves are not annotated. Clang extended the syntax to allow __attribute__((nonnull)) to be used within an argument list, but the GNU compiler does not support that.

Historically, the semantics of __attribute__((nonnull)) weren’t very useful: it only detected cases where a null pointer constant was specified directly as a function argument. However, version 10 of the GNU compiler introduced a new feature, -fanalyzer, which uses the same __attribute__ information during a static analysis pass:

<source>:8:3: warning: use of NULL 'dest' where non-null expected [CWE-476] [-Wanalyzer-null-argument]
8 | my_memcpy(dest, src, 10); // no compiler warning
| ^~~~~~~~~~~~~~~~~~~~~~~~
'test': events 1-3
|
| 7 | char *dest = NULL, *src = NULL;
| | ^~~~ ~~~
| | | |
| | | (2) 'dest' is NULL
| | (1) 'dest' is NULL
| 8 | my_memcpy(dest, src, 10); // no compiler warning
| | ~~~~~~~~~~~~~~~~~~~~~~~~
| | |
| | (3) argument 1 ('dest') NULL where non-null expected
|
<source>:4:7: note: argument 1 of 'my_memcpy' must be non-null
4 | void *my_memcpy(void *dest, const void *src, size_t len) __attribute__((nonnull (1, 2)));
| ^~~~~~~~~

RFC: Nullability qualifiers (2015) proposed not one but three new type annotations: _Nullable, _Nonnull and _Null_unspecified. Support for these was added in version 3.7 of Clang, but GCC doesn’t recognize them. Like GCC without -fanalyzer, Clang itself only detects cases where a null pointer constant is specified directly as a function argument:

void *my_memcpy(void *_Nonnull dest, const void *_Nonnull src, size_t len);
void test(void)
{
  char *dest = NULL, *src = NULL;
  my_memcpy(NULL, NULL, 10); // warning: Null passed to a callee that requires a non-null 1st parameter
  my_memcpy(dest, src, 10); // no compiler warning
}

However, Clang-tidy, a standalone tool based on Clang, can issue warnings about misuse of pointers that it is able to infer based on annotations and path-sensitive analysis:

<source>:9:3: warning: Null pointer passed to 1st parameter expecting 'nonnull' [clang-analyzer-core.NonNullParamChecker]
my_memcpy(dest, src, 10); // no compiler warning
^ ~~~~
<source>:7:9: note: 'dest' initialized to a null pointer value
char *dest = NULL, *src = NULL;
^~~~
<source>:9:3: note: Null pointer passed to 1st parameter expecting 'nonnull'
my_memcpy(dest, src, 10); // no compiler warning
^ ~~~~

Clang’s syntax is less verbose and error-prone than __attribute__, but the requirement to annotate all pointers as either _Nullable or _Nonnull makes code harder to read and write. Most pointers should not be null: consider the instance pointer passed to every method of a class. It’s no longer safe to write such declarations in traditional style with economy of effort. I also think the semantics of these annotations (discussed later) are far more complex than befits a simple language like C, and likely to cause confusion.

I’ve seen Clang’s nullability qualifiers described as “enormous and useless noise, while providing doubtful value” and the very idea of annotating pointers called a “naive dream”. I agree with the first statement, but not the second: other languages have shown that null safety is achievable and useful, whilst C lags with competing partial solutions that are confusing, error-prone, and intrusive.

Clang’s annotations provide no value to other compilers, which either ignore (if removed by macros) or reject them. Not all developers use special build machines costing thousands of pounds: I do a lot of coding on a Raspberry Pi, using a toolchain that dates from the 1980s but is still actively maintained (and recently gained support for C17). For me, having a rapid edit-compile-run cycle is paramount.

Even tiny compilers such as cc65 can check that the addresses of objects declared with const or volatile are not passed to functions that do not accept such pointers, because the rules for type compatibility are simple (for the benefit of machines and people). This is exactly the niche that the C language should be occupying.

I postulate that improved null safety does not require path-sensitive analysis.

Syntactic and semantic precedents

Type qualifiers (as we understand them today) didn’t exist in pre-ANSI C, which consequently had a stronger similarity between declarations and expressions, since qualifiers can’t appear in expressions (except as part of a cast).

The second edition of ‘The C Programming Language’ (K&R, 1988) says only that:

Types may also be qualified, to indicate special properties of the objects being declared.

Notably, the special properties conferred by const, volatile, restrict and _Atomic all relate to how objects are stored or how that storage is accessed — not the range of values representable by an object of the qualified type.

Is the property of being able to represent a null pointer value the kind of property that should be indicated by a type-qualifier? Restrictions on the range of values representable by an object are usually implied by its type-specifiers (although long, short, signed and unsigned are intriguingly also called “qualifiers” by K&R, presumably because their text predates ANSI C).

Pointers are a special type of object though. Multiple levels of indirection can be nested within a single declaration, as in the following declaration of baz (an array of pointers to arrays of pointers to int):

int bar;
int *foo[2] = {NULL, &bar};
int *(*baz[3])[2] = {&foo, NULL, NULL};

It’s therefore necessary to specify whether null is permitted for every level of indirection within a declarator (e.g. for both baz[3] and (*baz[3])[2]). The only existing element of C’s existing syntax that has such flexibility is a type-qualifier.

It’s not meaningful to specify whether null is permitted as part of the declaration-specifiers (e.g. static int) on the lefthand side of a declaration, because this property only applies to pointers. The restrict qualifier already has this limitation.

Here’s an example of how the above declaration might look with Clang’s nullability qualifiers:

int bar;
int *_Nullable foo[2] = {NULL, &bar};
int *_Nullable (*_Nullable baz[3])[2] = {&foo, NULL, NULL};

Syntactically, this may look like a perfect solution; semantically, I will argue that it is not!

A variable of type char *const (const pointer to char) can be assigned to a variable of type char * (pointer to char), but a variable of type const char * (pointer to const char) cannot. After a learner internalizes the knowledge that qualifiers on a pointer target must be compatible, whereas qualifiers on a pointer value are discarded, this rule can be applied to any assignment or initialization:

int *const x = NULL;
int *s = x; // no warning
int *volatile y = NULL;
int *t = y; // no warning
int *restrict z = NULL;
int *r = z; // no warning

One might not expect the same laxity to apply to the _Nullable and _Nonnull qualifiers, because they relate to the assigned value, not the storage access properties of a particular copy of it. Despite that, Clang-tidy allows an assigned value to be _Nullable unless the type of the assigned-to-object is qualified as _Nonnull:

extern int *_Nullable getptr(void);
int *_Nullable z = getptr();
int *q = z; // no warning
int *_Nonnull p = z; // warning: Nullable pointer is assigned to a pointer which is expected to have non-null value
*q = 10; // warning: Nullable pointer is dereferenced

This compromise between the traditional semantics of assignment (discard top-level qualifiers) and the semantics needed to track nullability (ensure compatible qualifiers) looks like a weak basis for null safety; however, it is mitigated by the fact that the static analyser tracks whether a pointer value may be null regardless of its type. In turn, that makes it impossible to tell what constraints apply to a pointer value simply by referring to its declaration.

A related issue is that top-level qualifiers on arguments are redundant in a function declaration (as opposed to definition) because arguments are passed by value. Callers don’t care what the callee does with its copy of a pointer argument — only what it does with the pointed-to object.

Consequently, such qualifiers are ignored when determining compatibility between declarations and definitions of the same function. The normative part of an argument declaration is to the left of the asterisk:

void myfunc(const char *const s);
//          ^^^^^^^^^^  ^^^^^
//          Normative   Not normative
//          vvvvvvvvvv  vvvvvvvv
void myfunc(const char *restrict s)
{
}

Notably, this rule also applies to restrict-qualified arguments, despite an apparent conflict with a principle stated in WG14’s charter:

Application Programming Interfaces (APIs) should be self-documenting when possible

The same laxity should not apply to the _Nullable and _Nonnull qualifiers, because they relate to the passed value, not its storage access properties. Despite that, Clang ignores any differences between rival declarations of a function, except in cases where contradictory qualifiers were used.

It is permissible to write [] instead of * in a parameter declaration, to hint that an array is passed (by reference) to a function. One might expect this [] syntax to be incompatible with qualifying the type of the pointer (as opposed to the type of array elements). On the contrary, Clang allows nullability qualifiers to appear between the brackets:

void myfunc(const char s[_Nullable]); // s may be a null pointer

This syntax is not intuitive to me (usually [] indicates an index or size) but it does follow 6.7.5.3 of the C language standard:

A declaration of a parameter as ‘‘array of type’’ shall be adjusted to ‘‘qualified pointer to type’’, where the type qualifiers (if any) are those specified within the [ and ] of the array type derivation.

Thinking outside the box

An essential feature of a new type qualifier expressing ‘may be null’ is that this property must not be lost when a qualified pointer is copied (including when it is passed as a function argument).

Qualifiers on a pointed-to type must be compatible in assignments, initializations, and function calls, whereas qualifiers on a pointer type need not be. The fact that every programmer has internalized this rule makes me reluctant to propose (or embrace) any change to it for nullability qualifiers on a pointer type.

I’m tempted to say that both restrict and the Clang annotations _Nullable and _Nonnull are in the wrong place. The restrict qualifier frees an optimizer to generate more efficient code, almost like the opposite of volatile. Isn’t the quality of being aliased a property of an object, rather than any single pointer to it?

At the heart of C’s syntax is the primacy of fundamental types such as int. Every declaration is a description of how a chain of indirections leads to such a type. Can we reframe the ‘may be null’ property as a quality of the pointed-to object, rather than the pointer?

Yes!

const int *i; // *i is an int that may be stored in read-only memory
volatile int *j; // *j is an int that may be stored in shared memory
_Optional int *k; // *k is an int for which no storage may be allocated

I chose the name _Optional to bootstrap existing knowledge of Python and make a clear distinction between this qualifier and _Nullable. I also like the idea of Python giving something back to C.

_Optional is the same length as _Nullable and only one character longer than volatile. C’s syntax isn’t known for its brevity, anyway. (Think not of functions such as strcpy, but of declarations such as const volatile unsigned long int.)

Modifying a const object only has undefined behaviour if the object was originally declared as const, which is not always the case when an object is modified by dereferencing a pointer from which a const qualifier was cast away. Likewise, accessing an _Optional object will only have undefined behaviour if the pointer used to access the object is actually null.

Read-only objects are often stored in a separate address range so that illegal write accesses generate a segmentation fault (on machines with an MMU). Likewise, null pointer values encode a reserved address, which is typically neither readable nor writable by user programs. In both cases (const and _Optional), a qualifier on the pointed-to object indicates something about its address.

Unlike assignment to a variable with const-qualified type, no error should be reported when compiling code which accesses a variable with _Optional-qualified type. Were that my intent, I would have proposed a name like _None rather than _Optional. Requiring the _Optional qualifier to be cast away before accessing a so-qualified object would be tiresome and would sacrifice type safety for null safety. I do not think that is a good trade-off.

Despite this limitation, the new qualifier is useful:

  • It allows interfaces to be self-documenting. (Function declarations must match their definition.)
  • It allows the compiler to report errors on initialization or assignment, if implicitly converting a pointer to _Optional into a pointer to an unqualified type.
  • It provides information to static analysis tools, which can warn about dereferences of a pointer to _Optional if path-sensitive analysis does not reveal a guarding check for null in the preceding code.

Here is some example usage:

void foo(int *);

void bar(_Optional int *i)
{
  *i = 10; // optional warning of unguarded dereference

  if (i) {
    *i = 5; // okay
  }

  int *j = i; // warning: initializing discard qualifiers
  j = i; // warning: assignment discards qualifiers
  foo(i); // warning: passing parameter discards qualifiers
}

Here’s an example of complex declarations that I used earlier, updated to use the proposed qualifier:

   int bar;

   _Optional int *foo[2] = {NULL, &bar};
// ^^decl-spec^^ ^^decl^

   _Optional int *(*qux[3])[2] = {&foo, &foo, &foo};
// ^^decl-spec^^ ^^declarator^

   _Optional int *_Optional (*baz[3])[2] = {&foo, NULL, NULL};
//                           ^^decl^
// ^^decl-spec^^ ^^pointer^ ^^dir-decl^^
//               ^^^^^^declarator^^^^^^^

Let’s break it down:

  • Storage is allocated for an object, bar, of type int. This will be used as the target of a pointer to _Optional int but doesn’t need to be qualified as such (any more than a const array must be passed to strlen).
  • Storage is allocated for an array, foo, of two pointers to _Optional int. _Optional in the declaration-specifiers indicates that elements of foo may be null; an expression resembling the declarator (e.g. *foo[0]) may have undefined behaviour.
  • Storage is allocated for an array, qux, of three pointers to arrays of pointers to _Optional int. _Optional in the declaration-specifiers indicates that elements of the pointed-to arrays may be null; an expression resembling the declarator (e.g. *(*qux[0])[0]) may have undefined behaviour.
  • Storage is allocated for an array, baz, of three pointers to _Optional arrays of pointers to _Optional int. _Optional in the pointer(opt) of the top-level declarator indicates that elements of baz may be null; an expression resembling the inner declarator (e.g. *baz[0]) may have undefined behaviour. _Optional in the declaration-specifiers has the same meaning as for qux.

Note that an ‘optional pointer’ is not a pointer that may have the value null; it’s a pointer that may not exist. This is like the existing rule that a ‘const pointer’ is not a pointer to read-only memory; it’s a pointer that may be stored in read-only memory.

Parameter declarations using [] syntax can be written more naturally using an _Optional qualifier than using Clang’s _Nullable qualifier:

void myfunc(_Optional const char s[]); // s may be a null pointer

With the above exception, it isn’t useful to declare a non-pointed-to object as _Optional (although so-qualified types will exist during expression evaluation). Such declarations could be disallowed, like similar abuse of restrict, to avoid confusion.

Conversions from maybe-null to not-null

I presented the idea of warnings when a pointer-to-_Optional is passed to a function with incompatible argument types as an unalloyed good. In fact, such usage has legitimate applications.

Consider the following veneer for the strcmp function which safely handles null pointer values by substituting the empty string:

int safe_strcmp(_Optional const char *s1, _Optional const char *s2)
{
  if (!s1) s1 = "";
  if (!s2) s2 = "";
  return strcmp(s1, s2); // warning: passing parameter discards qualifiers
}

In the above situation, both s1 and s2 would both need to be cast before calling strcmp:

int safe_strcmp(_Optional const char *s1, _Optional const char *s2)
{
  if (!s1) s1 = "";
  if (!s2) s2 = "";
  return strcmp((const char *)s1, (const char *)s2);
}

The above solution would be detrimental to readability and type safety.

It could be argued that any mechanism to remove _Optional from the target of a pointer without first checking its value (at runtime) fatally compromises null safety. I disagree: C provides tools to write type-safe code, whilst allowing leniency where it is pragmatic to do so.

It might be possible to use some combination of _Generic and unqual_typeof to remove only a specific qualifier from a type (like const_cast in C++) but such casts would still clutter the code and therefore seem likely be rejected by programmers who prefer to rely solely on path-sensitive analysis.

What is required is a solution that accommodates both advanced compilers and compilers which report errors based only on simple type-compatibility rules. Compilers capable of doing so must be able to validate conversions from maybe-null to not-null in the same way as they would validate a real pointer dereference.

One of my colleagues suggested just such a solution:

int safe_strcmp(_Optional const char *s1, _Optional const char *s2)
{
  if (!s1) s1 = "";
  if (!s2) s2 = "";
  return strcmp(&*s1, &*s2);
}

This idiom has the benefit that it is already ‘on the radar’ of implementers (and some programmers) because of an existing rule that neither operator of &* is evaluated. It’s searchable, easy to type (& and * are on adjacent keys), and not too ugly.

Do not underestimate the importance of &* being easy to type! I must have written it thousands of times by now. The alternatives that I considered would have made updating a large existing codebase unbearable.

The way I envisage this working is:

  • All compilers implicitly remove the _Optional qualifier from the type of the pointed-to object in the result of the expressions &*s1 and &*s2.
  • A compiler that does not attempt path-sensitive analysis will not warn about the expressions &*s1 and &*s2, since it cannot tell whether s1 and s2 are null pointers.
  • A compiler that warns about dereferences of pointers to _Optional, in cases where such pointers cannot be proven to be non-null, may warn about the expressions &*s1 and &*s2 if the guarding if statements are removed.

However, this proposal might entail a modification to the description of the address and indirection operators in the C standard:

If the operand [of the unary & operator] is the result of a unary * operator, neither that operator nor the & operator is evaluated and the result is as if both were omitted, except that the constraints on the operators still apply and the result is not an lvalue.

C++ does not currently allow indirection on an operand of type void *. This rule would either need to be aligned with C, or else C++ programmers would need to cast away the qualifier from _Optional void * in some circumstances, rather than using an idiom such as &*.

The devil in the details

I haven’t yet explained how such an expression such as &*s would remove the _Optional qualifier from the type of a pointed-to object.

Whereas a qualifier that applies to a pointer type is naturally removed by dereferencing that pointer, a qualifier (such as _Optional) that applies to a pointed-to object is not:

int *const x;
typeof(&*x) y; // y has type 'int *' not 'int *const'
y = 0;

int b;
int const *a = &b;
typeof(&*a) c; // c has type 'int const *'
*c = 0; // error: read-only variable is not assignable

Consequently, modified semantics are required for the unary * operator, the unary & operator, or both.

It’s tempting to think that the appropriate time to remove a maybe-null qualifier from a pointer is the same moment at which undefined behaviour would ensue if the pointer were null. I prototyped a change to remove the _Optional qualifier from the result of unary *, but found it onerous to add &* everywhere it was necessary to remove the _Optional qualifier from a pointer.

Moreover, many previously simple expressions became unreadable:

  • &(&*s)[index] (instead of &s[index])
  • &(&*s)->member (instead of &s->member)

Whilst it would have been possible to improve readability by using more intermediate variables, that isn’t the frictionless experience I look for in a programming language. (The same consideration applies to reliance on casts in the absence of modified operator semantics.)

The proposed idiom &*s is merely the simplest expression that incorporates a semantic dereference without accessing the pointed-to object. A whole class of similar expressions exist, all of which typically compile to a machine-level instruction to move or add to a register value (rather than a load from memory):

  • &s[0]
  • &0[s] (by definition, E1[E2] is equivalent to (*((E1)+(E2))))
  • &(*s).member
  • &s->member

There is only one way to get the address of an object (excepting arithmetic), whereas there are many ways to dereference a pointer. Therefore, I propose that any _Optional qualifier be implicitly removed from the operand of the unary & operator, rather than modifying the semantics of the unary *, subscript [] and member-access -> operators.

The operand of & is already treated specially, being exempt from conversion from an lvalue to the value stored in the designated object, and from implicit conversion of an array or function type into a pointer. It therefore seems less surprising to add new semantics for & than *.

Another class of expressions that generate an address from a pointer without accessing the pointed-to object are arithmetic expressions in which one operand is a pointer:

  • 1 + s
  • s - 1
  • ++s

None of the above expressions affect the qualifiers of a pointed-to object in the result type: if the type of s is a pointer-to-const then so is the type of s + 1.

Although s + n is equivalent to &s[n] in current code, it does not occur often enough to justify modifying arithmetic operators to remove any _Optional qualifier from a pointed-to object. This also avoids the question of equivalent changes to prefix/postfix operators such as ++ and compound assignments such as +=. The alternative substitution of &*s + n is tolerably readable.

Function pointers

C’s declaration syntax does not permit type qualifiers to be specified as part of a function declaration:

<source>:4:6: error: expected ')' [clang-diagnostic-error]
int (const *f)(int); // pointer to const-qualified function
^

A syntactic way around this limitation is to use an intermediate typedef name:

typedef int func_t(int);
const func_t *f; // pointer to const-qualified function

That doesn’t solve the underlying problem, though. GCC does not warn about such declarations, but Clang does:

<source>:5:1: warning: 'const' qualifier on function type 'func_t' (aka 'int (int)') has unspecified behavior [clang-diagnostic-warning]
const func_t *f; // pointer to const-qualified function
^~~~~~

The C language standard currently says:

If the specification of a function type includes any type qualifiers, the behavior is undefined.

Making this behaviour well-defined (as in C++) would make the language safer, whereas extending the declaration syntax is beyond the scope of my proposal.

Migration of existing code

Functions which consume pointers that can legitimately be null can be changed with no effect on compatibility. For example, void free(_Optional void *) ​ can consume a pointer to an optional​-qualified type, or a pointer to an unqualified type, without casting.

‘Safe’ wrappers for existing functions that produce null pointers could also be written, for example _Optional FILE *safe_fopen(const char *, const char *) would produce a pointer that can only be passed to functions which accept pointers to optional-qualified types.

Requiring implementations to redefine the constant to which the NULL macro expands as ((_Optional void *)0) is unthinkable because it would invalidate all existing code. However, it might be useful to standardize an alternative macro for use in place of NULL. I have not specified such a macro because NULL is not a core part of the language.

Here is an example of one type of change that I made to an existing codebase:

Before

entry_t *old_entries = d->entries;
d->entries = mem_alloc(sizeof(entry_t) * new_size);

if (NULL == d->entries)
{
  d->entries = old_entries;
  return ERROR_OOM;
}

After

_Optional entry_t *new_entries = mem_alloc(sizeof(entry_t) * new_size);

if (NULL == new_entries)
{
  return ERROR_OOM;
}

d->entries = &*new_entries;

This pattern avoids the need to qualify the array pointed to by struct member entries as _Optional, thereby simplifying all other code which uses it. When nullability is part of the type system, more discipline and less constructive ambiguity is required. General-purpose struct types for which pointer nullability depends on specific usage become a liability.

Of course, programmers are free to eschew the new qualifier, just as many do not consider const correctness to be worth their time.

Proposed language extension

  • A new type qualifier, _Optional, indicates that a pointer to a so-qualified type may be null. This does not preclude any other pointer type from being null.
  • Types other than those of a pointed-to object or pointed-to incomplete type shall not be _Optional-qualified in a declaration.
  • The semantics of the unary & operator are modified so that if its operand has type “type” then its result has type “pointer to type”, with the omission of any _Optional qualifier of the pointed-to type.
  • If an operand is a pointer to an _Optional-qualified type and its value cannot be statically proven never to be null, then implementations may generate a warning of any undefined behaviour that would occur if the value were null.
  • A specification of a function type that includes type qualifiers no longer has undefined behaviour. Qualifiers that are not applicable are ignored (as in C++).

The _Optional qualifier is treated like existing qualifiers when determining compatibility between types, and when determining whether a pointer may be implicitly converted to a pointer to a differently-qualified type.

Prototyping in Clang

(All changes in the stack are required)
D142738 Warn if _Optional used at top-level of decl
D142737 Updated getNullabilityAnnotation for checkers
D142736 Add QualType::getNullability for _Optional
D142734 Updated CheckAddressOfOperand
D142733 Add _Optional as fast qualifier

Updating Clang to recognize the _Optional qualifier was not as straightforward as I had hoped because a fixed number of address bits are allocated to store ‘fast’ qualifiers such as const and volatile. Increasing the number of bits used for that purpose required pointed-to objects to be allocated with coarser granularity. I didn’t consider any performance effects of this change significant for a prototype.

Only three methods needed updating to implement special semantics (different from volatile and const) for the new type qualifier:

  • Sema::CheckAddressOfOperand to remove any _Optional qualifier from the type to which this method creates a pointer.
  • GetFullTypeForDeclarator to generate an error for declarations of _Optional objects (except in the context of a typedef, or an array parameter within a function prototype).
  • Sema::ParsedFreeStandingDeclSpec to generate an error for declarations of _Optional types with no declarator.

Having prototyped the above changes, I found that the new qualifier was already useful for finding issues caused by not handling null values defensively. This was exactly what I had hoped, because a new qualifier cannot be justified unless it provides value in the absence of static analysis.

Prototyping in Clang’s static analyser

(All but the last of these address pre-existing issues)
D142744 Re-analyze functions as top-level
D142743 Fix nullability checking of top-level functions
D142742 Generate ImplicitNullDerefEvent from CallAndMessageChecker
D142741 Fix ProgramState::isNull for non-region symbols
D142739 Standalone checker for use of _Optional qualifier

The core of Clang’s static analyser does path-sensitive analysis to determine (for each step in execution of a program) whether a symbol is known to be null, known to be non-null, or could be either.

An existing checker understands _Nullable and _Nonnull attributes, which it uses to maintain extra metadata about whether memory regions supposedly contain nullable or non-null values. This is used when path-sensitive analysis alone cannot determine whether a value is null.

It was simple to update the nullability checker to treat pointers to _Optional types as equivalent to _Nullable. This ensures static analysis of code using my proposed type qualifier is at least as good as analysis using existing attributes, thereby removing one barrier to adoption.

This was not sufficient because the analyser ignores many instances of undefined behaviour. For example, it allows expressions like &self->super when self is null. This latitude is also required because many commonly-used macros such as offsetof and container_of have undefined behaviour. The simplest definition of offsetof incorporates an explicit null pointer dereference:

#define offsetof(st, m) \
  ((size_t)&(((st *)0)->m))

Such expressions must be rejected when applied to pointers to _Optional values, otherwise it would not be safe to remove _Optional from a pointer target by use of my proposed &* idiom (or any equivalent). Effectively, qualifying a type as _Optional must enable an enhanced level of checking for undefined behaviour, which operates partly at a syntactic level rather than solely at the level of simulated memory accesses.

My proposed qualifier does not require the property of nullability to be tracked using extra metadata: if a pointer target’s type is _Optional then that pointer may be null, regardless of its provenance. I therefore created a separate checker for optional values, instead of extending the nullability checker. This requires only checkPreStmt methods to generate a warning unless pointers to _Optional values used in expressions are provably not null.

The ProgramState::isNull method used by pre-existing checkers was good at telling that a pointer definitely is null, but bad at telling that it definitely isn’t null. For example, it returned ‘not sure’ in the following trivial case, which caused a spurious warning from my new checker:

int main(void)
{
  int p;
  int _Optional *q = &p;
  if (q) {
    *q = 0; // spurious warning
  }
  return 0;
}

When analyzing the above program, the statement if (q) does not create a constraint such as range [1, 18446744073709551615] for use in future inferences about the value of q. This is because SimpleConstraintManager::assumeInternal uses SValBuilder::evalCast to convert a pointer type to Boolean, which replaces the condition with 1 (“Non-symbolic memory regions are always true”) if invoked on a pointer (such as q) that lacks an associated symbolic region.

I added code in isNull to do the same evalCast and check for a zero result if invoked on an expression which is not a constant and does not wrap a symbol. This merely aligns the result of isNull with the path already assumed to have been taken.

Clang does not generate ImplicitCastExpr <LValueToRValue> nodes for unary operators (prefix or postfix ++/ --) or compound assignment operators (+=/-=) in its Abstract Syntax Tree. Consequently, I found it hard to verify that implicit removal of the _Optional qualifier from the result of such operations is safe and therefore decided against it.

It would have been possible to change only the semantics of the arithmetic operators + and -, but I didn’t want to break an equivalence specified in 6.5.16.2 of the C standard:

A compound assignment of the form E1 op = E2 differs from the simple assignment expression E1 = E1 op (E2) only in that the lvalue E1 is evaluated only once.

Possible objections

The need to define a typedef name before declaring a pointer to an _Optional function is an undeniable drawback of qualifying the pointed-to type rather than the pointer type. I would argue that code clarity and documentation is often improved by composing complex declarations from type aliases, and that this limitation of the declaration syntax is outweighed by the benefit of regular semantics of actual usage.

Some may struggle to accept a novel syntax for adding nullability information to pointers, given the existence of more prosaic solutions. I can only urge them to consider whether a solution inspired by pointer-to-const is really such a novelty — especially in comparison to the irregular new semantics required when the pointer type itself is qualified.

Others may agree with Stroustrup (in “The Design and Evolution of C++”, 1994) that C’s syntax and semantics are a “known mess” of “perversities”. Nevertheless, I believe that pointer nullability should be added in a way that conforms to long-established C language idioms rather than violating such norms (as C++ references do) in the hope of satisfying users who will never like C anyway.

Let’s make it happen

If you’ve read this far, and you like what you’ve read, then the most important thing is that you don’t do nothing. Changes to programming languages only happen with the collaboration and support of many people.

Here are some practical steps you can take:

  • Spread the word to your C programmer friends and colleagues.
  • Download my prototype stack, build Clang, and try using the new type qualifier in your own projects.
  • Help write tests for Clang’s support of the new qualifier.
  • Help review my patches and get them merged to mainline.
  • Implement support for the new qualifier in other open source compilers. This is essential, because the C standards committee only standardizes existing practice.
  • Contact the national standards body for your country, and tell them that you support this proposal as described in paper n3089, “_Optional: a type qualifier to indicate pointer nullability”. (Unfortunately individuals and companies cannot participate directly in the work of ISO technical committees.)

Thank you for reading.

Acknowledgements

I would like to recognize the following people for their ideas, help, feedback, and encouragement: Mihail Atanassov, Nikunj Patel, Peter Smith, Matthew Clarkson, Mats Petersson, Anastasia Stulova, Raffaele Aquilone, Jonathan Ely, Jim Chaney, James Renwick, Aaron Ballman, Alejandro Colomar, and Elizabeth Bazley.

:cross_mark: Lack of consensus called in this message.

2 Likes
2

There is a verbose and somewhat similar approach:
[RFC] New attribute `annotate_type` (iteration 2)

3

_Optional int *p really does feel wrong. Surely the right thing to do is to just teach Clang to work with int * _Optional p, that is conceptually the right way to do it syntactically. Note there’s also more precedent with MSVC’s __ptr32/__ptr64, which goes after the *.

For hybrid code in CHERI LLVM we have void * __capabilty as a way to change a pointer from being an integer to being a capability, and it’s not safe to drop the __capability like you say is the problem. We manage just fine and emit warnings/errors as requested by the user (and the rather newer and more experimental GCC port by Arm seems to have no issue with this either). This also makes the &* idiom more natural, as the * inherently drops the _Optional.

On the front, yes, putting the qualifiers in there is weird but that is how C says it’s to be done (though interestingly [__ptr32] isn’t supported by either Clang or MSVC, which is most likely just an oversight on MSVC’s part as it’s an obscure corner of the spec).

1 Like
4

_Optional int *p really does feel wrong. Surely the right thing to do is to just teach Clang to work with int * _Optional p , that is conceptually the right way to do it syntactically. Note there’s also more precedent with MSVC’s __ptr32/__ptr64, which goes after the *.

The keywords that go after * decorate types and carry across types. _Optional is (probably) not a part of the type, and is meant to be a keyword-like attribute. The same practice is used in MSVC’s SAL annotations, _In_opt_, _Out_opt_, _Inout_opt_, and _Ret_maybenull_.

See also: Understanding SAL | Microsoft Learn, SAL 2 Function Parameters Annotations - CodeProject

5

If you’re trying to be like optional types in other languages then it definitely is part of the type.

SAL annotations have never made the most sense to me, but my guess is they’re the way they are because of the complexity of trying to do it properly.

1 Like
6

If you’re trying to be like optional types in other languages then it definitely is part of the type.

All right, I finished reading the proposal. The entire proposal is explaining why _Optional should be in the type that qualifies the pointee rather than the pointer, and I’m convinced.

1 Like
7

Would it help if I ask you to consider all null pointers as conceptually pointing to a singleton object which has only one instance? The conceptual singleton object has no address.

Compilers (and software more generally) can in principle do almost anything, which is one of the things that clouds discussions about programming languages. What is harder is to exercise restraint to avoid violating norms.

Most of the arguments I make in the paper are to explain why the syntax that you propose would not be "scenic” in my eyes: because C’s semantics for function calls, compatibility of declarations, assignment and initialization follow more-or-less directly from its syntax.

One of the things that drove me to submit a paper to WG14 (which I never imagined doing in my lifetime) was a deep fear that someone else would propose syntax like int * _Optional p, and worse, it might actually be accepted. Maybe that will happen one day. I hope I’m retired by then. :slight_smile:

I’m more concerned with preserving the day-to-day habits of C programmers, than fixating on syntax. The declaration syntax I proposed is the one necessary to give the right semantics for every other part of the language.

8

There is a verbose and somewhat similar approach:
[RFC] New attribute annotate_type (iteration 2)

That’s interesting, but it looks far too verbose and complex for me, a humble C programmer, to consider using. I just like to declare things as const int *. That’s all. :slight_smile:

1 Like
9

One of my colleagues raised a question that seems worth answered publically:

I expect a question you’re going to be asked a lot is “why _Optional instead of _Mandatory”, and I think munging any rationales for that into the RFC would help your cause

I do wish I’d addressed that directly in my RFC. Basically, the answer is twofold:

  1. Pointers that can be null in function parameters are outnumbered by pointers that cannot be null without provoking undefined behaviour. I believe that a count of parameters to functions in the C standard library would show this, although I haven’t actually done such a count.
  2. Assigning a pointer to a _Mandatory-qualified type to a pointer to an unqualified type would provoke a warning, whereas a pointer to an unqualified type could be assigned to a pointer to a _Mandatory-qualified type without provoking a warning. (This is the opposite of the required semantics for type-safety.)

Again, I believe I’ve proposed the only solution that is workable.

10

No, because a null pointer explicitly does not point to an object. C99 6.3.2.3p3 has the following sentence:

If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function.

Not to mention that, were it to point to an object, you would be permitted to dereference it as a char * or some other character type (and potentially some other type depending on what effective type this supposed object has), which is clearly nonsense.

I don’t see that, I see you stating one point (that casts, whether implicit or explicit, currently are free to drop qualifiers for the type of the value of an expression), and then explaining how all the rules for your proposal would work, but the existence of rules that make your proposal work don’t disprove the existence of rules that make a more principled proposal work where the qualifier qualifies the thing that you’re actually talking about, namely the pointer.

I don’t believe that to be true, based on my experience with the viability of int * qualifier within the context of CHERI where the qualifier cannot be dropped freely.

11

This is just pedantry about terminology. Everyone knows perfectly well that null doesn’t point to an object in the sense that the standard uses that word.

However, its other properties (any two null pointers comparing equal, no other pointer comparing equal to a null pointer) are the same as if it were a pointer to a singleton object.

12

What is your definition of “object” then, if not what C says it is?

13

I hoped it would be clear from my use of the word “conceptual” that I was not using the word to mean what C says it is. Sorry for confusing the issue.

In my analogy, an object is the thing referenced by a pointer value, or the pointee if you prefer.

_Optional means that a pointee may not exist, const means that a pointee may be read-only and volatile means a pointee’s value may change unexpectedly. None of those properties preclude the pointer’s value from changing or mean that it must point to an object of that type (optional, const or volatile).

The fact that an _Optional object may not be able to be copied, initialised, accessed byte-by-byte, etc. doesn’t preclude it from existing as part of the language’s syntax any more than a pointer to void.

In my original idea, I didn’t bother forbidding declarations such as
_Optional int x;
because “may not exist” doesn’t mean “cannot exist”, and fewer rules are preferable. I changed that based on feedback from colleagues and on the forums.

14

I originally intended _Optional to be usable in any part of a
declaration and only banned its use at top level based on
feedback from others.

One interesting thing pointed out by a Redditor is that banning usage
of _Optional at top level could be justified better if so-qualified
values could not exist as the result of dereferencing a pointer, yet I
proposed no change to the semantics of *, -> and [] to remove any
_Optional qualifier from the result type.

I’d be just as happy for pointer dereference operations to implicitly
remove the _Optional qualifier from their result. I already have code to
implement that behaviour for Clang kicking around on a local branch
somewhere, and I wrote all the wording for the alternative proposal
but ultimately didn’t use it. 99% of programs using _Optional would
work without modification with either implementation.

The reasons that I specified that & should remove the _Optional
qualifier instead of *, [] and -> are given in my paper, but they aren’t
incontestable.

15

I’m not aware of any such requirement. Did you find that documented somewhere? It isn’t clear to me that _Nullable semantically differs from your proposed _Optional.

That appears to have come from one person in a code review that appears to have been accepted (and which includes changes to switch from the GNU attribute syntax to the Clang qualifiers). Do you have evidence that these qualifiers have not proved useful? I haven’t verified, but my understanding is that Apple has rather widely deployed them in their framework libraries and diagnoses them in their Xcode IDE.

I think it is interesting that the linked code review is for a change that deploys both _Nullable (10 occurrences) and _Nonnull (130+ occurrences) (_Null_unspecified was not deployed). This is just one data point of course, but it has _Nonnull deployed more than 10 times as often as _Nullable.

Unless I missed it, I don’t see any technical criticism of Clang’s existing qualifiers.

I’m afraid I don’t see a conflict here. Declarations may have the restrict qualifier present; they just aren’t required to.

I don’t find the example code presented compelling. Were you to show a before/after example comparing _Optional with _Nullable, the code would look very much the same; the proposal doesn’t offer anything that we can’t write today.

I find the proposed semantics for &* confusing. I think it would be challenging to explain to someone why, given a variable p declared as _Optional const int *volatile p, the expression &*p yields a type of const int *. The mechanics you desire for _Optional are already in the language.

My take away from this proposal is that we should work towards standardizing _Nullable and _Nonnull assuming there is sufficient motivation to standardize anything at all.

1 Like
16

Thank you for putting together this RFC! For others, something along these same lines was previously proposed by ISO C3X proposal: nonnull qualifier which did not end with strong consensus but did have significant interest from folks in the community.

I have specific comments below, but a summary of my current position is: despite the importance of this topic, I don’t think it should be added to Clang at this time. I disagree with the syntactic choice made to put the qualifier on the pointee rather than the pointer. This design space is littered with attempts to solve the same problem and it’s not clear that this approach is going to be an improvement. The proposal is largely experimental in terms of design and needs a stronger indication that a standards body really believes in this design (in terms of syntactic choices, at the very least) and data demonstrating how this feature catches bugs that cannot be caught by other features in the same space (not just due to missing diagnostics that are possible to implement). The proposal also needs to work for all pointer types without requiring the user to jump through hoops (like using typedefs).

The existing features in this space that Clang supports are, at least:

  • [[gnu::nonnull]]
  • [[gnu::returns_nonnull]]
  • _Nonnull (and friends)
  • static array extents (C only)
  • references (C++ only)

The proposal touches on why the author believes some of these are deficient, but it doesn’t really talk about which ones are repairable in terms of diagnostic behavior or how the new functionality will interact with existing functionality. More examples comparing and contrasting the existing features with the proposed feature could perhaps help make this more clear.

In terms of syntactic choices, as mentioned above, I disagree that the qualifier belongs on the pointee rather than the pointer. Losing the qualifier on lvalue conversion does not seem to be an issue, as after lvalue conversion the loaded value can never be changed. So once you’ve done an lvalue conversion to read the pointer value itself, optionality of the pointee is resolved (so dropping the qualifier should not lose any diagnostic fidelity).

There’s also some details missing about how the feature plays with various languages. In Objective-C, object pointers are a bit special; should this qualifier be usable for them? In C++, if a member function is marked as being _Optional, what are the semantics? Separately, do you expect this qualifier to be mangled as part of a function signature? Can users in C++ overload functions (or in C with __attribute__((overloadable))) or create template specialization based on this qualifier? In C, how does type compatibility work? e.g., are these valid redeclarations? void func(int *i); void func(_Optional int *i) {} (Note, if these are not valid redeclarations, that’s another difference in behavior from other qualifiers in that position and is likely to cause confusion.) What is the ABI of passing an optional pointer and does it differ from the ABI of passing a regular pointer? What happens if one TU declares an extern pointer variable and another TU defines it as being an optional pointer (or vice versa)?

Despite my current feeling of “we should not add this at this time”, I think it’s useful to continue the discussion and iterate on the design to see if we can come to something that we think should be added at some point in the future.

This isn’t fully accurate. As you note below, static array extents are the way in which you signal that a pointer value cannot be null. However, adoption of static array extents in industry has been slow, common tooling often misses helpful diagnostics, the syntax is currently limited to only function interfaces, the syntax is awkward for void pointers, etc. But the syntax and desired semantics do exist in C already today.

The concerns around applying to parameters is a non-issue now that C has [[]] style syntax (and also given that Clang supports __attribute__ syntax on parameters directly). Also, intrusive and verbose are personal style concerns rather than technical issues with the functionality, so “I want this to be a keyword because I don’t like the way attributes look” is not very compelling, especially given how often attributes wind up hidden behind macros.

I think all pointer types already have the property of being able to represent a null pointer value. From that perspective, a type qualifier is unnecessary, that’s just the way the language already works today. However, being able to signal (to readers, to an analyzer, to the optimizer, etc) that a pointer value is expected to never be null is useful. The converse, a type signaling that a pointer is only ever null, is already handled by the nullptr_t type in C2x. But any annotation that says “this pointer might be null” seems like a non-starter because that is how pointers already work, so incremental adoption would be very unlikely.

Hmm, this has nothing to do with pointers though. This is how lvalue conversions work, and it works that way on everything, not just pointers. This is observable via _Generic already today (try getting it to associate with a qualified type, it won’t happen unless the qualifier is on another level of the declarator).

I don’t agree with that assessment. They’re qualifying the pointer to give information about what values the pointer may have, not what objects they point to. So, to me, they’re qualifying exactly what I would expect. And lvalue conversion gives you exactly the properties I’d expect – after obtaining the value of the pointer itself, you no longer need any marking to say whether it’s nonnull or not because you already have the value and it’s too late to ask that question. Basically, lvalue conversion is the point at which you would test whether the pointer could be null or not. It can never change state after lvalue conversion.

I’m not sold on the name given the current proposal. To me, _Optional has very little to do with pointer types and everything to do with values. e.g., I would not want to close the door on being able to write:

  _Optional int get_value(struct something *ptr) {
    if (ptr)
      return ptr->field;
    return _None;
  }

  int main() {
    _Optional int val = get_value(nullptr);
    if (val)
      return *val; // Steals C++ syntax, but the * could be replaced by another syntactic marker
    return EXIT_FAILURE;
  }

this is more in line with Python’s optional functionality, as well as the same idea from C++ (though they solved it with a library feature rather than a language feature).

That said, I think:

  int * _Optional get_ptr_value(struct something *ptr) {
    if (ptr && ptr->field)
      return ptr->field;
    return _None;
  }

is along the same lines of what you’re proposing, except that the optionality is lost on lvalue conversion. But again, once you have an rvalue, the state of the original object cannot change in a way that effects the loaded rvalue and so losing the _Optional qualifier is not harmful.

To be honest, if you designed _Optional to be less about pointers and more about values in general, I think the feature becomes much more compelling.

This is novel in C; no other qualifier works this way. So I’m not convinced that this is going to be less incompatible, confusing, and error-prone than other solutions. Also, const doesn’t always indicate anything about an object’s address (it can in theory for globals, but doesn’t for things like function parameters).

This continues to be an unresolved issue showing the irregularity of the syntax choice of the proposal to qualify something other than the pointer. The fact that you need to jump through hoops for function pointers, specifically, is a serious concern – these are pointers I would expect people would very much want to annotate if they’re plausibly going to be null pointers.

FWIW, the reason for the limitation against qualified function types in C is because functions are not data, but the function pointer is. That’s why you can qualify the pointer but not the function type itself.

2 Likes
17

Ok so the proposal has a lot of details but the core idea is relatively simple, I’ll split it in two parts:

  1. An attribute to annotate pointers that need to always be checked for null before every use.

    • The annotation spreads virally across all pointer assignments, including through function calls.
    • The programmers are only allowed to use them inside an if-statement with a null check, or possibly after an assert.

  2. A much stronger programming model that assumes all unannotated pointers to never be null.

I’m assuming that this proposal doesn’t target C++, because C++ has check-required-pointers at home (check-required-pointers at home: std::optional<std::reference_wrapper<T>>).

I think (1.) is a reasonable proposition. Pointers that deserve such annotation definitely exist. For example, standard function gets() that may unpredictably fail with input/output error would be a great candidate for this. It’s going to be a much harder sell to put these annotations on pointers that may be null predictably (eg., “the result is null when the input is null, otherwise it’s always non-null”), as it’ll start flagging existing code that relies on that contract. But even in this case, I can totally see some projects committing to cleaning up their code according to that programming model.

Solution (1.) can be useful without (2.), as a standalone feature.

I think (2.) is insanely expensive, and the amount of people who can realistically write code like that, is miniscule. It is roughly equivalent to emitting a warning every time an unannotated pointer is assigned null (which is forbidden) or checked for null (given that it can’t be null, so the check doesn’t make sense). That’s an extremely aggressive approach that would flag enormous amounts of perfectly correct code. The adoption cost will in most cases outweigh the benefits.

Aside from annotation burden, code refactoring associated with (2.) may introduce more bugs than it fixes. For example, if a non-null (i.e. unannotated) pointer is initialized with non-trivial control flow:

int *x = NULL;
if (condition()) {
  // expensive computations
  x = ...;
} else {
  // different expensive computations
  x = ...;
}
use_nonnull(x);

Then the initial null-initialization can no longer be performed, so a programmer who tries to minimize code changes caused by keeping the pointer non-null will be tempted to simply drop the initializer:

int *x;
// same code below

Then if one of the branches forgets to initialize the pointer, you’ll end up with a garbage pointer that’s much more terrifying than a null pointer.

Separately, while it may be possible to annotate all your project code, I don’t see how you can easily extend this contract on third-party code that you’re using. Sometimes you really need to pass a null pointer into a third-party library function (even into a C standard library function) and you may be in a situation when adding such annotation into that library may be problematic.

So I strongly suspect that this approach will need massive quality of life improvements which will ultimately make it much more in line with existing solutions, which were the root cause of your frustration in the first place. There’s good reasons why existing solutions don’t simply do what you propose.

So that’s my initial opinion. Let me make a few other less important points here and there.

Well, if it was a property of the pointed-to object, then it would also make sense to write the same with zero stars:

const int i; // i is an int that _is_ stored in read-only memory
volatile int j; // j is an int that _is_ stored in shared memory
_Optional int k; // k is an int for which no storage is allocated???

Given that this doesn’t make sense for your _Optional qualifier, it sounds to me as if you just introduced a different way to spell a pointer qualifier (“let’s put it before the *, not after the *”) without any substantial differences to semantics.

You probably already noticed that but for the other readers, worth bringing up that this is the feature available in clang itself under the clang --analyze flag (aka Clang Static Analyzer), which gcc -fanalyze tries to mimic, and clang-tidy simply imports as-is. This check is also ancient in our case, it’s much older than the nullability annotations _Nonnull and _Nullable (which suggests that you have a typo in your example, it has to still be __attribute__((nonnull)), godbolt).

Well, no, it’s a property of a given group of pointers to the object. Just because an object has many pointers point to it, i.e. generally participates in aliasing, doesn’t mean you can’t pass it as an argument to

memcpy(void *restrict dst, const void *restrict src, size_t n);

It only becomes a problem when both pointers passed to memcpy() point to the same object. So, just these specific pointers, regardless of every other pointer in the program.

First of all, I don’t think this specific problem even needs an in-code solution at all. Instead, the analysis that causes the warning to appear near strcmp can be made smart enough to recognize that s1 and s2 are never null at this point. You need such smarts in the analysis anyway, to cover another very important case:

int foo(_Optional const char *s1, _Optional const char *s2)
{
  if (...) {
    return 0;
  }
  assert(s1);
  assert(s2);
  return strcmp(s1, s2);
}

This is a relatively easy flow-sensitive analysis that you can conduct over Clang CFG, or ClangIR whenever that becomes available. Such analysis would provide very good quality of life. Of course you can go for a purely syntactic solution instead, which will have cleaner rules but be worse in terms of quality of life.

Now, obviously, sometimes you really need a “force-unwrap” operator to indicate that you’re sure the pointer can’t be null here. In this case “easy to type” isn’t necessarily valuable; say, Rust chose the syntax .unwrap() which is designed to catch the eye, be easy to notice and audit. On the other hand, Swift uses ! which is, yes, easier to type. So what you can do is a compiler builtin:

_Optional int *x = ...;
int *y = __builtin_unwrap_optional(x);

Then if users want something fancier, they can have a macro, which can even include a debug-mode check:

#define unwrap(x)                   \
({                                  \
  __typeof(x) __tmp = x;            \
  assert(__tmp);                    \
  __builtin_unwrap_optional(__tmp); \
})

_Optional int *x = ...;
int *y = unwrap(x);

The assert is very useful because it’ll diagnose the problem at the moment of unwrap, not at the moment of dereference, which may happen much later.

So like I said in the other thread, I think this attribute doesn’t need to be checked by the static analyzer. The contract behind your attribute can be much simpler, fully resolved with either purely syntactic analysis or with very basic flow-sensitive analysis.

The static analyzer can take advantage of it. You can introduce a warning about any unchecked dereference of the _Optional pointer. The easiest way to introduce such warning is to perform a state split every time the pointer is encountered: in one state the pointer is null, in the other state it’s non-null. Then the null case simply becomes a path that the analyzer has to explore.

However, again, if you simply implement a simpler syntactic or flow-sensitive warning in the compiler, the analyzer work becomes redundant, because the problem is entirely defined away by the compiler warning.

Maybe there’s still room in the analyzer to warn about invalid force-unwraps, but most of such warnings would be about potential execution paths that the developer has just explicitly said aren’t there, aka false positives. Same reason we wouldn’t warn about asserts that fail on certain execution paths: the developer believes this execution path doesn’t exist, and even if it does, there’s already dynamic analysis (i.e. assert()) for this exact problem.

18

I would like to see a new type qualifier to indicate nullability.

At the moment I put attribute((nonnull)); on the end of functions (I don’t number usually, so that I start by default checking all are valid.

I compile using GCC -fsanitizer and null,returns-nonnull-attribute,address

-Wnonnull
-Wconversion-null
-Wno-nonnull-compare

19

Ok so the proposal has a lot of details but the core idea is relatively simple

Thank goodness someone noticed! I don’t think I could possibly have made a simpler proposal.

I’ll split it in two parts:
An attribute to annotate pointers that need to always be checked for null before every use.
The annotation spreads virally across all pointer assignments, including through function calls.

Yes, exactly like const.

The programmers are only allowed to use them inside an if-statement with a null check, or possibly after an assert.

Assertions mean different things in different contexts.

Conventionally, code to check for assertion failure isn’t included in release build configurations, so I’d expect assertions to be ignored for static analysis purposes (since they don’t protect anything). That could be done using macros.

Conversely, if assertions aren’t omitted (for example in test code) then I’d expect the static analyzer to recognize the longjmp, abort or equivalent that they wrap and treat following pointer dereferences as properly guarded.

A much stronger programming model that assumes all unannotated pointers to never be null.

People may assume that unannotated pointers are never null; the compiler must not.

I think a lot of people already do assume that pointers are not null (based on comments like “I love null pointers - they are guaranteed to generate a SEGFAULT!”)

I make the same assumption in most user space code except that I’ve been Stockholm-syndromed into adding

XXXX_ASSERT_POINTER(x);
XXXX_ASSERT_POINTER(y);
XXXX_ASSERT_POINTER(z);

on entry to every function with pointer arguments. I assume that most normal people don’t do that – either because they don’t know or care what undefined behaviour is, or because life is too short.

It’s going to be a much harder sell to put these annotations on pointers that may be null predictably (eg., “the result is null when the input is null, otherwise it’s always non-null”), as it’ll start flagging existing code that relies on that contract. But even in this case, I can totally see some projects committing to cleaning up their code according to that programming model.

Yes, this is the value proposition: the human writes more defensive code, and in return the compiler helps the human not to make stupid mistakes. Still, it’s entirely up to the individual to decide which functions use the new qualifier.

Solution (1.) can be useful without (2.), as a standalone feature.

I agree. Thank you.

I think (2.) is insanely expensive, and the amount of people who can realistically write code like that, is miniscule. It is roughly equivalent to emitting a warning every time an unannotated pointer is assigned null (which is forbidden) or checked for null (given that it can’t be null, so the check doesn’t make sense).

That’s why it wasn’t part of my proposition, although a surprising number of people have been telling me that (1.) is useless without (2.). I do think that if (1.) is adopted then eventually, someone will implement (2.), but I’d expect it to be opt-in.

My increasing opinion of -Wsign-conversion is that it is too noisy to be useful in a language like C that clearly wasn’t designed for it, and that it often makes existing code worse by forcing programmers to change working code (maybe breaking it in the progress), often by adding redundant casts which make the code less readable and sacrifice type-safety. Still, a lot of people seem to like it.

Then the initial null-initialization can no longer be performed, so a programmer who tries to minimize code changes caused by keeping the pointer non-null will be tempted to simply drop the initializer:

…

Then if one of the branches forgets to initialize the pointer, you’ll end up with a garbage pointer that’s much more terrifying than a null pointer.

I already see this behaviour as a result of warnings about values which are initialized but never used. I can’t remember whether they are generated by Coverity, a compiler, or some other tool. I think this is a case of “pick your poison”. If you use a tool which produces such warnings, and choose to fix them in that way, you’d better be sure it also warns you about garbage values.

Separately, while it may be possible to annotate all your project code, I don’t see how you can easily extend this contract on third-party code that you’re using.

The project that I used for prototyping uses very little third-party code. There’s nothing stopping you from passing a null pointer constant into third-party code directly, because I deliberately did not respecify null. In other cases, a wrapper might be required:

Here’s a wrapper for free which works for C or C++:

static inline void my_free(_Optional void *ptr)
{
	free((void *)ptr);
}

Here’s a nicer (albeit less efficient, but probably not measurably) wrapper which would be sufficient for C:

static inline void my_free(_Optional void *ptr)
{
	if (ptr)
	{
		free(&*ptr);
	}
}

You might find this horrifying, but it is one small function definition in a huge project, in which all the rest of the code can benefit from tracking null pointers as part of the type.

That’s enough for one reply.

20

You’re aware free is defined to work on NULL pointers?..