-O2 Over-optimization

Hi, I find a aggressive optimization by clang.

#include <stdio.h>
#include <stdlib.h>

int main()
  long size = 1024 * 535975264L;
  printf("malloc size:%lu KB\n", size/1024);

  char *p = (char *)malloc(size);

  if (p == NULL) {
    printf("malloc failed!\n");
  } else {
    printf("malloc success!\n");
  return 0;

PS:Assume that CommitLimit < 531853152 kB and malloc fails.

  1. Clang -O0 test.c, the output is malloc failed!

  2. Clang -O2 test.c, the output is malloc success!

I disassemble it and find that the malloc instruction is Disappeared!

main:                                   # @main
        push    rax
        lea     rdi, [rip + .L.str]
        mov     esi, 535975264
        xor     eax, eax
        call    printf@PLT
        lea     rdi, [rip + .Lstr]
        call    puts@PLT
        xor     eax, eax
        pop     rcx
        .asciz  "malloc size:%lu KB\n"

        .asciz  "malloc success!"

However, the behaviour of GCC is the same whether it’s O0 or O2. (look at this Compiler Explorer)

Is the behavior of clang -O2 reasonable?

Yes. What part of the specification would make you think otherwise?

No specification, I’m just surprised that the Clang 02 optimization can changes the results of the program. And GCC doesn’t.

This is indeed a bit surprising; the justification for the transform is that the code doesn’t access the allocated memory, so Clang assumes that it can replace the malloc call with a version that always succeeds.

I think passing -fno-builtin is intended to force clang to consider all these calls as “library functions” instead of internal to the compiler.