-O2 Over-optimization

IR & Optimizations
1

Hi, I find a aggressive optimization by clang.

#include <stdio.h>
#include <stdlib.h>

int main()
{
  long size = 1024 * 535975264L;
  printf("malloc size:%lu KB\n", size/1024);

  char *p = (char *)malloc(size);

  if (p == NULL) {
    printf("malloc failed!\n");
  } else {
    printf("malloc success!\n");
  }
  free(p);
  return 0;
}

PS:Assume that CommitLimit < 531853152 kB and malloc fails.

  1. Clang -O0 test.c, the output is malloc failed!

  2. Clang -O2 test.c, the output is malloc success!

I disassemble it and find that the malloc instruction is Disappeared!

main:                                   # @main
        push    rax
        lea     rdi, [rip + .L.str]
        mov     esi, 535975264
        xor     eax, eax
        call    printf@PLT
        lea     rdi, [rip + .Lstr]
        call    puts@PLT
        xor     eax, eax
        pop     rcx
        ret
.L.str:
        .asciz  "malloc size:%lu KB\n"

.Lstr:
        .asciz  "malloc success!"

However, the behaviour of GCC is the same whether it’s O0 or O2. (look at this Compiler Explorer)

Is the behavior of clang -O2 reasonable?

2

Yes. What part of the specification would make you think otherwise?

5

No specification, I’m just surprised that the Clang 02 optimization can changes the results of the program. And GCC doesn’t.

6

This is indeed a bit surprising; the justification for the transform is that the code doesn’t access the allocated memory, so Clang assumes that it can replace the malloc call with a version that always succeeds.

7

I think passing -fno-builtin is intended to force clang to consider all these calls as “library functions” instead of internal to the compiler.