sanitize=cfi-icall in real life application on x86 and ARM

Project Infrastructure LLVM Dev List Archives
#1

Hi,

After going through small samples showing that "icall" is working, I wanted to give a try with more complex stuff.
I decided to use nginx. As first, I tried with x86 platform. It went quite smoothly, there is one runtime error reported by CFI, I fixed that the server was working fine, without any issues.
Then I switched to the destination platform, ARMv7 based. This time nginx with cfi-icall enabled flag became completely unusable. Worker processes crashed just after spawning them.
I checked with different "sanitize" flags - like SafeStack, etc. - in all these cases it was fine. So only with "icall" there were problems.
Started debugging it:
===GDB OUTPUT===
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0007e2cc in ngx_execute_proc.cfi ()

(gdb) backtrace
#0 0x0007e2cc in ngx_execute_proc.cfi ()
#1 0x0007e05c in ngx_spawn_process ()
#2 0x0007f8f0 in ngx_master_process_cycle.cfi ()
#3 0x000585dc in main ()

#2

After more debugging I found that the problem is with passing a function pointer as parameter and calling it.
So in case of nginx it looks following:

ngx_start_worker_processes(ngx_cycle_t *cycle, ngx_int_t n, ngx_int_t type)
{
    ngx_int_t i;
    ngx_channel_t ch;

    ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "start worker processes");

    ngx_memzero(&ch, sizeof(ngx_channel_t));

    ch.command = NGX_CMD_OPEN_CHANNEL;

    for (i = 0; i < n; i++) {

        ngx_spawn_process(cycle, ngx_worker_process_cycle,
                          (void *) (intptr_t) i, "worker process", type);

        ch.pid = ngx_processes[ngx_process_slot].pid;
        ch.slot = ngx_process_slot;
        ch.fd = ngx_processes[ngx_process_slot].channel[0];

        ngx_pass_open_channel(cycle, &ch);
    }
}

ngx_pid_t
ngx_spawn_process(ngx_cycle_t *cycle, ngx_spawn_proc_pt proc, void *data,
    char *name, ngx_int_t respawn)
{
      ...
      proc(cycle, data);
      ...
}

inside nginx_spawn_process instead of calling ngx_worker_process_cycle, there is another function invoked - ngx_execute_proc, which expects different data (definitely not 0,1,2,3,...). As results nginx crashes.
And again, this happens ONLY with CFI (icall) enabled.

Could you please help?

regards,
Michal