[ASan][Windows] Interceptor function type not compatible with intercepted function


I triggered a build failure on a Windows-sanitizer by making the sanity checking in ASAN_INTERCEPT_FUNC a bit stricter.
My best guess is that the type of the defined interceptor is not compatible (in C++ typing terms) with the “real” function.

This seems to be the case for the following 2 functions:

CreateThread “no conversion”:

While I’m not an active LLVM dev at the moment, this piqued my interest.

It looks like the interceptor function is trying to return a DWORD from CreateThread, where it should be returning a HANDLE (which is basically a void*, something I’ve exploited in the past for statically checking resource leaks & handle misuse). The C specific handler looks like the same thing, returning int instead of EXCEPTION_DISPOSITION. I bet that’d fix it.

It looks like the parameters of the interceptors were written in more familiar/basic types rather than their official Windows formats. I’ve seen that done in another project to avoid pulling in some of the more obscure Windows headers, although that doesn’t seem to be a problem here. Maybe it was done to avoid noisy casts in the interceptor body, I don’t know. Ideally the fix is to “just” use the correct types, but maybe it’s not that simple. Try it and see what happens.

I think we can get away with forward declaring the types as needed to avoid a hard dependency on windows.h.

It looks like this may have found a real bug, though, since the real CreateThread returns a full pointer (HANDLE) and ours appears to return DWORD, which is too small.