clang feature request: integer overflow check in operator new

Dear clang developers,

I have a humble feature request for clang, now that it officially
supports C++: integer overflow check in operator new.

Basically, in code like this:

   int* foo=new int[somevalue];

the compiler does an implicit somevalue*sizeof(int) and passes that
value to operator new. If that multiplication overflows, this is a
security vulnerability. The Microsoft solution is to use the overflow
flag on x86 to set the resulting value to (size_t)-1, which will then
make operator new fail. There are caveats to this, obviously, but it is
an important step to make and it cannot be done without compiler help.

Would it be possible to get clang to do something like this in the code
generator?

That would really help the world be a more secure place in the long run,
and maybe it can even help convince g++ to follow suit. Also, I really
like how you place emphasis on good diagnostic messages in clang, and I
will try to come up with suggestions on what to do even better there.
My biggest hope long-term would be to provide a framework for data flow
analysis using the link time optimization framework. Now finally
compilers have enough context to do that kind of analysis, it is a great
opportunity to do some good. I am thinking of some kind of tainting
warning. And it would also be very helpful if clang supported something
like SAL<http://msdn.microsoft.com/en-us/library/ms235402(VS.80).aspx>

Basically anything that allows me to annotate my library so that using
it incorrectly triggers better warnings would be greatly appreciated.

Thanks,

Felix

This looks like it should be relatively easy to add. Line 444 of CGExprCXX.cpp just needs changing to emit a call to the overflow-checking intrinsic (see EmitOverflowCheckedBinOp() in CGExprScalar.cpp), and then a little extra handling for the case where it overflows. I'm not sure what the effect of passing (size_t)max to new in GNU libstdc++ is - perhaps the best thing would be to bypass the new entirely and have it return 0 in overflow?

David

-- Sent from my IBM 1620

Hi Felix,

Please file a bugzilla for this specic issue, it certainly sounds like
a reasonable addition.

- Daniel

new should throw an exception if allocation fails, unless you use the
nothrow variant that returns NULL.

if you return NULL you may introduce a new vulnerability (null dereference).

Best regards,
--Edwin

Thus spake David Chisnall (theraven@sucs.org):

> I have a humble feature request for clang, now that it officially
> supports C++: integer overflow check in operator new.
>
> Basically, in code like this:
>
> int* foo=new int[somevalue];
>
> the compiler does an implicit somevalue*sizeof(int) and passes that
> value to operator new. If that multiplication overflows, this is a
> security vulnerability. The Microsoft solution is to use the overflow
> flag on x86 to set the resulting value to (size_t)-1, which will then
> make operator new fail. There are caveats to this, obviously, but it is
> an important step to make and it cannot be done without compiler help.
This looks like it should be relatively easy to add. Line 444 of CGExprCXX.cpp just needs changing to emit a call to the overflow-checking intrinsic (see EmitOverflowCheckedBinOp() in CGExprScalar.cpp), and then a little extra handling for the case where it overflows. I'm not sure what the effect of passing (size_t)max to new in GNU libstdc++ is - perhaps the best thing would be to bypass the new entirely and have it return 0 in overflow?

operator new is supposed to throw an exception when it fails to allocate
memory, not return NULL. But you can call operator new in a way that
tells it to return NULL, too.

So some caution is required to get this right :slight_smile:

Thanks,
Felix

C++0x says that the type of that thrown expression is std::bad_array_new_length.

-Howard