From: "Jonathan Roelofs" <jonathan@codesourcery.com>
To: "Hal Finkel" <hfinkel@anl.gov>
Cc: "Eric Fiselier" <eric@efcs.ca>, "clang developer list" <cfe-dev@lists.llvm.org>, "Chandler Carruth"
<chandlerc@gmail.com>, "Kostya Serebryany" <kcc@google.com>, "Evgenii Stepanov" <eugenis@google.com>
Sent: Monday, August 15, 2016 9:24:17 AM
Subject: Re: [cfe-dev] Making MSAN Easier to Use: Providing a Sanitized Libc++
>> From: "Jonathan Roelofs via cfe-dev" <cfe-dev@lists.llvm.org> To:
>> "Eric Fiselier" <eric@efcs.ca>, "clang developer list"
>> <cfe-dev@lists.llvm.org>, "Chandler Carruth"
>> <chandlerc@gmail.com>,
>> "Kostya Serebryany" <kcc@google.com>, "Evgenii Stepanov"
>> <eugenis@google.com> Sent: Sunday, August 14, 2016 7:07:00 PM
>> Subject: Re: [cfe-dev] Making MSAN Easier to Use: Providing a
>> Sanitized Libc++
>>
>>
>>
>>> Sanitizers such as MSAN require the entire program to be
>>> instrumented, anything less leads to plenty of false positives.
>>> Unfortunately this can be difficult to achieve, especially for
>>> the C and C++ standard libraries. To work around this the
>>> sanitizers provide interceptors for common C functions, but the
>>> same solution doesn't work as well for the C++ STL. Instead users
>>> are forced to manually build and link a custom sanitized libc++.
>>> This is a huge PITA and I would like to improve the situation,
>>> not just for MSAN but all sanitizers. I'm working on a proposal
>>> to change this. The basis of my proposal is:
>>>
>>> Clang should install/provide multiple sanitized versions of
>>> Libc++ and a mechanism to easily link them, as if they were a
>>> Compiler-RT runtime.
>>>
>>> The goal of this proposal is:
>>>
>>> (1) Greatly reduce the number of false positives caused by using
>>> an un-sanitized STL. (2) Allow sanitizers to catch user bugs that
>>> occur within the STL library, not just its headers.
>>>
>>> The basic steps I would like to take to achieve this are:
>>>
>>> (1) Teach the compiler-rt CMake how to build and install each
>>> sanitized libc++ version along side its other runtimes. (2) Add
>>> options to the Clang driver to support linking/using these
>>> libraries.
>>>
>>> I think this proposal is likely to be contentious, so I would
>>> like to focus on the details it. Once I have some feedback on
>>> these details I'll put together a formal proposal, including a
>>> plan for implementing it. The details I would like input on are:
>>>
>>> (A) What kind and how many sanitized versions of libc++ should
>>> we provide?
>>> ---------------------------------------------------------------------------------------------------------------
>>>
>>>
>>>
I think the minimum set would be Address (which includes Leak),
>>> Memory (With origin tracking?), Thread, and Undefined. Once we
>>> get into combinations of sanitizers things get more complicated.
>>> What other sanitizer combinations should we provide?
>>>
>>> (B) How should we handle UBSAN?
>>> ---------------------------------------------------
>>>
>>> UBSAN is really just a collection of sanitizers and providing
>>> sanitized versions of libc++ for every possible configuration is
>>> out of the question. Instead we should figure out what subset of
>>> UBSAN checks we want to enable in sanitized libc++ versions. I
>>> suspect we want to disable the following checks.
>>>
>>> * -fsanitize=vptr * -fsanitize=function *
>>> -fsanitize=float-divide-by-zero
>>>
>>> Additionally UBSAN can be combined with every other sanitizer
>>> group (ie Address, Memory, Thread). Do we want to provide a
>>> combination of UBSAN on/off for every group, or can we simply
>>> provide an over-sanitized version with UBSAN on?
>>>
>>> (C) How should the Clang driver expose the sanitized libraries
>>> to the users?
>>> -------------------------------------------------------------------------------------------------------------
>>>
>>>
>>>
I would like to propose the driver option '-fsanitize-stdlib' and
>>> '-fsanitize-stdlib=<sanitizer>'. The first version deduces the
>>> best sanitized version to use, the second allows it to be
>>> explicitly specified.
>>>
>>> A couple of other options are:
>>>
>>> * -fsanitize=foo: Implicitly turn on a sanitized STL. Clang
>>> deduces which version. * -stdlib=libc++-<sanitizer>: Explicitly
>>> turn on and choose a sanitized STL.
>>>
>>> (D) Should sanitized libc++ versions override libc++.so?
>>> -------------------------------------------------------------------------------------------
>>>
>>>
>>>
For example, what happens when a program links to both a sanitized
>>> and non-sanitized libc++ version? Does the sanitized version
>>> replace the non-sanitized version, or should both versions be
>>> loaded into the program?
>>>
>>> Essentially I'm asking if the sanitized versions of libc++
>>> should have the "soname" libc++ so they can replace non-sanitized
>>> version, or if they should have a different "soname" so the
>>> linker treats them as a separate library.
>>>
>>> I haven't looked into the consequences of either approach in
>>> depth, but any input is appreciated.
>>
>> In a sense, these are /just/ multilibs, so my inclination would be
>> to make all the soname's the same, and just stick them in
>> appropriately named subfolders relative to their normal location.
>
> I'm not sure that's true; there's no property of the environment
> that
> determines which library path you need. As a practical matter, I
> can't set $PLATFORM and/or $LIB in my rpath and have ld.so do the
> right thing in this context. Moreover, it is really a property of
> how
> you compiled, so I think using an alternate library name is
> natural.
Multilibs solve exactly the problem of "it's a property of how you
compiled". The thing that's subtly different here is that the usual
thing that people do with multilibs is to provide ABI incompatible
versions of the same library (which are made incompatible via
compiler
flags, -msoft-float, for example), whereas these libraries just so
happen to be ABI compatible with their non-instrumented variants.
I'm not sure I understand what you're saying about $PLATFORM and
$LIB,
but I /think/ it's a red herring: the compiler takes care of adding
in
the multilib suffixes where appropriate, so shouldn't the answer to
"which library do I stick in the rpath?" include said suffix (when
compiled with Eric's proposed flag)?
I'm not sure what color herring it is 
-- I'm trying to understand the system you're proposing:
1. User A compiles/installs Clang/LLVM/libc++ on system A in /local/clang, and so we get a /local/clang/lib/libc++.so and a /local/clang/lib/msan/libc++.so. User A compiles a program, foo, with msan enabled, and foo gets an rpath of /local/clang/lib/msan. User A also compiles another program, prod, without any sanitizers, and those get an rpath of /local/clang/lib.
2. User B compiles/installs Clang/LLVM/libc++ on system B in /soft/clang, and so we get a /soft/clang/lib/libc++.so and a /soft/clang/lib/msan/libc++.so. User A sends User B the executables foo and prod. Those executables have rpaths with /local/clang/..., but those don't help User B. User B has an environment with LD_LIBRARY_PATH=/soft/clang/lib so that the executables compiled by User A will run.
3. User B has no good option, because if LD_LIBRARY_PATH is set to /soft/clang/lib, then prod will behave as expected (i.e. not be sanitized), but foo will not. If LD_LIBRARY_PATH is set to /soft/clang/lib/msan, then foo will be sanitized as expected, but prod will run slower than usual.
4. User B compiles programs to send to User A. User A then sets LD_LIBRARY_PATH to /local/clang/lib. User A has the same problem as User B, and moreover, if User A compiles using -W,--enable-new-dtags, then the linker will use DT_RUNPATH (instead of, or in addition to, DT_RPATH; effect is the same), which is the recommended default on many systems, the rpath scheme won't even work for User A on User A's own executables (because LD_LIBRARY_PATH overrides DT_RUNPATH).
There are a few things, other than pure directory paths, that can appear in, or otherwise affect, LD_LIBRARY_PATH and DT_RPATH/DT_RUNPATH, but I don't think any of them help us here:
1. Pseudo variables $ORIGIN, $LIB and $PLATFORM - These are expanded by ld.so based on properties of the current execution environment (e.g. whether you're loading a 32-bit or 64-bit executable, the hardware architecture).
2. Hardware-capability strings - There are a fixed set of hardware capabilities, such as sse, sse2, altivec, etc. that are appended to the directory name to form alternate search paths.
3. The multilib suffix. This, AFAIK, is baked into the dynamic loader. The path to the loader itself has the multilib suffix, and that's specified in PT_INTERP.
Unfortunately, I don't think that any of these help us.
-Hal