run tests with ASAN (dotest.py)

Hi,

I'd like to run the tests, when LLVM, Clang and LLDB are all built with ASAN.
I am using release_60 version of LLDB.
The unittests just run fine, but with `dotest.py`, there is an error.
Maybe the LLDB build does not respect the global -DLLVM_USE_SANITIZER
flag of cmake ?

Any help would be appreciated,
Thanks,
Gabor

/var/jenkins_home/workspace/ctu_pipeline/llvm/tools/lldb/test/dotest.py
--executable /var/jenkins_home/workspace/ctu_pipeline/build/bin/lldb
/var/jenkins_home/workspace/ctu_pipeline/llvm/tools/lldb/packages/Python/lldbsuite/test/
-v --excluded ./lldb_test_exclude
lldb version 6.0.0 (https://github.com/llvm-mirror/lldb.git revision
b6df24ff1b258b18041161b8f32ac316a3b5d8d9)
  clang revision 64eed461cdd3705e7bc1ccc95df9858f7fe216a8
  llvm revision 089d4c0c490687db6c75f1d074e99c4d42936a50
['/var/jenkins_home/workspace/ctu_pipeline/llvm/tools/lldb/test/dotest.py',
'--executable',
'/var/jenkins_home/workspace/ctu_pipeline/build/bin/lldb',
'/var/jenkins_home/workspace/ctu_pipeline/llvm/tools/lldb/packages/Python/lldbsuite/test/',
'-v', '--excluded', './lldb_test_exclude']
LLDB library dir: /var/jenkins_home/workspace/ctu_pipeline/build/bin
LLDB import library dir: /var/jenkins_home/workspace/ctu_pipeline/build/bin
The 'lldb-mi' executable cannot be located.  The lldb-mi tests can not
be run as a result.
Traceback (most recent call last):
  File "/var/jenkins_home/workspace/ctu_pipeline/llvm/tools/lldb/test/dotest.py",
line 7, in <module>
    lldbsuite.test.run_suite()
  File "/var/jenkins_home/workspace/ctu_pipeline/llvm/tools/lldb/packages/Python/lldbsuite/test/dotest.py",
line 1129, in run_suite
    import lldb
  File "/var/jenkins_home/workspace/ctu_pipeline/build/lib/python2.7/site-packages/lldb/__init__.py",
line 53, in <module>
    _lldb = swig_import_helper()
  File "/var/jenkins_home/workspace/ctu_pipeline/build/lib/python2.7/site-packages/lldb/__init__.py",
line 49, in swig_import_helper
    _mod = imp.load_module('_lldb', fp, pathname, description)
ImportError: /var/jenkins_home/workspace/ctu_pipeline/build/lib/libLLVMDemangle.so.6:
undefined symbol: __asan_option_detect_stack_use_after_return

Hi Gábor,

That symbol appears to be defined in compiler-rt/lib/asan/asan_rtl.cc, so it should be provided by the asan runtime. Are you sure that the runtime expected by the host compiler is being dynamically loaded here? You can check this using ‘ldd’ (IIRC) on Linux or ‘otool -l’ on Darwin. Also, did you take the extra step needed to preload the runtime (LD_PRELOAD/DYLD_INSERT_LIBRARIES)?

best,
vedant

When I looked into this in the past (two years ago), the problem was
that the sanitizer runtimes were expected to be linked into the main
executable. For the dotest tests, the main executable is "python", so
unless you have built an asanified python, you will not have them.

You might be able to get them loaded via some LD_PRELOAD tricks, but I
am not sure if the overall result will be worth the trouble. In
general, the sanitizers expect that your whole process is sanitized,
and they tend to report a lot of false positives if that is not the
case.

Hi Vedant and Pavel,

Thanks for your reply.
I agree that asan may report false positives in case of leak
detection: E.g. library A (instrumented) allocates memory, while
library B (not instrumented by asan) is responsible for deallocation.
However, my goal is to discover memory corruption errors around the
ASTImporter. For that it is enough if all llvm and clang libs are
sanitized, plus liblldb.so too; I don't think that we need to sanitize
python as well.

So what I have discovered so far is that the lldb binary itself
statically links the asan library:

nm -C bin/lldb | ag __asan_op
000000000098cdf0 B __asan_option_detect_stack_use_after_return

In the meanwhile the liblldb.so is not:

nm -C ./lib/liblldb.so | ag __asan_op
                 U __asan_option_detect_stack_use_after_return

liblldb.so is then loaded by SWIG from dotest.py and that causes the error.
With LD_PRELOAD yes I can preload one libasan.so but it is not trivial
to find the compatible ASAN runtime.
When I could find the compatible ASAN runtime then I couldn't get rid
of an annoying error:
```
python dotest.py --executable ~/WORK/llvm2/build/debug_san/bin/lldb
../packages/Python/lldbsuite/test/ -v --excluded ~/lldb_test_exclude
--skip-category dataformatters --skip-category flakey --skip-category
lldb-mi
['dotest.py', '--executable',
'/home/egbomrt/WORK/llvm2/build/debug_san/bin/lldb',
'../packages/Python/lldbsuite/test/', '-v', '--excluded',
'/home/egbomrt/lldb_test_exclude', '--skip-category',
'dataformatters', '--skip-category', 'flakey', '--skip-category',
'lldb-mi']
LLDB library dir: /home/egbomrt/WORK/llvm2/build/debug_san/bin
LLDB import library dir: /home/egbomrt/WORK/llvm2/build/debug_san/bin
==25952==The following global variable is not properly aligned.
==25952==This may happen if another global with the same name
==25952==resides in another non-instrumented module.
==25952==Or the global comes from a C file built w/o -fno-common.
==25952==In either case this is likely an ODR violation bug,
==25952==but AddressSanitizer can not provide more details.

Hi Gábor,

That error is kind of correct. As far as c++ standard is concerned,
this is an ODR violation, as both lldb and liblldb link in a copy of
LLVMSupport.a. However, all LLVM symbols in liblldb have "hidden"
visibility (and we make sure we don't pass around llvm objects on the
SO boundary), so this is not a problem in practice.

I am not sure why ASAN_OPTIONS is not respected here, but you should
be able to get around this error by building with
LLVM_LINK_LLVM_DYLIB=On. This will cause both liblldb and lldb to link
llvm dynamically, so you should have only one llvm copy floating
around.