Code coverage of llvm, clang, lldb & polly

Subprojects LLDB
#1

Hello,

Plugged in the Jenkins instance used to build Debian & Ubuntu nightly
package, I configured a code coverage tool (lcov) with all the tests of
these projects.
Results are published here:
http://buildd-clang.debian.net/coverage/

This URL is updated twice a day (it might change since it is taking a
long time).

For those who would like to have their own setup, with gcc, use the
following declaration:
CXXFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS="-coverage -lgcov"

Rebuild everything, launch the tests and run:

REPORT=reports/scilab-code-coverage.info
BUILD_DIR=reports/
mkdir -p $BUILD_DIR
lcov --directory $BUILD_DIR --capture --ignore-errors source
--output-file $REPORT
lcov --remove $REPORT "/usr*" -o $REPORT
lcov --remove $REPORT "$BUILD_DIR/*" -o $REPORT
genhtml -o reports/coverage --show-details --highlight --legend $REPORT

Cheers,
Sylvestre

#2

This is awesome, thanks!

A problem that is quickly visible is that we have 0 tests for the pbqp
register allocator :frowning:

#3

Thanks for putting together the report, Sylvestre,

Do you have any indication that lcov/gcov has any support for dynamically loaded shared libraries? Note that much of lldb consists of plug-ins that are platform or architecture specific, and these are on the critical path for coverage. For instance, the report shows 0% coverage for many directories in lldb/source like /Expression and /DataFormatters, whereas the current test suite should give us something...

- Ashok

#4

Polly is run as a shared object and I can see code coverage for Polly. So there seems to be at least some support for shared libraries.

Cheers,
Tobias

#5

Hello,

As Tobias said, it works with shared libraries. If the plugins does not
show, it might be related to the fact that some lldb tests are failing.
They might be my fault. I have to dig.

python
/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/tools/lldb/test/dosep.ty
-o "--executable
/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/build-llvm/Release/bin/lldb
-q -s lldb-test-traces -u CXXFLAGS -u CFLAGS -C x86_64-linux-gnu-gcc"
Traceback (most recent call last):
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/dotest.py",
line 1124, in <module>
    os.path.walk(testdir, visit, 'Test')
  File "/usr/lib/python2.7/posixpath.py", line 238, in walk
    func(arg, top, names)
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/dotest.py",
line 1039, in visit
    suite.addTests(unittest2.defaultTestLoader.loadTestsFromName(base))
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/unittest2/loader.py",
line 111, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/lang/objc/objc-dynamic-value/TestObjCDynamicValue.py",
line 8, in <module>
    import lldb, lldbutil
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/build-llvm/Release/lib/python2.7/site-packages/lldb/__init__.py",
line 50, in <module>
    _lldb = swig_import_helper()
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/build-llvm/Release/lib/python2.7/site-packages/lldb/__init__.py",
line 46, in swig_import_helper
    _mod = imp.load_module('_lldb', fp, pathname, description)
ImportError: libLLVM-3.4.so.1: cannot open shared object file: No such
file or directory
Traceback (most recent call last):
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/dotest.py",
line 1124, in <module>
    os.path.walk(testdir, visit, 'Test')
  File "/usr/lib/python2.7/posixpath.py", line 238, in walk
    func(arg, top, names)
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/dotest.py",
line 1039, in visit
    suite.addTests(unittest2.defaultTestLoader.loadTestsFromName(base))
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/unittest2/loader.py",
line 111, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/lldb/test/lang/objc/blocks/TestObjCIvarsInBlocks.py",
line 5, in <module>
    import lldb
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/build-llvm/Release/lib/python2.7/site-packages/lldb/__init__.py",
line 50, in <module>
    _lldb = swig_import_helper()
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182719/build-llvm/Release/lib/python2.7/site-packages/lldb/__init__.py",
line 46, in swig_import_helper
    _mod = imp.load_module('_lldb', fp, pathname, description)
ImportError: libLLVM-3.4.so.1: cannot open shared object file: No such
file or directory

Full log:
http://llvm-jenkins.debian.net/job/llvm-toolchain-codecoverage-binaries/architecture=amd64,distribution=unstable/71/consoleFull

Cheers,
Sylvestre

#6

I just looked again at the polly numbers and it seems they are not shown any more. Did anything change recently?

Tobias

#7

Did you change anything lately on the test with the autotools ?

cd build-llvm/ && /usr/bin/make polly-test -C tools/polly/test/ || true
make[2]: Entering directory `/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182757/build-llvm/tools/polly/test'
make[2]: *** No rule to make target `polly-test'. Stop.

Full log:
http://llvm-jenkins.debian.net/job/llvm-toolchain-codecoverage-binaries/architecture=amd64,distribution=unstable/74/consoleFull
(it is quite big)

Sylvestre

#8

The polly-test target was recently renamed to check-polly to follow the
convention in other llvm projects:

http://llvm.org/viewvc/llvm-project?view=revision&revision=182171

-- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted
by The Linux Foundation

#9

Thanks for the information, that fixed the issue:
http://buildd-clang.debian.net/coverage/

Sylvestre

#10

Great. Thanks.

Tobias

#11

Hi all, I have a fix for the problem below.

Sylvestre, can you apply the attached to the 'rules' file in your
repository?

Thanks,
Dan

debian_rules_fix_lldb_tests.patch (723 Bytes)

#12

Thanks! It fixes our problem. However, there is an other one but I think
it might be caused by lldb itself here.

ython
/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182916/tools/lldb/test/dosep.ty
-o "--executable
/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182916/build-llvm/Release/bin/lldb
-q -s lldb-test-traces -u CXXFLAGS -u CFLAGS -C x86_64-linux-gnu-gcc"
Traceback (most recent call last):
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182916/lldb/test/dotest.py",
line 1210, in <module>
    print >> f, "Command invoked: %s\n" % getMyCommandLine()
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182916/lldb/test/dotest.py",
line 1074, in getMyCommandLine
    ps = subprocess.Popen(['ps', '-o', "command=CMD", str(os.getpid())],
stdout=subprocess.PIPE).communicate()[0]
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Full log:
http://llvm-jenkins.debian.net/job/llvm-toolchain-codecoverage-binaries/architecture=amd64,distribution=unstable/79/consoleFull

Sylvestre

#13

Indeed this looks like 'ps' utility is not being found.. Weird, but it may
be caused by not specifying a full path. I'll try a fix ASAP.

#14

Hi Sylvestre, I committed a potential fix for the 'ps' issue in 182965.

Out of curiosity, is the code-coverage run happening on a different
machine than the normal testing runs? For example, I'm seeing:

http://llvm-jenkins.debian.net/job/llvm-toolchain-quantal-binaries/architec
ture=amd64,distribution=quantal/166/consoleText

Doesn't have any problems finding 'ps' when running the tests. It seems
odd that 'ps' is not being found..could it maybe be in a (different)
chroot environment where ps is not available for some reason?

Cheers,
Dan

#15

Hi Sylvestre, I committed a potential fix for the 'ps' issue in 182965.

It fails with:
Traceback (most recent call last):
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182989/lldb/test/dotest.py",
line 1210, in <module>
    print >> f, "Command invoked: %s\n" % getMyCommandLine()
  File
"/tmp/buildd/llvm-toolchain-snapshot-3.4~svn182989/lldb/test/dotest.py",
line 1074, in getMyCommandLine
    ps = subprocess.Popen([which('ps'), '-o', "command=CMD",
str(os.getpid())], stdout=subprocess.PIPE).communicate()[0]
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
AttributeError: 'NoneType' object has no attribute 'rfind'

Out of curiosity, is the code-coverage run happening on a different
machine than the normal testing runs? For example, I'm seeing:

http://llvm-jenkins.debian.net/job/llvm-toolchain-quantal-binaries/architec
ture=amd64,distribution=quantal/166/consoleText

Doesn't have any problems finding 'ps' when running the tests. It seems
odd that 'ps' is not being found..could it maybe be in a (different)
chroot environment where ps is not available for some reason?

Besides the code coverage build flags, there is no difference.
However, all the system is different. That means that the Python under
quantal is probably different from the one in Debian Unstable.

Cheers,
Sylvestre

#16

Thanks for the update. It's clear that ps is not being installed in the
chroot environments. I checked the debian/control file and there's no
build depends on it; try the attached patch that adds a Build-Depends on
procps package.

Thanks,
Dan

build_depend_procps.patch (516 Bytes)

#17

Oh. I would have say that ps was installed by default in a chroot :slight_smile:
Maybe it was(is?) the case for Ubuntu.
Anyway, I just applied your patch and relaunched a build (but note that
the code coverage takes
between 4h to 5h30)

Thanks :slight_smile:
Sylvestre

#18

Yeah, it's somewhat surprising that 'ps' is not being found..it doesn't
seem to be on the PATH. In any case, no rush, I will continue looking at
some of the other test failures that are happening only on Debian builds
in the mean time..

Thanks,
Dan

#19

http://buildd-clang.debian.net/coverage/
Your patch fixed the issue!
Thanks to it, the LLVM toolchain is close to the 80 % lines test
coverage (which is excellent).

Sylvestre

#20

Note that line-based coverage is far from being precise. To get more
realistic estimation of coverage, you need to use coverage tool that
provides decision and condition coverage, e.g. TestCocoon.