Configuring test-suite

Hi,

I'm trying to setup the test-suite and am getting short of running it...

Following the steps here:

http://llvm.org/docs/lnt/quickstart.html

I got as far as getting lnt installed, but when I run the "lnt runtest
nt" it fails:

compilers.py:81: error: unable to determine compiler version:
'clang++': 'clang version 3.2 '

Seems my string doesn't have a "(branch rev)" at the end, ok
hard-coded it to "(branch rev)" and got a new error:

$ sandbox/bin/python sandbox/bin/lnt runtest nt --sandbox sandbox --cc
debug/bin/clang++ --test-suite test-suite
nt.py:1197: note: inferred C++ compiler under test as:
'/home/rengolin/devel/llvm/debug/bin/clang++'
2012-10-26 21:43:48: checking source versions
2012-10-26 21:43:49: scanning for LNT-based test modules
2012-10-26 21:43:49: found 0 LNT-based test modules
2012-10-26 21:43:49: using nickname: 'brubeck__clang_DEV__x86_64'
2012-10-26 21:43:49: starting test in
'/home/rengolin/devel/llvm/sandbox/test-2012-10-26_21-43-48'
2012-10-26 21:43:49: configuring...
2012-10-26 21:43:56: building test-suite tools
nt.py:748: fatal error: unable to build tools, aborting!

Going to the dir where it built, and typing:

$ make -C tools/
make: Entering directory
`/home/rengolin/devel/llvm/sandbox/test-2012-10-26_21-43-48/tools'
make -C /home/rengolin/devel/llvm/sandbox/test-2012-10-26_21-43-48/tools all \
    ORIGINAL_CC="cc" \
    ORIGINAL_CXX="cc"
o timeit-target /home/rengolin/devel/llvm/test-suite/tools/timeit.c -O3
make: o: Command not found
make: [timeit-target] Error 127 (ignored)
E -o TargetConfig.mk -x c
/home/rengolin/devel/llvm/test-suite/tools/TargetConfig.mk.in
make: E: Command not found
make: [TargetConfig.mk] Error 127 (ignored)
...

o? E? Might be environment messup, but I'm a little lost. Does this
work out-of-the box to anyone?

I hit the same problems as you did when I tried it a few weeks ago.
Haven't bothered with it since.

There is a way to run tests without using lnt. You can use the
Makefiles directly in your test-suite build directory. For running
nightly tests, you can do:
make report.html TEST=nightly

That will generate a html file with the test results in.

Amara

[+Daniel, in case he knows what's going on here]

Hi,

I'm trying to setup the test-suite and am getting short of running it...

Following the steps here:

http://llvm.org/docs/lnt/quickstart.html

I got as far as getting lnt installed, but when I run the "lnt runtest
nt" it fails:

compilers.py:81: error: unable to determine compiler version:
'clang++': 'clang version 3.2 '

Fixed LNT to recognize this compiler in
  http://llvm.org/viewvc/llvm-project?view=rev&revision=166842

Seems my string doesn't have a "(branch rev)" at the end, ok
hard-coded it to "(branch rev)" and got a new error:

$ sandbox/bin/python sandbox/bin/lnt runtest nt --sandbox sandbox --cc
debug/bin/clang++ --test-suite test-suite
nt.py:1197: note: inferred C++ compiler under test as:
'/home/rengolin/devel/llvm/debug/bin/clang++'
2012-10-26 21:43:48: checking source versions
2012-10-26 21:43:49: scanning for LNT-based test modules
2012-10-26 21:43:49: found 0 LNT-based test modules
2012-10-26 21:43:49: using nickname: 'brubeck__clang_DEV__x86_64'
2012-10-26 21:43:49: starting test in
'/home/rengolin/devel/llvm/sandbox/test-2012-10-26_21-43-48'
2012-10-26 21:43:49: configuring...
2012-10-26 21:43:56: building test-suite tools
nt.py:748: fatal error: unable to build tools, aborting!

Can you include the output of build-tools.log in that directory, please?

Going to the dir where it built, and typing:

$ make -C tools/
make: Entering directory
`/home/rengolin/devel/llvm/sandbox/test-2012-10-26_21-43-48/tools'
make -C /home/rengolin/devel/llvm/sandbox/test-2012-10-26_21-43-48/tools all \
          ORIGINAL_CC="cc" \
          ORIGINAL_CXX="cc"
o timeit-target /home/rengolin/devel/llvm/test-suite/tools/timeit.c -O3
make: o: Command not found
make: [timeit-target] Error 127 (ignored)
E -o TargetConfig.mk -x c
/home/rengolin/devel/llvm/test-suite/tools/TargetConfig.mk.in
make: E: Command not found
make: [TargetConfig.mk] Error 127 (ignored)
...

o? E? Might be environment messup, but I'm a little lost.

The command you ran is not expected to work, the Makefiles depend on
getting a lot of variables passed in, when run in the way LNT runs
them (that is, without piggy backing off of an LLVM installation).

Does this
work out-of-the box to anyone?

Yup! :slight_smile:

- Daniel

Ah, yes! Relative path on clang++ binary.

I understand why it works to get the version but not here (different paths).

Would be good to have some warning (or even an error) if one passes a
relative path. Or, possibly, save the current path and pass it to the
compiler during the tests. Nothing critical, just for consistency.

It's running now, thanks!

Round 2, fight!

I managed to run many tests (not sure all of them), and some failed,
some didn't. All errors in the 'test' rule were dutifully ignored (I
think). So far so good.

But I had some errors on the 'report' rule, which I wasn't expecting.

make: *** [report.simple.raw.out] Error 2
make: Target `report' not remade because of errors.
/home/rengolin/devel/llvm/test-suite/GenerateReport.pl -csv
/home/rengolin/devel/llvm/test-suite/TEST.simple.report <
report.simple.raw.out > report.simple.csv
/bin/sh: 1: cannot open report.simple.raw.out: No such file
make: *** [report.simple.csv] Error 2

And that stops the process.

$ make report

Didn't work, but I guess that's deeply nested and probably also need
many environment config to be just right. Is there a way to re-run
just the report part of an old test using lnt?

I couldn't find any *.out file in the test dir (recursively), maybe
there's something wrong with the 'test' rule?

cheers,
--renato

Exactly how are you running the tests? Via LNT or directly?

If via LNT, make sure you test suite repository is clean (no in tree
configure or make results).

- Daniel

Ah, yes! Relative path on clang++ binary.

I understand why it works to get the version but not here (different paths).

Would be good to have some warning (or even an error) if one passes a
relative path. Or, possibly, save the current path and pass it to the
compiler during the tests. Nothing critical, just for consistency.

I'll fix LNT to detect this and just resolve the path.

- Daniel

Exactly how are you running the tests? Via LNT or directly?

Via LNT, like this:

./sandbox/bin/python sandbox/bin/lnt runtest nt \
     --sandbox sandbox \
     --test-suite ~/devel/llvm/test-suite \
     --cc ~/devel/llvm/debug/bin/clang++

If via LNT, make sure you test suite repository is clean (no in tree
configure or make results).

$ git status
# On branch master
nothing to commit (working directory clean)

cheers,
--renato

Exactly how are you running the tests? Via LNT or directly?

Via LNT, like this:

./sandbox/bin/python sandbox/bin/lnt runtest nt \
    --sandbox sandbox \
    --test-suite ~/devel/llvm/test-suite \
    --cc ~/devel/llvm/debug/bin/clang++

You probably make --cc be clang not clang++, but I wouldn't expect
that to fatally break the running of the tests.

The test suite makefiles can be pretty fragile. If you send me the
test.log file I will take a look. If you want to try running on a
smaller subset you can ad something like "--only-test
SingleSource/UnitTests".

- Daniel

(replying on list in case others are interested, Renato sent me the
log offline).

Okay, I found problem, the MultiSource/Applications/Burg test is
breaking the running of the test suite because it can't find yacc. Can
you file a bugzilla for this? We should just add a configure check or
something to make sure that doesn't cause the test suite to fall over.
You should be able to work around it by installing yacc.

- Daniel

Thanks Daniel!

http://llvm.org/bugs/show_bug.cgi?id=14201

I'll install yacc and report back.

FYI, that fixed everything, nightly tests ran smoothly to the end,
producing great reports!

It took 1h20min on a single CoreDuo 2GHz, should be much faster if I
split it -jN.

Thanks!