tests broken

After an update of both llvm and clang, configure, and rebuild on a Linux system in a virtual machine, hundreds of tests are now failing.

Just in case you didn’t already know…

Or is my system messed up?

-John

Hundreds sounds a little high, but there are currently failing tests
in trunk clang; see http://google1.osuosl.org:8011/waterfall . If
you're still seeing failures after that gets cleared up, please send
another email.

-Eli

On a VM running Ubuntu 32-bit I’m seeing 1032 failures, after a fresh update, configure, make. My Python version is 2.5.2. A week and a half ago, there were no failures.

Having gotten a build on MSYS/MinGW to suceed, in running the tests I’m seeing 336 failures. Python is 2.6.1.

Any suggestions on debugging this? I have VERBOSE logs, but they’re probably too big to post here.

-John

On a VM running Ubuntu 32-bit I'm seeing 1032 failures, after a fresh
update, configure, make. My Python version is 2.5.2. A week and a half
ago, there were no failures.

Having gotten a build on MSYS/MinGW to suceed, in running the tests I'm
seeing 336 failures. Python is 2.6.1.

Any suggestions on debugging this? I have VERBOSE logs, but they're
probably too big to post here.

In general, my advice is (a) look at the buildbots to see whether
failures are expected or not. If not, pick any failure which isn't
failing on a build bot and trace it down (you can run individual tests
using TestRunner.sh). In this case, currently all tests are passing on
many platforms, so its likely a local configuration issue. If you
can't figure it out, just file a bug with that one particular test
case, and the details on why it fails.

MSYS/MinGW is of course a separate issue from Ubuntu, but the same
strategy applies.

- Daniel

Under MSYS/MinGW, I looked at a few of the failing tests:

Analysis/cast.c: Apparently MinGW doesn’t have sys/socket.h, which is referenced in a #include. I searched for a package containing socket.h, but didn’t find one. A MinGW mailing list search yielded a hint that socket.h is present in Cygwin, but not MinGW. Any MinGW users out there see this in running the tests?

CodeGenCXX/class-layout.cpp: The test does “grep ‘%.truct.A = type { i8 }’ %t” in the class-layout.cpp.tmp file, but this file has “%struct.A = type { i8 }”.

CodeGenCXX/copy-constructor-elim.cpp: Greps for a mangled symbol not in the output file. Also has this message in stderr: “‘count’ is not recognized as an internal or external command”. I found a count script in llvm/test/Scripts and added it to the path, but still get the same error. I see that a lot of the failures are similar to this one, and for others of those scripts.

CodeGenCXX\reference-field.cpp: Here’s the output from the log:

On Ubuntu32 with gcc version 4.2.4 I ran the Analysis/casts.c test, which had a segmentation fault.

Sorry, the crash I was having in the tests on Linux was my fault. I thought I had an un-modified tree, but I had some experimental changes of my own there that crashed. The perils of experimenting with multiple trees…

It looks like progress is being made on the Visual Studio-based tests. I'm
not sure what the status of the work on the tests in general is, so excuse
me if what I'm seeing is just an intermediate stage of the work.

Things are far enough along that, with patches, you can get it so that
(I think) the remaining failures are real problems.

I just tried the latest clang-test project, which seems to work, as far as
putting out messages. Currently it shows 342 failures, compared to 354 on
MSYS/MinGW. Are you seeing this too? Or do I still have some problem with
my set up?

The main thing I have modified in my setup is to add count.bat and
not.bat scripts in the llvm/test/Scripts directly, so that they work.

Note that I got 3 or 4 abort dialogs, while they were running. It would be
nice if these exceptions where caught instead.

Also, is there a way to get the verbose output when running the tests from
Visual Studio?

You can modify the clang-test build step to call MultiTestRunner.py
with the '-v' argument. That's probably a reasonable default.

Running the script directly appears to be a work-around.

I ran one of the scripts directly for a failing test, and here's the output:

In general running the script directly isn't going to work on Windows.
In fact, in --no-sh mode the script doesn't even get created
currently.

I attached the llvm and clang patches I have on my Windows box.
Currently I run the test runner by hand from a cmd prompt, like this:

llvm.patch (1.44 KB)

clang.patch (1.67 KB)

2009/8/11 Daniel Dunbar <daniel@zuster.org>

add_python_exe_path.patch (779 Bytes)

Thanks, Daniel. I’ve been experimenting with the patches.

I know the count.bat and not.bat scripts are experimental and that you only temporarily have the them using hard-coded paths for the Python executable and the Python scripts they call (i.e. g:\public\llvm\test\Scripts\not.py).

I think the Python executable scripts could just rely on python being in the path. (I use Python 2.6, so the path is different.)

I’m not sure what the best solution is for the Python script path, however. Some possibilities:

  1. Use an environment variable for the LLVM root (i.e. LLVM_ROOT) in the script. Using environment variables, of course, would be unfortunate.

  2. Would setting PYTHONPATH to (prefix)\llvm\test\Scripts and just using a bare script file name in the batch files work?

  3. Have users edit the scripts manually.

Or perhaps you have a better solution already.

With Baptiste’s clang-win32-path.patch and llvm-win32-path.patch files applied I’m down to 64 failures.

I’m going to put back and try Baptiste’s other script patches again, as my failures might have been due to not noticing the path issue with the count.bat and not.bat scripts.

-John

I've been looking at the remaining test failures. Most seem to be
related to one of the following:

1. Some forms of the search pattern argument given to grep are
problematic. For example, in CodeGen/mangle.c given @"\\01foo", the
pattern @"\01foo" is in the file being searched, but apparently the
argument isn't being processed and pass through as expected. I tried
several usages of "\" to escape things, to no avail. The
double-quotes are probably the problem.

2. Some differences in floating point formatting. For example, in
cast-to-union.c, 7.300000e+01 is searched for, but the output contains
7.300000e+001. Maybe use a regular expression for the possibly extra
0?

3. Seemingly mal-formed grep commands like this (for Driver\openbsd.c):

Command 1: "c:\Msys\1.0\bin\grep.EXE" "clang-cc" "-triple"
"i386-pc-openbsd""
"C:\Tools\llvm\tools\clang\test\Output\Driver\openbsd.c.tmp.log"

4. Unexpected warnings related to _Complex.

5. "/dev/null" on clang-cc commandline (i.e. PCH\pr4489.c).

6. Missing #include files, such as socket.h, reloc2.h, not found in
Visual Studio.

7. Errors or #error directives in Visual Studio headers.

8. Missing or unexpected errors or warnings (i.e. Parser\cxx-template-decl.cpp).

There are probably other classes of problems, but these seem to be the majority.

Perhaps someone is already working on these. Otherwise how can I best help?

My ultimate goal is to get back to working on the semantics for using
declarations, but I was hoping first to see or help get the tests
working with respect to Visual Studio. Please point me where my
limited skills can be of the most help.

Thanks.

-John