Bad gcc versions

I have been tracking down various build problems and found some
more gcc versions that don't work. 4.4.x and 4.5.x seem to
be particularly bad.

In addition to the set of bad gcc versions posted on the web,
can we create a list of known-good gcc versions? It took me
a long time to figure out a good combination.

On SLES 10.1 x86_64, gcc 4.1.2 works reasonably well, though one
clang test fails.

Builds with gcc 4.4.4, 4.4.3, 4.4.2 and 4.5.1 all failed significant
numbers of regression tests.

                               -Dave

I think the problem is also platform dependent, and I have been trying to come up with the known-good list of build gcc/g++ on various platforms for a long time.

E.g.,
on Debian-32 5.0.5 (Intel): gcc-4.0.4, gcc-4.1.2 are bad, gcc-4.2.4 seems to be fine.
on Debian-64 5.0.4 (Intel): the default gcc (4.3.2) seems to be fine.

We may want to cover this for a wide range of possible platforms. The ones listed in the Getting Started Guide is in complete.

Chuck

I have been tracking down various build problems and found some
more gcc versions that don't work. 4.4.x and 4.5.x seem to
be particularly bad.

What are we left with then? Only 4.2 and 4.3?
I only use 4.4 since a while, and it works fairly well.

Are you sure it is not a bug in the regression tests themselves
(strict-aliasing bugs, etc.)?
Which regression tests are failing with LLVM 2.8 and GCC 4.4.x?

In addition to the set of bad gcc versions posted on the web,
can we create a list of known-good gcc versions? It took me
a long time to figure out a good combination.

On SLES 10.1 x86_64, gcc 4.1.2 works reasonably well, though one
clang test fails.

In ClamAV we enable LLVM only if GCC is >= 4.1.2 [*], and that has
worked fairly well.

We only had problems with Python (<2.4 are problematic),custom built Pythons lacking thread support, some Python versions (2.5+)
just crash, etc. We also had problems with some tests (they got fixed
AFAICT) that didn't work when some backend was not built (like
X86-specific tests enabled on PPC).

So in the latest version we don't run LLVM's make check at all, but
aside from Python bugs I didn't see any compiler bugs causing them to
fail.

[*] Certain old versions of 4.1.2 on RHEL were buggy, but recently updated
4.1.2 on RHEL worked.

Best regards,
--Edwin

Hi David,

I have been tracking down various build problems and found some
more gcc versions that don't work. 4.4.x and 4.5.x seem to
be particularly bad.

I've been using 4.4 and 4.5 from ubuntu for some time and they work
fine for me.

Ciao, Duncan.

Chuck Zhao <czhao@eecg.toronto.edu> writes:

I think the problem is also platform dependent

Absolutely. I find that different versions work on different platforms.
It seems odd, but that's what I see. gcc has changed ABI a few times
so some combination of gcc and library version may not work. But it
looks like more than just that.

We may want to cover this for a wide range of possible platforms. The
ones listed in the Getting Started Guide is in complete.

Yep, that was my thought as well. A table of gcc x platform might work.

                            -Dave

Török Edwin <edwintorok@gmail.com> writes:

What are we left with then? Only 4.2 and 4.3?

On SLES 10.1 at least. I think it is highly platform dependent.

I only use 4.4 since a while, and it works fairly well.

On what platform?

Are you sure it is not a bug in the regression tests themselves
(strict-aliasing bugs, etc.)?

No, I'm not sure.

Which regression tests are failing with LLVM 2.8 and GCC 4.4.x?

Too many to list.

So in the latest version we don't run LLVM's make check at all

That's dangerous in my experience.

                               -Dave

Török Edwin <edwintorok@gmail.com> writes:

> What are we left with then? Only 4.2 and 4.3?

On SLES 10.1 at least. I think it is highly platform dependent.

Also keep in mind that llvm-gcc uses the 4.2 unwinder, so if you are
seeing EH failures maybe the EH info generated by llvm-gcc is not
compatible with 4.4 or 4.5
(we had this problem in the past, but got fixed).

> I only use 4.4 since a while, and it works fairly well.

On what platform?

Debian and RHEL5, x86-64. Early versions of 4.1.2 that comes with RHEL5
are not very reliable.

> Are you sure it is not a bug in the regression tests themselves
> (strict-aliasing bugs, etc.)?

No, I'm not sure.

> Which regression tests are failing with LLVM 2.8 and GCC 4.4.x?

Too many to list.

Can you give me 2 or 3 examples (that fail with LLVM 2.8 and GCC 4.4
but work with LLVM 2.8 and GCC 4.3), also I'd like to know how they
fail.
If I have some time I'll check with my 4.4 to see if they fail.

> So in the latest version we don't run LLVM's make check at all

That's dangerous in my experience.

Well I still check locally, but I don't want anymore "Python doesn't
work" bugreports on ClamAV bugzilla.

Best regards,
--Edwin

Török Edwin <edwintorok@gmail.com> writes:

> Which regression tests are failing with LLVM 2.8 and GCC 4.4.x?

Too many to list.

Can you give me 2 or 3 examples (that fail with LLVM 2.8 and GCC 4.4
but work with LLVM 2.8 and GCC 4.3), also I'd like to know how they
fail.
If I have some time I'll check with my 4.4 to see if they fail.

Well, I have some failures even with 4.1.2, attached below.

I have hundreds of failures scattered all over the tests.

Often I run a few different builds in parallel, with different obj/build
directories. Is it possible that the test infrastructure writes
something to the source directories or some common temp directory? That
could confuse things when doing parallel build/test and would explain
all these failures. When I don't run in parallel, things seem to work
much better.

                                   -Dave

[x86_64-off-dbg]: FAIL: Clang :: Driver/hello.c (1160 of 8402)
[x86_64-off-dbg]: ******************** TEST 'Clang :: Driver/hello.c' FAILED ********************
[x86_64-off-dbg]: Script:
[x86_64-off-dbg]: --
[x86_64-off-dbg]: /ptmp/dag/build.llvm.trunk.official.debug/x86_64-unknown-linux-gnu/Debug+Asserts/bin/clang -ccc-echo -o /ptmp/dag/build.llvm.trunk.official.debug/x86_64-unknown-linux-gnu/tools/clang/test/Driver/Output/hello.c.tmp /ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/Driver/hello.c 2> /ptmp/dag/build.llvm.trunk.official.debug/x86_64-unknown-linux-gnu/tools/clang/test/Driver/Output/hello.c.tmp.log
[x86_64-off-dbg]: grep 'clang" -cc1 .*hello.c' /ptmp/dag/build.llvm.trunk.official.debug/x86_64-unknown-linux-gnu/tools/clang/test/Driver/Output/hello.c.tmp.log
[x86_64-off-dbg]: /ptmp/dag/build.llvm.trunk.official.debug/x86_64-unknown-linux-gnu/tools/clang/test/Driver/Output/hello.c.tmp > /ptmp/dag/build.llvm.trunk.official.debug/x86_64-unknown-linux-gnu/tools/clang/test/Driver/Output/hello.c.tmp.out
[x86_64-off-dbg]: grep "I'm a little driver, short and stout." /ptmp/dag/build.llvm.trunk.official.debug/x86_64-unknown-linux-gnu/tools/clang/test/Driver/Output/hello.c.tmp.out
[x86_64-off-dbg]: --
[x86_64-off-dbg]: Exit Code: 1
[x86_64-off-dbg]:
[x86_64-off-dbg]: ********************

[x86_64-off-dbg]: FAIL: LLVM :: Analysis/BasicAA/2010-09-15-GEP-SignedArithmetic.ll (2727 of 8402)
[x86_64-off-dbg]: ******************** TEST 'LLVM :: Analysis/BasicAA/2010-09-15-GEP-SignedArithmetic.ll' FAILED ********************
[x86_64-off-dbg]: Script:
[x86_64-off-dbg]: --
[x86_64-off-dbg]: opt < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/2010-09-15-GEP-SignedArithmetic.ll -basicaa -aa-eval -print-all-alias-modref-info -disable-output |& grep {1 may alias}
[x86_64-off-dbg]: --
[x86_64-off-dbg]: Exit Code: 1
[x86_64-off-dbg]:
[x86_64-off-dbg]: ********************

[x86_64-off-dbg]: FAIL: LLVM :: Analysis/BasicAA/args-rets-allocas-loads.ll (2728 of 8402)
[x86_64-off-dbg]: ******************** TEST 'LLVM :: Analysis/BasicAA/args-rets-allocas-loads.ll' FAILED ********************
[x86_64-off-dbg]: Script:
[x86_64-off-dbg]: --
[x86_64-off-dbg]: opt -basicaa -aa-eval -print-all-alias-modref-info -disable-output < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/args-rets-allocas-loads.ll |& FileCheck /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/args-rets-allocas-loads.ll
[x86_64-off-dbg]: --
[x86_64-off-dbg]: Exit Code: 1
[x86_64-off-dbg]: Command Output (stderr):
[x86_64-off-dbg]: --
[x86_64-off-dbg]: /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/args-rets-allocas-loads.ll:171:10: error: expected string not found in input
[x86_64-off-dbg]: ; CHECK: Both ModRef: Ptr: double* %arg_a0 <-> %normal_ret_a0 = call double* @normal_returner()
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: <stdin>:122:2: note: scanning from here
[x86_64-off-dbg]: ModRef: Ptr: double* %arg_a0 <-> %normal_ret_a0 = call double* @normal_returner() ; <double*> [#uses=1]
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: --
[x86_64-off-dbg]:
[x86_64-off-dbg]: ********************

[x86_64-off-dbg]: FAIL: LLVM :: Analysis/BasicAA/featuretest.ll (2733 of 8402)
[x86_64-off-dbg]: ******************** TEST 'LLVM :: Analysis/BasicAA/featuretest.ll' FAILED ********************
[x86_64-off-dbg]: Script:
[x86_64-off-dbg]: --
[x86_64-off-dbg]: opt < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/featuretest.ll -basicaa -gvn -instcombine -dce -S | FileCheck /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/featuretest.ll
[x86_64-off-dbg]: --
[x86_64-off-dbg]: Exit Code: 1
[x86_64-off-dbg]: Command Output (stderr):
[x86_64-off-dbg]: --
[x86_64-off-dbg]: /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/featuretest.ll:126:10: error: expected string not found in input
[x86_64-off-dbg]: ; CHECK: ret i16 %.ret
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: <stdin>:58:32: note: scanning from here
[x86_64-off-dbg]: define i16 @zext_sext_confusion(i16* %row2col, i5 %j) nounwind {
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: <stdin>:60:2: note: possible intended match here
[x86_64-off-dbg]: ret i16 0
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: --
[x86_64-off-dbg]:
[x86_64-off-dbg]: ********************

[x86_64-off-dbg]: FAIL: LLVM :: Analysis/BasicAA/getmodrefinfo-cs-cs.ll (2736 of 8402)
[x86_64-off-dbg]: ******************** TEST 'LLVM :: Analysis/BasicAA/getmodrefinfo-cs-cs.ll' FAILED ********************
[x86_64-off-dbg]: Script:
[x86_64-off-dbg]: --
[x86_64-off-dbg]: opt < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/getmodrefinfo-cs-cs.ll -basicaa -aa-eval -print-all-alias-modref-info -disable-output |& FileCheck /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/getmodrefinfo-cs-cs.ll
[x86_64-off-dbg]: --
[x86_64-off-dbg]: Exit Code: 1
[x86_64-off-dbg]: Command Output (stderr):
[x86_64-off-dbg]: --
[x86_64-off-dbg]: /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/getmodrefinfo-cs-cs.ll:4:10: error: expected string not found in input
[x86_64-off-dbg]: ; CHECK: Just Ref: call void @ro() <-> call void @f0()
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: <stdin>:1:1: note: scanning from here
[x86_64-off-dbg]: Function: test0: 0 pointers, 2 call sites
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: <stdin>:5:2: note: possible intended match here
[x86_64-off-dbg]: NoModRef: Ptr: i8* @B <-> call void @llvm.memset.p0i8.i64(i8* @A, i8 0, i64 1, i32 1, i1 false)
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: --
[x86_64-off-dbg]:
[x86_64-off-dbg]: ********************

[x86_64-off-dbg]: FAIL: LLVM :: Analysis/RegionInfo/mix_1.ll (2791 of 8402)
[x86_64-off-dbg]: ******************** TEST 'LLVM :: Analysis/RegionInfo/mix_1.ll' FAILED ********************
[x86_64-off-dbg]: Script:
[x86_64-off-dbg]: --
[x86_64-off-dbg]: opt -regions -analyze < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll | FileCheck /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll
[x86_64-off-dbg]: opt -regions -stats < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll |& FileCheck -check-prefix=STAT /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll
[x86_64-off-dbg]: opt -regions -print-region-style=bb -analyze < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll |& FileCheck -check-prefix=BBIT /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll
[x86_64-off-dbg]: opt -regions -print-region-style=rn -analyze < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll |& FileCheck -check-prefix=RNIT /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll
[x86_64-off-dbg]: --
[x86_64-off-dbg]: Exit Code: 1
[x86_64-off-dbg]: Command Output (stderr):
[x86_64-off-dbg]: --
[x86_64-off-dbg]: opt: Unknown command line argument '-regions'. Try: 'opt -help'
[x86_64-off-dbg]: /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/RegionInfo/mix_1.ll:51:10: error: expected string not found in input
[x86_64-off-dbg]: ; CHECK: [0] 0 => <Function Return>
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: <stdin>:1:1: note: scanning from here
[x86_64-off-dbg]:
[x86_64-off-dbg]: ^
[x86_64-off-dbg]: --
[x86_64-off-dbg]:
[x86_64-off-dbg]: ********************

[x86_64-off-dbg]: FAIL: LLVM :: Bitcode/null-type.ll (3030 of 8402)
[x86_64-off-dbg]: ******************** TEST 'LLVM :: Bitcode/null-type.ll' FAILED ********************
[x86_64-off-dbg]: Script:
[x86_64-off-dbg]: --
[x86_64-off-dbg]: not llvm-dis < /ptmp/dag/llvm-project.official/llvm/trunk/test/Bitcode/null-type.ll.bc > /dev/null |& grep "Invalid MODULE_CODE_FUNCTION record"
[x86_64-off-dbg]: --
[x86_64-off-dbg]: Exit Code: 1
[x86_64-off-dbg]:
[x86_64-off-dbg]: ********************

greened@obbligato.org (David A. Greene) writes:

Often I run a few different builds in parallel, with different obj/build
directories. Is it possible that the test infrastructure writes
something to the source directories or some common temp directory? That
could confuse things when doing parallel build/test and would explain
all these failures. When I don't run in parallel, things seem to work
much better.

There is definitely something to this. If I take a random failing
testcase and run the test in isolation in the shell, it works. So what,
if anything, does lit/FileCheck/etc. do that might run interference
if there is another copy of lit/FileCheck/etc. running at the same time?
I tried strace -etrace=file but nothing obvious pops out.

                               -Dave

I don't see anything wrong with FileCheck either.

However looks here, that .bc file is in the *source* tree, not the obj tree:
not llvm-dis < /ptmp/dag/llvm-project.official/llvm/trunk/test/Bitcode/null-type.ll.bc > /dev/null |& grep "Invalid MODULE_CODE_FUNCTION record"

And I guess the .bc file is created by something during the test, multiple builds in parallel == everything breaks.
Daniel, is the .bc file created by 'lit'?

Best regards,
--Edwin

Török Edwin <edwintorok@gmail.com> writes:

I don't see anything wrong with FileCheck either.

However looks here, that .bc file is in the *source* tree, not the obj tree:
not llvm-dis < /ptmp/dag/llvm-project.official/llvm/trunk/test/Bitcode/null-type.ll.bc > /dev/null |& grep "Invalid MODULE_CODE_FUNCTION record"

I think that's there from checkout. You can see it on the web.

It seems I spoke too soon about build-level parallelization (running
multiple parallel builds in parallel). 4.1.2 works fine for
Debug+Asserts builds but fails horribly on Release+Asserts buils on SLES
10.1 when run by itself. I'm assuming there's nothing wrong with doing
an LLVM parallel build (make -j).

Sigh. More compilers to try...

                             -Dave

greened@obbligato.org (David A. Greene) writes:

> Often I run a few different builds in parallel, with different
> obj/build directories. Is it possible that the test infrastructure
> writes something to the source directories or some common temp
> directory? That could confuse things when doing parallel
> build/test and would explain all these failures. When I don't run
> in parallel, things seem to work much better.

There is definitely something to this. If I take a random failing
testcase and run the test in isolation in the shell, it works. So
what, if anything, does lit/FileCheck/etc. do that might run
interference if there is another copy of lit/FileCheck/etc. running
at the same time? I tried strace -etrace=file but nothing obvious
pops out.

Perhaps its this? - http://llvm.org/bugs/show_bug.cgi?id=8199

-jason

Török Edwin <edwintorok@gmail.com> writes:

> I don't see anything wrong with FileCheck either.
>
> However looks here, that .bc file is in the *source* tree, not the
> obj tree: not llvm-dis
> < /ptmp/dag/llvm-project.official/llvm/trunk/test/Bitcode/null-type.ll.bc
> > /dev/null |& grep "Invalid MODULE_CODE_FUNCTION record"

I think that's there from checkout. You can see it on the web.

Ah OK.

It seems I spoke too soon about build-level parallelization (running
multiple parallel builds in parallel). 4.1.2 works fine for
Debug+Asserts builds but fails horribly on Release+Asserts buils on
SLES 10.1 when run by itself.

Can you try running some of the failing lines manually and see if they
still fail?

Also are you doing these tests on LLVM 2.8, or trunk?

There's one caveat with running tests though:
if you have opt, llc, etc. in your path then 'make check' might use
those instead of the ones you just built.
IIRC I've bumped into that when I had an old LLVM 2.6 in /usr/local, and
2.7 tests failed.

I'm assuming there's nothing wrong
with doing an LLVM parallel build (make -j).

make -j is fine, I do it all the time. I've never run make check
in parallel though.

Sigh. More compilers to try...

Try 4.3 if SLES has it.

Best regards,
--Edwin

Török Edwin <edwintorok@gmail.com> writes:

It seems I spoke too soon about build-level parallelization (running
multiple parallel builds in parallel). 4.1.2 works fine for
Debug+Asserts builds but fails horribly on Release+Asserts buils on
SLES 10.1 when run by itself.

Can you try running some of the failing lines manually and see if they
still fail?

The ones I try manually pass. I assume all of them do. That means
there's something about my build environment that's wonky, I think.

Also are you doing these tests on LLVM 2.8, or trunk?

Trunk.

There's one caveat with running tests though:
if you have opt, llc, etc. in your path then 'make check' might use
those instead of the ones you just built.

Hmm...I'll double-check that. This seems like a fragile test setup. I
would expect hard-coded paths to the tools being tested.

Yep, I do have those tools in my path. But why would results change for
Debug vs. Release?

IIRC I've bumped into that when I had an old LLVM 2.6 in /usr/local, and
2.7 tests failed.

I'm assuming there's nothing wrong
with doing an LLVM parallel build (make -j).

make -j is fine, I do it all the time. I've never run make check
in parallel though.

I don't run make check in parallel either. I do make check-all, though.
I wonder if the extra tests are the ones failing. I'll change that and
the path and see what happens.

                           -Dave

Jason Kim <jasonwkim@google.com> writes:

There is definitely something to this. If I take a random failing
testcase and run the test in isolation in the shell, it works. So
what, if anything, does lit/FileCheck/etc. do that might run
interference if there is another copy of lit/FileCheck/etc. running
at the same time? I tried strace -etrace=file but nothing obvious
pops out.

Perhaps its this? - http://llvm.org/bugs/show_bug.cgi?id=8199

Good catch! I think that is probably it. I have old installations of
llvm-gcc/opt/llc/etc. that I use to tell the new build where llvm-gcc
is. It would explain why results for Debug/Release differ because I do
separate installations for each build flavor and branch.

This may also explain why I've sometimes seen tests working in the past
where others see failures.

Wow, this probably explains a lot.

For now, I think if I tweak the way I do the build to always build
without pointing to llvm-gcc first, build and test LLVM then build
llvm-gcc and re-build LLVM, it should work. It will take much longer,
though. :frowning:

                                 -Dave

greened@obbligato.org (David A. Greene) writes:

For now, I think if I tweak the way I do the build to always build
without pointing to llvm-gcc first, build and test LLVM then build
llvm-gcc and re-build LLVM, it should work. It will take much longer,
though. :frowning:

I updated the bug explaining what I'm seeing. I think the correct fix
is to use absolute paths to tools being tested rather than relying on
PATH.

                               -Dave

greened@obbligato.org (David A. Greene) writes:

greened@obbligato.org (David A. Greene) writes:

For now, I think if I tweak the way I do the build to always build
without pointing to llvm-gcc first, build and test LLVM then build
llvm-gcc and re-build LLVM, it should work. It will take much longer,
though. :frowning:

I updated the bug explaining what I'm seeing. I think the correct fix
is to use absolute paths to tools being tested rather than relying on
PATH.

So I tried adding this to llvm.exp::substitute:

[...]

  global llvmdsymutil valgrind grep gas bugpoint_topts llvmtoolsdir

[...]

# Replace references to llvm tools to tools in OBJDIR.
  regsub -all {bugpoint } $new_line "$llvmtoolsdir/bugpoint " new_line
  regsub -all {llc } $new_line "$llvmtoolsdir/llc " new_line
  regsub -all {lli } $new_line "$llvmtoolsdir/lli " new_line
  regsub -all {llvm-ar } $new_line "$llvmtoolsdir/llvm-ar " new_line
  regsub -all {llvm-as } $new_line "$llvmtoolsdir/llvm-as " new_line
  regsub -all {llvm-bcanalyzer } $new_line "$llvmtoolsdir/llvm-bcanalyzer " new_line
  regsub -all {llvm-dis } $new_line "$llvmtoolsdir/llvm-dis " new_line
  regsub -all {llvm-extract } $new_line "$llvmtoolsdir/llvm-extract " new_line
  regsub -all {llvm-ld } $new_line "$llvmtoolsdir/llvm-ld " new_line
  regsub -all {llvm-link } $new_line "$llvmtoolsdir/llvm-link " new_line
  regsub -all {llvm-nm } $new_line "$llvmtoolsdir/llvm-nm " new_line
  regsub -all {llvm-prof } $new_line "$llvmtoolsdir/llvm-prof " new_line
  regsub -all {llvm-ranlib } $new_line "$llvmtoolsdir/llvm-ranlib " new_line
  regsub -all {([^a-zA-Z_-])opt } $new_line "$llvmtoolsdir/\\1opt " new_line
  regsub -all {opt } $new_line "$llvmtoolsdir/opt " new_line
  regsub -all {tblgen } $new_line "$llvmtoolsdir/tblgen " new_line

It appears to work for clang tests:

[x86_64-off-opt]: FAIL: Clang :: CodeCompletion/ordinary-name.c (432 of 8411)
[x86_64-off-opt]: ******************** TEST 'Clang :: CodeCompletion/ordinary-name.c' FAILED ********************
[x86_64-off-opt]: Script:
[x86_64-off-opt]: --
[x86_64-off-opt]: /ptmp/dag/build.llvm.trunk.official.opt/x86_64-unknown-linux-gnu/Release+Asserts/bin/clang -cc1 -isystem /ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/CodeCompletion/Inputs -fsyntax-only -code-completion-at=/ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/CodeCompletion/ordinary-name.c:6:9 /ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/CodeCompletion/ordinary-name.c -o - | FileCheck -check-prefix=CHECK-CC1 /ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/CodeCompletion/ordinary-name.c
[x86_64-off-opt]: /ptmp/dag/build.llvm.trunk.official.opt/x86_64-unknown-linux-gnu/Release+Asserts/bin/clang -cc1 -isystem /ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/CodeCompletion/Inputs -fsyntax-only -code-completion-at=/ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/CodeCompletion/ordinary-name.c:1:11 /ptmp/dag/llvm-project.official/llvm/trunk/tools/clang/test/CodeCompletion/ordinary-name.c
[x86_64-off-opt]: --
[x86_64-off-opt]: Exit Code: 134

but it does not appear to do anything for LLVM tests:

[x86_64-off-opt]: FAIL: LLVM :: Analysis/BasicAA/2010-09-15-GEP-SignedArithmetic.ll (2731 of 8411)
[x86_64-off-opt]: ******************** TEST 'LLVM :: Analysis/BasicAA/2010-09-15-GEP-SignedArithmetic.ll' FAILED ********************
[x86_64-off-opt]: Script:
[x86_64-off-opt]: --
[x86_64-off-opt]: opt < /ptmp/dag/llvm-project.official/llvm/trunk/test/Analysis/BasicAA/2010-09-15-GEP-SignedArithmetic.ll -basicaa -aa-eval -print-all-alias-modref-info -disable-output |& grep {1 may alias}
[x86_64-off-opt]: --
[x86_64-off-opt]: Exit Code: 1
[x86_64-off-opt]:
[x86_64-off-opt]: ********************

I cannot figure out where in lit the test script gets printed out.

Any hints on how to make absolute paths stick?

                               -Dave

greened@obbligato.org (David A. Greene) writes:

Any hints on how to make absolute paths stick?

Ok, I finally got that to work. I'm running experiments now.

                          -Dave

greened@obbligato.org (David A. Greene) writes:

greened@obbligato.org (David A. Greene) writes:

Any hints on how to make absolute paths stick?

Ok, I finally got that to work. I'm running experiments now.

After fixing a few Python bugs in my code, I now have everything
passing. Prepending the path to the tools being tested fixed all the
problems. I will post a proposed patch to PR 8199 just as soon as I get
approval to do so.

Thanks to everyone who helped track this down!

                              -Dave

greened@obbligato.org (David A. Greene) writes:

After fixing a few Python bugs in my code, I now have everything
passing. Prepending the path to the tools being tested fixed all the
problems. I will post a proposed patch to PR 8199 just as soon as I get
approval to do so.

Patch posted. Someone (Daniel?) please let me know if I can commit it.
I can't make any progress sending other code upstream without this as
my testing isn't reliable without this fix.

Thanks!

                               -Dave