[3.7 Release] RC1 has been tagged, Testing Phase I begins

I've been bitten by the same thing. The mipsel-linux-gnu packages have an expected failure due to an NFS option (noatime) and both mips-linux-gnu/mipsel-linux-gnu have additional non-regression failures related to newly built projects.

After realizing the problem I re-ran with r241599 reverted but unfortunately I hadn't noticed the 'rm -rf' commands that delete the objdir and destdir so the mips-linux-gnu is being rebuilt from scratch. I should be able to salvage an initial package for mipsel-linux-gnu though.

...

Hm, strangely enough, this version of the script does not go further than the Phase 2 installation, and does not run any tests? This used to work fine for the release_36 branch.

I think it is because of the "set -o pipefail" which was introduced, but I don't yet understand why this causes the Phase 2 installation to appear to fail, as there is no visible error. I will investigate, or work around it by removing the pipefail option again.

It appears to be caused by the clean_RPATH() function, which has this line:

      rpath=`objdump -x $Candidate | grep 'RPATH' | sed -e's/^ *RPATH *//'`

If the objdump'd file does not have any RPATH, the whole statement will fail, since set -o pipefail is in effect. This terminates the script, since set -e is also in effect.

The line can be replaced with this instead:

      rpath=`objdump -x $Candidate | sed -ne'/RPATH/{s/^ *RPATH *//;p;}'`

-Dimitry

use-autoconf-on-freebsd-too-2.diff (2.03 KB)

Hi Hans,

I uploaded

    c55199c92877ed068b2f7d1ece6b0bb4f121a63a clang+llvm-3.7.0-rc1-x86_64-apple-darwin.tar.xz

to SFTP. Testing was OK except for differences between phase 2 and 3 which I consider being not harmful (or even: usual). No problem with the build script here, but OS X also does not use RPATH.

Cheers,
Sebastian

Another new problem with the test-release.sh script is that it can cause the whole machine (!) to run out of memory, when comparing phase 2 with phase 3, due to the following fragment:

   470 echo "# Comparing Phase 2 and Phase 3 files"
   471 for p2 in `find $llvmCore_phase2_objdir -name '*.o'` ; do
   472 p3=`echo $p2 | sed -e 's,Phase2,Phase3,'`
   473 # Substitute 'Phase2' for 'Phase3' in the Phase 2 object file in
   474 # case there are build paths in the debug info. On some systems,
   475 # sed adds a newline to the output, so pass $p3 through sed too.
   476 if ! cmp --ignore-initial=16 <(sed -e 's,Phase2,Phase3,g' $p2) \
   477 <(sed -e '' $p3) > /dev/null 2>&1 ; then
   478 echo "file `basename $p2` differs between phase 2 and phase 3"
   479 fi
   480 done

Because cmp(1) on FreeBSD does not support the --ignore-initial option, which is a GNU extension, the command immediately fails. Then, the <(...) constructs on lines 476 and 477 spawn two new bash instances per iteration, and these never get cleaned up, at least not on FreeBSD. This may very well be a bash bug.

Alternatively, the 'skip' values can be specified as the third and fourth argument on the cmp(1) command line, and this works on both Linux, FreeBSD and OSX; e.g. this:

            if ! cmp -s <(sed -e 's,Phase2,Phase3,g' $p2) <(sed -e '' $p3) \
                    16 16 ; then

I think this is the final adjustment I have to make to get the building and testing to complete. Hans, is it OK if I commit these changes to test-release.sh in trunk? Then I'll merge them to release_37 afterwards.

-Dimitry

use-autoconf-on-freebsd-too-3.diff (2.42 KB)

...

I think this is the final adjustment I have to make to get the building and testing to complete. Hans, is it OK if I commit these changes to test-release.sh in trunk? Then I'll merge them to release_37 afterwards.

After those adjustments, building and testing went fine, and no further failures were seen. I have uploaded the following tarballs:

SHA256 (clang+llvm-3.7.0-rc1-amd64-unknown-freebsd10.tar.xz) = e6f5228624a8931878b78d6e1f9efaf8899bf036c92611eef745d0b963fdfeca
SHA256 (clang+llvm-3.7.0-rc1-i386-unknown-freebsd10.tar.xz) = 32026df3017514eb20bff731c441de05e948f27fa4393eee8eb62f0ba84c5be4

-Dimitry

Here's the current status for the Mips packages.

With the following source changes to the tag:
* Applied r242331 - Switch the release script to build with CMake by default (PR21561)
* Applied r242422 - Fixed an error in cuda-options.cu test -target option must be used without '='.
* Reverted r241599 - Fix bug in test-release.sh where the script would not exit if any of the build stages that are sent through a pipe (e.g. tee) failed.

And the following change to the resulting packages:
* Move/symlink $package/usr/local/* to $package/*
(Note the packages uploaded are as built by the script and do not have this change)

clang+llvm-3.7.0-rc1-mips-linux-gnu.tar.xz
        Failing Tests (6):
            AddressSanitizer-mips-linux :: TestCases/Linux/kernel-area.cc
            AddressSanitizer-mips-linux :: TestCases/Posix/coverage-direct-large.cc
            UBSan-ASan-mips :: TestCases/Float/cast-overflow.cpp
            UBSan-Standalone-mips :: TestCases/Float/cast-overflow.cpp
            libc++abi :: backtrace_test.pass.cpp
            libc++abi :: test_demangle.pass.cpp
        Test-suite: Just started this morning.

clang+llvm-3.7.0-rc1-mipsel-linux-gnu.tar.xz
        Failing Tests (4):
            AddressSanitizer-mipsel-linux :: TestCases/Linux/kernel-area.cc
            Clang :: Modules/prune.m <- expected since this system uses an NFS mount with noatime.
            libc++abi :: backtrace_test.pass.cpp
            libc++abi :: test_demangle.pass.cpp
            Test-suite: Still running but ok so far.

clang+llvm-3.7.0-rc1-x86_64-linux-gnu-ubuntu-14.04.tar.xz (cross compiling to Mips):
  Not started since packages not uploaded yet. Presumably due to the packaging problems discussed elsewhere in this thread.

With the exception of prune.m (which is expected on that machine) the 'make check-all' failures are all related to projects that weren't built in the 3.6 releases. It will take a day or two for the test-suite results to finish since these machines lack an FPU and have to trap-and-emulate. I have a little endian with an FPU that I can try the test-suite on which should be much faster.

On Ubuntu 14.04 x64 (with cmake) I get these failures at Phase3 and no
tarball:

The tarball is an issue Hans is aware of. We're fixing it.

Failing Tests (18):
     AddressSanitizer-x86_64-linux :: TestCases/Posix/readv.cc

Did you use to build compiler-rt on the previous (autoconf) releases?

I don't know. I didn't specifically enable it.

If not, this might explain why you're only seeing these errors now.

Perhaps.

How should I proceed?

1) Use the autoconf build and upload if I get a tarball?
2) Disable compiler-rt (how?)?
3) Disable the -o pipefail and upload if I get a tarball?
4) Something else?

There are no updates on the trunk test-release scripts (I haven't followed the discussion too closely).

Ben

AArch64 with Compiler-RT up. Both tested with the test-suite, all pass.

cheers,
--renato

Should be fixed by r242706.

Thanks,
Hans

Thanks!

The file doesn't seem to have -rc1 in the name like the others, but it
does seem to have been built at the rc1 tag. Do you think we can just
rename it?

Thanks,
Hans

Renato ran into the same problem of some components not working when
building on ARM and has a patch out for disabling them:
http://reviews.llvm.org/D11326

That might be a better approach actually. Since we didn't use to
include libcxx or libcxxabi in previous releases, feel free to disable
those (but please file bugs for the problems anyway). For compiler-rt,
we did include that in previous releases so it would be good if we
could make it work. What errors are you running into?

Thanks,
Hans

Hi Kun,

Hi Hans,
      Test Result Report: Ubuntu 15.04 AMD64 3.7.0-RC1 release . Everything
works fine without any problem.

Nice!

       SFTP Upload: Not done yet. Since do not have the SFTP access
information.

Anton, can you help Kun get access to the sftp?

Thanks,
Hans

...

Hm, strangely enough, this version of the script does not go further than the Phase 2 installation, and does not run any tests? This used to work fine for the release_36 branch.

I think it is because of the "set -o pipefail" which was introduced, but I don't yet understand why this causes the Phase 2 installation to appear to fail, as there is no visible error. I will investigate, or work around it by removing the pipefail option again.

It appears to be caused by the clean_RPATH() function, which has this line:

      rpath=`objdump -x $Candidate | grep 'RPATH' | sed -e's/^ *RPATH *//'`

If the objdump'd file does not have any RPATH, the whole statement will fail, since set -o pipefail is in effect. This terminates the script, since set -e is also in effect.

Nice find!

The line can be replaced with this instead:

      rpath=`objdump -x $Candidate | sed -ne'/RPATH/{s/^ *RPATH *//;p;}'`

How about just guarding it with an if statement? I'm worried about
adding non-obvious commands to the script, both for making it harder
to understand and for portability.

Thanks,
Hans

Hi Hans,

I uploaded

    c55199c92877ed068b2f7d1ece6b0bb4f121a63a clang+llvm-3.7.0-rc1-x86_64-apple-darwin.tar.xz

to SFTP.

Thanks!

Testing was OK except for differences between phase 2 and 3 which I consider being not harmful (or even: usual). No problem with the build script here, but OS X also does not use RPATH.

Do you have any idea what the diff is though? The script tries hard to
build and compare in a way that should make the builds compare equal.

Thanks,
Hans

The compilation fails because it cannot find unwind.h, since we still
have not cleaned up our act in FreeBSD, and chose one of the 4 (or so)
available versions we have in our tree, to install into the standard
system include directories.

Other than that, I recall there were still several test failures in the
sanitizer parts. This needs more work to get it completely done.

I'm not sure if the llvm libunwind you added in r242543 will get picked
up during the build of compiler-rt, but that could at least provide
*some* form of unwinding library. It would be better for FreeBSD to
just import that wholesale, so we can finally ditch libgcc... :slight_smile:

-Dimitry

That seems like a good solution, feel free to commit that.

I wonder why the first 16 bytes need to be skipped though. Does anyone know?

Thanks,
Hans

On Ubuntu 14.04 x64 (with cmake) I get these failures at Phase3 and no
tarball:

The tarball is an issue Hans is aware of. We're fixing it.

Failing Tests (18):
     AddressSanitizer-x86_64-linux :: TestCases/Posix/readv.cc

Did you use to build compiler-rt on the previous (autoconf) releases?

I don't know. I didn't specifically enable it.

If not, this might explain why you're only seeing these errors now.

I think it was built by default before, but that the autoconf build
didn't run as many tests. Having these tests run is one of the
benefits of using the cmake build, but it of course has the downside
that we have to make them pass :slight_smile:

1) Use the autoconf build and upload if I get a tarball?
2) Disable compiler-rt (how?)?
3) Disable the -o pipefail and upload if I get a tarball?
4) Something else?

I don't think the test failures are correlated to not getting a
tarball. It's probably the "rpath problem" that Dimitry mentioned
further up in the thread. Try using his patch for it, or disabling -o
pipefail.

And please see if you can dig out some more details about the failing
tests and get bugs filed.

Thanks,
Hans

If r242543 fixes it, that's great. Otherwise, feel free to exclude
compiler-rt or fall back to autoconf (doesn't that need unwind.h too,
though?) - whatever is most appropriate for FreeBSD.

Thanks,
Hans

Another new problem with the test-release.sh script is that it can cause the whole machine (!) to run out of memory, when comparing phase 2 with phase 3, due to the following fragment:

  470 echo "# Comparing Phase 2 and Phase 3 files"
  471 for p2 in `find $llvmCore_phase2_objdir -name '*.o'` ; do
  472 p3=`echo $p2 | sed -e 's,Phase2,Phase3,'`
  473 # Substitute 'Phase2' for 'Phase3' in the Phase 2 object file in
  474 # case there are build paths in the debug info. On some systems,
  475 # sed adds a newline to the output, so pass $p3 through sed too.
  476 if ! cmp --ignore-initial=16 <(sed -e 's,Phase2,Phase3,g' $p2) \
  477 <(sed -e '' $p3) > /dev/null 2>&1 ; then
  478 echo "file `basename $p2` differs between phase 2 and phase 3"
  479 fi
  480 done

Because cmp(1) on FreeBSD does not support the --ignore-initial option, which is a GNU extension, the command immediately fails. Then, the <(...) constructs on lines 476 and 477 spawn two new bash instances per iteration, and these never get cleaned up, at least not on FreeBSD. This may very well be a bash bug.

Alternatively, the 'skip' values can be specified as the third and fourth argument on the cmp(1) command line, and this works on both Linux, FreeBSD and OSX; e.g. this:

           if ! cmp -s <(sed -e 's,Phase2,Phase3,g' $p2) <(sed -e '' $p3) \
                   16 16 ; then

That seems like a good solution, feel free to commit that.

Committed just that change in r242721.

I wonder why the first 16 bytes need to be skipped though. Does anyone know?

Probably to skip the ELF e_ident field [1], but I'm unsure why this
would be different for stages 2 and 3. This was added by Bill Wendling
in r142173; CC'ing him so he may enlighten us, if he is available.

-Dimitry

[1] ELF Header

...

The line can be replaced with this instead:

     rpath=`objdump -x $Candidate | sed -ne'/RPATH/{s/^ *RPATH *//;p;}'`

How about just guarding it with an if statement? I'm worried about
adding non-obvious commands to the script, both for making it harder
to understand and for portability.

It should be portable enough, as this works with with BSD and GNU sed,
but it may indeed be harder to understand. So how about this:

Index: test-release.sh