Expanding the Clang Status page to report the status of several OSes and projects

Hi,

I just noticed that the table listing projects and Clang compatibility
was recently removed. This idea still has a lot of merit, and has been
breeding in my mind for several days.

I would like to provide test result data for several popular and high
quality open source libraries that provide an extensive test suite so
the compiled result is strictly analysed. I would do this on Windows
(mingw-w64 CRT+ GCC 4.6 libstdc++ and linker), but I would like this
to be more than a single email to the mailing list.

To this effect, I thought that perhaps expanding the first table that
was on http://clang.llvm.org/cxx_status.html could be expanded
significantly, in vertical (projects) and horizontal
(platforms=OS+arch(+crt in the case of Windows)).

Although I believe that Clang is production ready on Mac OS, it is far
think it is useful to keep track of wide-scale progress in a
centralized place, because I feel the bugtracker doesn't get enough
attention (at least the problems I am experiencing ;-)).

Concretely, I suggest a larger table with colors like the C++0x table,
with red = doesn't build, orange/yellow: builds, but tests fail due to
a problem in clang (which also means that the tests pass for GCC), and
green is that it builds and all tests pass without a hitch. Links
could be placed to bug reports about the issue, where detailed
information for interested developers could be placed.

The libraries I suggest that should be used as a "quality mark" are
(more are of course welcome!):

Qt: Huge GUI and core library framework. High quality code that is
extremely cross-platform. Has an internal test suite, but I haven't
yet figured out how to run it (I can't as the build fails on Windows)
wxWidgets: that other tadbit less huge GUI and core framework. Also
high quality code.
GSL: GNU Scientific library: comprehensive test suite.
FFTW: Known around the world, comprehensive test suite
Eigen: idem
Flac: idem
GMP, MPFR, MPC, PPL, CLooG: very fragile and comprehensive libraries.
Also all feature a nice test suite.
Perhaps all the open source graphics libraries (Tiff, png, jpeg, turbojpeg...)

What do you guys think? I do not know HTML, so the table editing would
have to be almost trivial for me to get it done :s. I can test
regularly and see if certain revisions fix the problem (on Windows).

Ruben

Hi,

I just noticed that the table listing projects and Clang compatibility
was recently removed. This idea still has a lot of merit, and has been
breeding in my mind for several days.

I would like to provide test result data for several popular and high
quality open source libraries that provide an extensive test suite so
the compiled result is strictly analysed. I would do this on Windows
(mingw-w64 CRT+ GCC 4.6 libstdc++ and linker), but I would like this
to be more than a single email to the mailing list.

To this effect, I thought that perhaps expanding the first table that
was on http://clang.llvm.org/cxx_status.html could be expanded
significantly, in vertical (projects) and horizontal
(platforms=OS+arch(+crt in the case of Windows)).

Although I believe that Clang is production ready on Mac OS, it is far
from my elementary trust in the Windows MinGW world. Therefore, I
think it is useful to keep track of wide-scale progress in a
centralized place, because I feel the bugtracker doesn't get enough
attention (at least the problems I am experiencing ;-)).

The main problem with the Windows MinGW world is that there are far too few LLVM/Clang developers who actively work on MinGW. Mac OS, Linux, and FreeBSD (to name a few) have a number of LLVM/Clang developers who made Clang work well on that platform and have kept it that way. We need more interest from the MinGW world for that to happen there.

Concretely, I suggest a larger table with colors like the C++0x table,
with red = doesn't build, orange/yellow: builds, but tests fail due to
a problem in clang (which also means that the tests pass for GCC), and
green is that it builds and all tests pass without a hitch. Links
could be placed to bug reports about the issue, where detailed
information for interested developers could be placed.

The libraries I suggest that should be used as a "quality mark" are
(more are of course welcome!):

Qt: Huge GUI and core library framework. High quality code that is
extremely cross-platform. Has an internal test suite, but I haven't
yet figured out how to run it (I can't as the build fails on Windows)
wxWidgets: that other tadbit less huge GUI and core framework. Also
high quality code.
GSL: GNU Scientific library: comprehensive test suite.
FFTW: Known around the world, comprehensive test suite
Eigen: idem
Flac: idem
GMP, MPFR, MPC, PPL, CLooG: very fragile and comprehensive libraries.
Also all feature a nice test suite.
Perhaps all the open source graphics libraries (Tiff, png, jpeg, turbojpeg...)

What do you guys think? I do not know HTML, so the table editing would
have to be almost trivial for me to get it done :s. I can test
regularly and see if certain revisions fix the problem (on Windows).

The problem with putting these tables up on a web page is that they get out of date very, very quickly. So, for them to be useful, they really need to be updated automatically. Then best way to do that, in my mind, would be to set up a buildbot that tests these projects. Bring the buildbot online when one of the projects starts working on that platform, and keep it running: the buildbot will tell us when things are broken, so we won't regress.

  - Doug

Hi,

I just noticed that the table listing projects and Clang compatibility
was recently removed. This idea still has a lot of merit, and has been
breeding in my mind for several days.

I would like to provide test result data for several popular and high
quality open source libraries that provide an extensive test suite so
the compiled result is strictly analysed. I would do this on Windows
(mingw-w64 CRT+ GCC 4.6 libstdc++ and linker), but I would like this
to be more than a single email to the mailing list.

To this effect, I thought that perhaps expanding the first table that
was on http://clang.llvm.org/cxx_status.html could be expanded
significantly, in vertical (projects) and horizontal
(platforms=OS+arch(+crt in the case of Windows)).

Although I believe that Clang is production ready on Mac OS, it is far
from my elementary trust in the Windows MinGW world. Therefore, I
think it is useful to keep track of wide-scale progress in a
centralized place, because I feel the bugtracker doesn't get enough
attention (at least the problems I am experiencing ;-)).

The main problem with the Windows MinGW world is that there are far too few LLVM/Clang developers who actively work on MinGW. Mac OS, Linux, and FreeBSD (to name a few) have a number of LLVM/Clang developers who made Clang work well on that platform and have kept it that way. We need more interest from the MinGW world for that to happen there.

I understand completely. The GCC world has the same problem...
unfortunately for the users. Note that MinGW shouldn't differ all that
much from MSVC for Clang. The ABI of the resulting objects is (should
be) the same. I just hope that the *-*-mingw* triplets aren't being
forgotten in important parts of the object file creation process. I
hope that at least the developers working on Windows support are aware
of where MSVC and MinGW agree on things.

Concretely, I suggest a larger table with colors like the C++0x table,
with red = doesn't build, orange/yellow: builds, but tests fail due to
a problem in clang (which also means that the tests pass for GCC), and
green is that it builds and all tests pass without a hitch. Links
could be placed to bug reports about the issue, where detailed
information for interested developers could be placed.

The libraries I suggest that should be used as a "quality mark" are
(more are of course welcome!):

Qt: Huge GUI and core library framework. High quality code that is
extremely cross-platform. Has an internal test suite, but I haven't
yet figured out how to run it (I can't as the build fails on Windows)
wxWidgets: that other tadbit less huge GUI and core framework. Also
high quality code.
GSL: GNU Scientific library: comprehensive test suite.
FFTW: Known around the world, comprehensive test suite
Eigen: idem
Flac: idem
GMP, MPFR, MPC, PPL, CLooG: very fragile and comprehensive libraries.
Also all feature a nice test suite.
Perhaps all the open source graphics libraries (Tiff, png, jpeg, turbojpeg...)

What do you guys think? I do not know HTML, so the table editing would
have to be almost trivial for me to get it done :s. I can test
regularly and see if certain revisions fix the problem (on Windows).

The problem with putting these tables up on a web page is that they get out of date very, very quickly. So, for them to be useful, they really need to be updated automatically. Then best way to do that, in my mind, would be to set up a buildbot that tests these projects. Bring the buildbot online when one of the projects starts working on that platform, and keep it running: the buildbot will tell us when things are broken, so we won't regress.

I understand, that's probably the reason it was removed in the first
place. The buildbot is an awesome idea, and LLVM/Clang wouldn't be the
first Apple sponsored project to have every commit go through a
build+test project (yes, I'm looking at you, WebKit). This kind of
large-scale infrastructure would have to be LLVM-wide to be effective,
as Clang and LLVM development are intertwined.

Are there any plans for such a buildbot and continuous testing infrastructure?

Ruben

Well, there's already

  http://google1.osuosl.org:8011/

and there are plans to get a much-expanded, much-improved version online in the near future. However, I don't know if anyone has plans to actually bring buildbots online for MinGW specifically, or introduce your suggested libraries into the mix.

  - Doug

FWIW, I have nightly builds & tests, using clang, of several open source projects that my company depends on:

CMake:
<http://www.cdash.org/CDash/index.php?project=CMake>

VTK:
<http://www.cdash.org/CDash/index.php?project=VTK>

ITK:
<http://www.cdash.org/CDash/index.php?project=Insight>

GDCM:
<http://www.cdash.org/CDash/index.php?project=GDCM>

HDF5:
<http://cdash.hdfgroup.uiuc.edu/index.php?project=HDF518>

vxl:
<http://www.cdash.org/CDash/index.php?project=vxl>

I've often wondered why there is no similar dashboard for llvm/clang, as it can already be built with cmake, it would be quite easy to start using ctest also.

Cheers,

Mail me off-list if you continue to have problems with that.

– Erik.

Buildbots that test MinGW in one way or another:

http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32
http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32-win7
http://google1.osuosl.org:8011/builders/llvm-gcc-mingw32-cross-arm-linux-gnueabi-hard-float
http://google1.osuosl.org:8011/builders/llvm-gcc-build-x86_64-darwin10-x-mingw32-x-armeabi
http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-self-mingw32
http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-cross-mingw32

What additional coverage would you like to see?

deep

Those look pretty darn good to me! :slight_smile:

  - Doug

Buildbots that test MinGW in one way or another:

http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32
http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32-win7
http://google1.osuosl.org:8011/builders/llvm-gcc-mingw32-cross-arm-linux-gnueabi-hard-float
http://google1.osuosl.org:8011/builders/llvm-gcc-build-x86_64-darwin10-x-mingw32-x-armeabi
http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-self-mingw32
http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-cross-mingw32

What additional coverage would you like to see?

Yes, i noticed these; they test the llvm/clang build, but don’t touch real world projects. This is important for a compiler, as some (most) changes won’t directly affect the internal clang build/test process. My first email on the subject suggested some external common libraries with extensive test suites that could help a build bot find more exotic bugs.

Completely orthogonal to this, is a test for MinGW-w64, including 64-bit. Or even a visual studio test bot. All these would be nice to have. I understand the later might be harder (visual studio won’t run in Unix environments…)

Thanks for the interest!

Ruben

Buildbots that test MinGW in one way or another:

http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32
http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32-win7

http://google1.osuosl.org:8011/builders/llvm-gcc-mingw32-cross-arm-linux-gnueabi-hard-float

http://google1.osuosl.org:8011/builders/llvm-gcc-build-x86_64-darwin10-x-mingw32-x-armeabi
http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-self-mingw32

http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-cross-mingw32

What additional coverage would you like to see?

Yes, i noticed these; they test the llvm/clang build, but don't touch real
world projects. This is important for a compiler, as some (most) changes
won't directly affect the internal clang build/test process. My first email
on the subject suggested some external common libraries with extensive test
suites that could help a build bot find more exotic bugs.

Would adding an LNT bot cover this?

Completely orthogonal to this, is a test for MinGW-w64, including 64-bit. Or
even a visual studio test bot. All these would be nice to have. I understand
the later might be harder (visual studio won't run in Unix environments...)

A MinG4-w64 test would be great. I wasn't under the impression that
build could even bootstrap yet.

There was an MSVC bot, but it's been down a long time.

deep

>
>>
>> Buildbots that test MinGW in one way or another:
>>
>> http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32
>> http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32-win7
>>
>> http://google1.osuosl.org:8011/builders/llvm-gcc-mingw32-cross-arm-linux-gnueabi-hard-float
>>
>> http://google1.osuosl.org:8011/builders/llvm-gcc-build-x86_64-darwin10-x-mingw32-x-armeabi
>> http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-self-mingw32
>>
>> http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-cross-mingw32
>>
>> What additional coverage would you like to see?
>
> Yes, i noticed these; they test the llvm/clang build, but don't touch real
> world projects. This is important for a compiler, as some (most) changes
> won't directly affect the internal clang build/test process. My first email
> on the subject suggested some external common libraries with extensive test
> suites that could help a build bot find more exotic bugs.

Would adding an LNT bot cover this?

You're tickling my curiosity, what's an LNT bot?

> Completely orthogonal to this, is a test for MinGW-w64, including 64-bit. Or
> even a visual studio test bot. All these would be nice to have. I understand
> the later might be harder (visual studio won't run in Unix environments...)

A MinG4-w64 test would be great. I wasn't under the impression that
build could even bootstrap yet.

Good point. Testing that now:
CMake builds with MinGW Makefiles and clang(++) as compiler fails at
link stage:
Linking CXX executable ..\..\bin\tblgen.exe
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -limagehlp.lib
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -lpsapi.lib
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -limagehlp.lib
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -lpsapi.lib
collect2: ld returned 1 exit status
clang++: error: linker (via gcc) command failed with exit code 1 (use
-v to see invocation)

Those ".lib"s shouldn't be there.

I have built LLVM/Clang with my prebuilt LLVM/Clang using MSYS and
configure. The prebuilt version I used is located here (in case you
missed it the first time):
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/rubenvb/4.6.2-1/
I configured with "/m/development/mingw64/source/llvm/configure
--host=x86_64-w64-mingw32 --disable-pthreads --enable-optimized
--disable-assertions"
make check all failed with:
make[2]: Leaving directory `/usr/home/Ruben/llvm/tools/clang/test'
( ulimit -t 600 ; ulimit -d 512000 ; ulimit -m 512000 ; ulimit -v 1024000 ; \
          /m/development/source/llvm/utils/lit/lit.py -s -v .
/usr/home/Ruben/llvm/test/../tools/clang/test )
/bin/sh: line 0: ulimit: cpu time: cannot modify limit: Invalid argument
/bin/sh: line 0: ulimit: data seg size: cannot modify limit: Invalid argument
/bin/sh: line 0: ulimit: -m: invalid option
ulimit: usage: ulimit [-SHacdfilmnpqstuvx] [limit]
/bin/sh: line 0: ulimit: virtual memory: cannot modify limit: Invalid argument
lit.py: TestingConfig.py:53: fatal: unable to load config from
'/m/development/source/llvm/test/lit.cfg'
make[1]: *** [check-local-all] Error 2

Otherwise, Clang on mingw-w64 is self-hosting (with libstdc++ and gcc
for linking of course).

What can I do to help get a mingw-w64 build bot set up?

Ruben

>
>>
>> Buildbots that test MinGW in one way or another:
>>
>> http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32
>> http://google1.osuosl.org:8011/builders/llvm-gcc-native-mingw32-win7
>>
>> http://google1.osuosl.org:8011/builders/llvm-gcc-mingw32-cross-arm-linux-gnueabi-hard-float
>>
>> http://google1.osuosl.org:8011/builders/llvm-gcc-build-x86_64-darwin10-x-mingw32-x-armeabi
>> http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-self-mingw32
>>
>> http://google1.osuosl.org:8011/builders/clang-x86_64-darwin10-cross-mingw32
>>
>> What additional coverage would you like to see?
>
> Yes, i noticed these; they test the llvm/clang build, but don't touch real
> world projects. This is important for a compiler, as some (most) changes
> won't directly affect the internal clang build/test process. My first email
> on the subject suggested some external common libraries with extensive test
> suites that could help a build bot find more exotic bugs.

Would adding an LNT bot cover this?

You're tickling my curiosity, what's an LNT bot?

http://llvm.org/docs/lnt/

> Completely orthogonal to this, is a test for MinGW-w64, including 64-bit. Or
> even a visual studio test bot. All these would be nice to have. I understand
> the later might be harder (visual studio won't run in Unix environments...)

A MinG4-w64 test would be great. I wasn't under the impression that
build could even bootstrap yet.

Good point. Testing that now:
CMake builds with MinGW Makefiles and clang(++) as compiler fails at
link stage:
Linking CXX executable ..\..\bin\tblgen.exe
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -limagehlp.lib
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -lpsapi.lib
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -limagehlp.lib
m:/development/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.6.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -lpsapi.lib
collect2: ld returned 1 exit status
clang++: error: linker (via gcc) command failed with exit code 1 (use
-v to see invocation)

Those ".lib"s shouldn't be there.

I have built LLVM/Clang with my prebuilt LLVM/Clang using MSYS and
configure. The prebuilt version I used is located here (in case you
missed it the first time):
http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Personal%20Builds/rubenvb/4.6.2-1/
I configured with "/m/development/mingw64/source/llvm/configure
--host=x86_64-w64-mingw32 --disable-pthreads --enable-optimized
--disable-assertions"
make check all failed with:
make[2]: Leaving directory `/usr/home/Ruben/llvm/tools/clang/test'
( ulimit -t 600 ; ulimit -d 512000 ; ulimit -m 512000 ; ulimit -v 1024000 ; \
/m/development/source/llvm/utils/lit/lit.py -s -v .
/usr/home/Ruben/llvm/test/../tools/clang/test )
/bin/sh: line 0: ulimit: cpu time: cannot modify limit: Invalid argument
/bin/sh: line 0: ulimit: data seg size: cannot modify limit: Invalid argument
/bin/sh: line 0: ulimit: -m: invalid option
ulimit: usage: ulimit [-SHacdfilmnpqstuvx] [limit]
/bin/sh: line 0: ulimit: virtual memory: cannot modify limit: Invalid argument
lit.py: TestingConfig.py:53: fatal: unable to load config from
'/m/development/source/llvm/test/lit.cfg'
make[1]: *** [check-local-all] Error 2

Otherwise, Clang on mingw-w64 is self-hosting (with libstdc++ and gcc
for linking of course).

What can I do to help get a mingw-w64 build bot set up?

I think the first step is to get the MinGW-w64 build to work. Submit
patches, walk them through the review process making requested
changes/improvements, and then commit them.

Once that's done, we appeal to the community to see who is willing to
dedicate a box to testing this build. Chances are good somebody is
willing to do so. I certainly have interest.

After that, we worry about LNT, additional real-world tests, etc.

deep