Is there room for another build system?

2 more roadblocks for Visual Studio users are the inability to compile gcc and the inability to compile and run the test suite. I would not want to submit a change unless I could still compile/run gcc and pass the test suite. (Testing before submission is the way we do it where I come from - I am assuming it’s the same here).

On a related note, has anyone gotten the LLVM test suite working on MinGW? I am in the process of trying to get LLVM working on my Windows box. I expect I can get the test suite working on Cygwin - but I’m not sure about MinGW.

Message: 6

"Jonathan Brumley" <jbrumley@gmail.com> writes:

2 more roadblocks for Visual Studio users are the inability to compile gcc
and the inability to compile and run the test suite. I would not want to
submit a change unless I could still compile/run gcc and pass the test
suite. (Testing before submission is the way we do it where I come from - I
am assuming it's the same here).

Completely agreed.

However, for VC++, I'm not thinking on LLVM contributors, but on LLVM
users.

On a related note, has anyone gotten the LLVM test suite working on
MinGW? I am in the process of trying to get LLVM working on my
Windows box. I expect I can get the test suite working on Cygwin -
but I'm not sure about MinGW.

People run the GCC test suite on MinGW and, IIRC, LLVM uses the same
framework, so it should be doable.

Jonathan Brumley wrote:

2 more roadblocks for Visual Studio users are the inability to compile gcc and the inability to compile and run the test suite. I would not want to submit a change unless I could still compile/run gcc and pass the test suite. (Testing before submission is the way we do it where I come from - I am assuming it's the same here).

On a related note, has anyone gotten the LLVM test suite working on MinGW? I am in the process of trying to get LLVM working on my Windows box. I expect I can get the test suite working on Cygwin - but I'm not sure about MinGW.

The LLVM test suite requires expect as a dependency of DejaGNU. The stock source of expect requires a *NIX fork, so it's relatively difficult to build under MinGW even though Tk/Tcl is easy to build under MinGW. (I'm not aware of a precompiled MinGW binary for expect, although it might be in one of the more obscure packages). I'd be interested in a successful build (with patches) for expect myself.

I've been thinking of constructing a mirror test suite coordinated using shell scripts (bash) to replace DejaGNU. (This is for maximal portability. For Windows, batch files are another option; however, they have some limitations relative to shell scripts.)

This is rather solidly in the "feasibility study/vaporware" stage for LLVM, although I've had moderate success testing the ideas with batch files on an internal project.

Kenneth

Kenneth Boyd <zaimoni@zaimoni.com> writes:

The LLVM test suite requires expect as a dependency of DejaGNU. The
stock source of expect requires a *NIX fork, so it's relatively
difficult to build under MinGW even though Tk/Tcl is easy to build
under MinGW. (I'm not aware of a precompiled MinGW binary for expect,
although it might be in one of the more obscure packages). I'd be
interested in a successful build (with patches) for expect myself

About two years ago, ActiveState added the Windows port of Expect to its
Tcl distribution, which is the de-facto canonical source of Tcl/Tk for
Windows users. Check www.activestate.com

[snip]

Hi Kenneth,

If the LLVM project is switching to CMake, then CTest might be the framework of choice to use rather than scripting up something in Bash.

--Sam

Hi!

I've used Cmake since one year, and even with LLVM. I haven't compiled
it with cmake of course, but projects using it. I think it is easy to
maintain and I felt very comfortable with it.
It can do every thing I've ever needed and when something rare is
missing you can eval an isolated bash script...

Samuel Crow wrote:

Hi Kenneth,

If the LLVM project is switching to CMake, then CTest might be the framework of choice to use rather than scripting up something in Bash.
  

I wouldn't care to try to defend the time overhead in rewriting the test cases for another framework. Thus, I'm requiring my proposed alternate test framework to use DejaGNU test cases. The bash scripting is simply so that I can *use* the test cases.

Also, I'm not seeing support for platform-specific testing controlled from within the testcase (DejaGNU feature) openly documented for CTest. (I do not think the Dashboard approach is robust enough.) So even if CTest could use the existing test cases verbatim, it looks harder to maintain.

Kenneth

Óscar Fuentes wrote:

About two years ago, ActiveState added the Windows port of Expect to its
Tcl distribution, which is the de-facto canonical source of Tcl/Tk for
Windows users. Check www.activestate.com
  

And at such time as I get approval to correctly patch the current build system to work with ActivePerl, the patch should be ready within a couple of days.

Kenneth

Kenneth Boyd wrote:

Óscar Fuentes wrote:

About two years ago, ActiveState added the Windows port of Expect to its
Tcl distribution, which is the de-facto canonical source of Tcl/Tk for
Windows users. Check www.activestate.com
  

And at such time as I get approval to correctly patch the current build system to work with ActivePerl, the patch should be ready within a couple of days.

Hmm...that didn't read right. "Submit a patch for consideration, to correctly patch the current build system to work with ActivePerl".

Kenneth

Please no, pick a real language. I love bash and it is fine for any 200 line or less program, but for much beyond that, something else. Python?c? C++?

Mike Stump wrote:

  

I've been thinking of constructing a mirror test suite coordinated using
shell scripts (bash)
    
Please no, pick a real language. I love bash and it is fine for any 200 line or less program, but for much beyond that, something else. Python?c? C++?
  

If I thought C was an appropriate choice to get this done, I'd be patching the expect source code instead to deal with Microsoft's non-implementation of POSIX.

If a language conceals the differences between POSIX and Microsoftian C for interprocess control, hides badly non-POSIX filepaths from me, and has a reasonable track record of forward compatibility, and has a reasonably universal build process, I'll consider crash-learning the language if I'm not already working in it.

[I do some programming in Python already. Python lost out because it hasn't been forward-compatible on my in-house projects; broken non-deprecated features have forced maintenance on the 2.2 |-> 2.3, 2.3

-> 2.4, and 2.4 |-> 2.5 version jumps. Annoying; the subprocess module

is an ideal match.]

Kenneth

Kenneth Boyd <zaimoni@zaimoni.com> writes:

I've been thinking of constructing a mirror test suite coordinated
using shell scripts (bash)

Please no, pick a real language. I love bash and it is fine for any
200 line or less program, but for much beyond that, something else.
Python?c? C++?
  

If I thought C was an appropriate choice to get this done, I'd be
patching the expect source code instead to deal with Microsoft's
non-implementation of POSIX.

If a language conceals the differences between POSIX and Microsoftian C
for interprocess control, hides badly non-POSIX filepaths from me, and
has a reasonable track record of forward compatibility, and has a
reasonably universal build process, I'll consider crash-learning the
language if I'm not already working in it.

For my test suite I use Tcl (with TclX, no Expect). It watches stdout
and stderr, gets exit codes and has a timer for killing hanged
processes. Process control works the same on Windows and Unix and takes
a less than 30 lines of code.

What else do you need?

Óscar Fuentes wrote:

For my test suite I use Tcl (with TclX, no Expect). It watches stdout
and stderr, gets exit codes and has a timer for killing hanged
processes. Process control works the same on Windows and Unix and takes
a less than 30 lines of code.

What else do you need?
  

Testing historical forward compatibility will be routine, as well.

Kenneth

A way to examine asm output and compate to expected patterns.

A way to run performance regression tests (i.e. looking at CPU time and
allowing for fuzz between test runs).

In my experience Tcl is very hard to work with due to the non-existence of
any reasonable debugging tools on all platforms.

                                                    -Dave

David Greene <dag@cray.com> writes:

For my test suite I use Tcl (with TclX, no Expect). It watches stdout
and stderr, gets exit codes and has a timer for killing hanged
processes. Process control works the same on Windows and Unix and takes
a less than 30 lines of code.

What else do you need?

A way to examine asm output and compate to expected patterns.

This is a text manipulation task, isn't it? No problem.

A way to run performance regression tests (i.e. looking at CPU time
and allowing for fuzz between test runs).

Do you mean that you want a language with that feature built-in?

In my experience Tcl is very hard to work with due to the
non-existence of any reasonable debugging tools on all platforms.

For one reason or another, my Tcl code never grows so large that I miss
a debugger. A `puts' here and there plus the repl is enough :slight_smile:

David Greene <dag@cray.com> writes:
>> For my test suite I use Tcl (with TclX, no Expect). It watches stdout
>> and stderr, gets exit codes and has a timer for killing hanged
>> processes. Process control works the same on Windows and Unix and takes
>> a less than 30 lines of code.
>>
>> What else do you need?
>
> A way to examine asm output and compate to expected patterns.

This is a text manipulation task, isn't it? No problem.

No problem with Perl either, or Python. Tcl is much less well-known.

Note that I don't particularly like any of these languages but I'm trying
not to let personal preference get in the way. :slight_smile:

> A way to run performance regression tests (i.e. looking at CPU time
> and allowing for fuzz between test runs).

Do you mean that you want a language with that feature built-in?

No, I mean in the future we should have tests that actually pass/fail based
on their runtime performance. To do that you need a way to time the test
and a way to account for normal system variations (the fuzz bit).

We don't have any of these kinds of tests yet. But I hope we do in the
future.

> In my experience Tcl is very hard to work with due to the
> non-existence of any reasonable debugging tools on all platforms.

For one reason or another, my Tcl code never grows so large that I miss
a debugger. A `puts' here and there plus the repl is enough :slight_smile:

How many people know Tcl? That has a direct impact on maintanability.

                                            -Dave

David Greene <dag@cray.com> writes:

No problem with Perl either, or Python. Tcl is much less well-known.

Note that I don't particularly like any of these languages but I'm trying
not to let personal preference get in the way. :slight_smile:

Neither I do. This sub-thread started with a discussion about the
feasibility of leaving behing DejaGNU and using `bash' instead or:

> If a language conceals the differences between POSIX and Microsoftian C
> for interprocess control, hides badly non-POSIX filepaths from me, and
> has a reasonable track record of forward compatibility, and has a
> reasonably universal build process, I'll consider crash-learning the
> language if I'm not already working in it.

Then I proposed Tcl as an example of such language. Maybe Perl, Python,
Ruby, Lua and others are okay too, but:

[snip]

How many people know Tcl? That has a direct impact on maintanability.

DejaGNU is built around Expect, wich is a Tcl extension, so LLVM is
already using Tcl. Tcl is pretty simple. You can learn Tcl in minutes,
although there are some very common pitfalls (related to quoting issues,
etc).

OTOH, right now I'm reading llvm-config and GenLibDeps.pl, wich are
Perl. I don't know Perl at all, but more or less I understand what I'm
reading (1). This is because whoever wrote those scripts did it well. Had
similar experiences with Python: from knowing nothing at all to
extending functionality of an existing large application in minutes.

So the real issue, IMHO, is not the language itself, but *who* does the
job. To a large extent, he has the right to choose the tool he
prefers. If he does a good job, rejecting it on the basis of the
language used would be foolish, as anyone could maintain the code with
minimal effort.

(1) Fortunately, I have no need to understand the code that has this
comment on top of it:

# To understand this code, you'll need a working knowledge of Perl 5,
# and possibly some quality time with 'man perlref'.

David Greene wrote:

  

Kenneth Boyd <zaimoni@zaimoni.com> writes:
    

I've been thinking of constructing a mirror test suite coordinated
using shell scripts (bash)
          

Please no, pick a real language. I love bash and it is fine for any
200 line or less program, but for much beyond that, something else.
Python?c? C++?
        

If I thought C was an appropriate choice to get this done, I'd be
patching the expect source code instead to deal with Microsoft's
non-implementation of POSIX.

If a language conceals the differences between POSIX and Microsoftian C
for interprocess control, hides badly non-POSIX filepaths from me, and
has a reasonable track record of forward compatibility, and has a
reasonably universal build process, I'll consider crash-learning the
language if I'm not already working in it.
      

For my test suite I use Tcl (with TclX, no Expect). It watches stdout
and stderr, gets exit codes and has a timer for killing hanged
processes. Process control works the same on Windows and Unix and takes
a less than 30 lines of code.

What else do you need?
    
....

In my experience Tcl is very hard to work with due to the non-existence of
any reasonable debugging tools on all platforms.
  

That is as much a matter of preferred techniques as anything. For any given language, not having debugging tools on any platform doesn't bother me exactly as long as I can enact design-by-contract.

What I haven't seen yet in any language (but think would be easier to add to CLang than GCC; I'm not familiar enough with the other open-source competition to say), is reliably intercepting undefined behavior at compile-time. This pretty much requires whole-program analysis.

Kenneth

David Greene wrote:

  

David Greene <dag@cray.com> writes:
    

For my test suite I use Tcl (with TclX, no Expect). It watches stdout
and stderr, gets exit codes and has a timer for killing hanged
processes. Process control works the same on Windows and Unix and takes
a less than 30 lines of code.

What else do you need?
        

A way to examine asm output and compate to expected patterns.
      

This is a text manipulation task, isn't it? No problem.
    
No problem with Perl either, or Python. Tcl is much less well-known.

Note that I don't particularly like any of these languages but I'm trying
not to let personal preference get in the way. :slight_smile:
  

I already use both Python and Perl. I ruled out Python, even though it is a very good match feature-wise (subprocess module For The Win), because my in-house applications (about 600 lines of code, maybe 800 lines of code and comments) are not nearly as forward-compatible when avoiding deprecated constructions as the public relations statements imply; something has broken on *every* major release since I started using it (2.2 |-> 2.3. |-> 2.4 |-> 2.5).

My coding style just finds those breakages that aren't caught in Python's regression testing :slight_smile: I'm fine with this for small applications, but reimplementing the testcase runner looks like it's going to be large enough to make the 2.5 |-> 2.6 change a large breakage.

A way to run performance regression tests (i.e. looking at CPU time
and allowing for fuzz between test runs).
      

Do you mean that you want a language with that feature built-in?
    
No, I mean in the future we should have tests that actually pass/fail based
on their runtime performance. To do that you need a way to time the test and a way to account for normal system variations (the fuzz bit).

We don't have any of these kinds of tests yet. But I hope we do in the future.
  

Agreed. This (and the lack of a reliable cross-platform way to grab both stdout and stderr) is what has me nervous about using Perl.

Kenneth

Óscar Fuentes wrote:

David Greene <dag@cray.com> writes:

No problem with Perl either, or Python. Tcl is much less well-known.

Note that I don't particularly like any of these languages but I'm trying
not to let personal preference get in the way. :slight_smile:
    
Neither I do. This sub-thread started with a discussion about the
feasibility of leaving behing DejaGNU and using `bash' instead or:

If a language conceals the differences between POSIX and Microsoftian C
for interprocess control, hides badly non-POSIX filepaths from me, and
has a reasonable track record of forward compatibility, and has a
reasonably universal build process, I'll consider crash-learning the
language if I'm not already working in it.
      
Then I proposed Tcl as an example of such language. Maybe Perl, Python,
Ruby, Lua and others are okay too, but:
  

No comment on Ruby (haven't worked in it enough to have an intuition for it). My opinions about Python and Perl are in an earlier reply.

I also have done some minimal work in Lua. Lua is *very* nice for simultaneously interoperating with C while supporting almost Perl-like manipulation of functions. I just don't have a gut reaction of it simplifying writing a test suite coordinator. (either embedding in C, or using C extension functions from Lua).

How many people know Tcl? That has a direct impact on maintanability.
    
DejaGNU is built around Expect, wich is a Tcl extension, so LLVM is
already using Tcl. Tcl is pretty simple. You can learn Tcl in minutes,
although there are some very common pitfalls (related to quoting issues,
etc).
  

Precisely (expect requires Tcl to build, although the releases I was failing to build are for a standalone program). The expect dependency on Tcl means that Tcl is a conservative choice of language.

Kenneth