An LLVM 1.3 Request

Project Infrastructure LLVM Dev List Archives
1

Hi List,

There is an outstanding bug, #257, pertaining to separating out the
detailed testing infrastructure from the main LLVM code. I would like to
see fixed in the 1.3 release time frame. The bug can be found here:

http://llvm.cs.uiuc.edu/bugs/show_bug.cgi?id=257

I would have taken this one and done the work necessary but
unfortunately I don't have CVS rights and this is almost completely a
CVS job.

There are a number of ancillary tasks that need to be done in
conjunction with this task. Things like:

- change configure.ac to _not_ support options that affect testing only.
  For example: --enable-spec200, --enable-spec95, --enable-povray.

- change Makefiles to not build "test"

- decide which portion(s) of test/Regression remain or are moved.

I would like to volunteer to help with these things but it doesn't make
sense without someone else to do the necessary CVS changes that go with
it.

The end goal of this change is to minimize the size of a standard
distribution of LLVM and make configuration and building easier.

What's the likelihood that someone could do this for 1.3, or the
likelihood that I could get CVS access and do it myself?

Reid.

2

There is an outstanding bug, #257, pertaining to separating out the
detailed testing infrastructure from the main LLVM code. I would like to
see fixed in the 1.3 release time frame. The bug can be found here:
http://llvm.cs.uiuc.edu/bugs/show_bug.cgi?id=257

I think that we would all like to see this bug fixed for 1.3. :slight_smile: That
said, we've all been really busy with a variety of things, and I don't
think there are any definite plans to do it.

I would have taken this one and done the work necessary but
unfortunately I don't have CVS rights and this is almost completely a
CVS job.

Well there are two big parts to it: changing the configure scripts and
makefiles to work differently, and actually moving the files. The biggest
part is changing the configure/makefile machinery... by contrast, moving
the directory is easy. :slight_smile:

There are a number of ancillary tasks that need to be done in
conjunction with this task. Things like:

- change configure.ac to _not_ support options that affect testing only.
  For example: --enable-spec200, --enable-spec95, --enable-povray.

- change Makefiles to not build "test"

- decide which portion(s) of test/Regression remain or are moved.

I would like to volunteer to help with these things but it doesn't make
sense without someone else to do the necessary CVS changes that go with
it.

Yes, definitely. I think the right way to do this is to make it be as
incremental as possible. In particular, the biggest benefit will be to
get llvm/test/Programs into a seperate tarball from the main LLVM tree, as
it is big and will (hopefully!) keep getting bigger. There are several
tasks that can be done incrementally to help out with this:

1. Make test/Programs be self contained, including not depending on
   makefiles in directories above it.
2. It should have its own (small) configure script, just like the other
   projects do.
3. It should "act" as if it were in llvm/projects/TestPrograms or
   something. Before the files are actually moved, this can be simulated
   with a symlink.

Once these three are done, it should just be a matter of moving the CVS ,v
files from llvm/test/Programs into llvm/projects/TestPrograms, and then
into a seperate CVS module when that works goes well.

I attached a small draft of one vision of a future LLVM directory layout
to the end of PR257. Please take a look and add a comment if you think
that something else would make sense.

The end goal of this change is to minimize the size of a standard
distribution of LLVM and make configuration and building easier.

This is *clearly* a useful change :slight_smile:

What's the likelihood that someone could do this for 1.3,

At this point, I would not say that it's particularly likely that someone
will do it straight out for 1.3. If you'd like to send in some patches
for the three items above, we would be happy to apply them, and when the
time comes, do the big move. If you get it to the point where we just
have to move the files, I can guarantee that it will be done by 1.3. :slight_smile:

or the likelihood that I could get CVS access and do it myself?

At this point, sending in patches is probably the easiest thing to do.
Figuring out how to set up the permissions, deal with the politics of
giving external access to university machines, and other annoying details
is just not something that is very appealing to me, and the actual CVS
work is pretty small.

-Chris

3

Yes, definitely. I think the right way to do this is to make it be as
incremental as possible. In particular, the biggest benefit will be to
get llvm/test/Programs into a seperate tarball from the main LLVM tree, as
it is big and will (hopefully!) keep getting bigger. There are several
tasks that can be done incrementally to help out with this:

1. Make test/Programs be self contained, including not depending on
   makefiles in directories above it.
2. It should have its own (small) configure script, just like the other
   projects do.
3. It should "act" as if it were in llvm/projects/TestPrograms or
   something. Before the files are actually moved, this can be simulated
   with a symlink.

This is completely opposite the way I work. If I had CVS access, I would
do the whole change in a branch. This would include fixing all
makefiles, moving all files, creating/fixing configure scripts, and
running all regressions to make sure it worked. I would then check in
the branch, merge mainline into it again and re-test. Once I was synched
with mainline and everything checked out, I'd commit again. The branch
would be verified independently by someone else before being merged back
to mainline.

Trying to do a global change in a piecemeal fashion will just create
pain for more people. For example, how would I supply a patch to
autoconf/configure.ac that removes support for, say --enable-spec2000
and that still lets you configure for testing? Attempting to do this
piecemeal (or "incrementally" as you put it :slight_smile: will just lead to
confusion for everyone. Trust me, I've been there and done that.

Once these three are done, it should just be a matter of moving the CVS ,v
files from llvm/test/Programs into llvm/projects/TestPrograms, and then
into a seperate CVS module when that works goes well.

I agree that moving the cvs files is a last step, but its also something
that needs to be done incrementally as the solution is built in a
branch.

I attached a small draft of one vision of a future LLVM directory layout
to the end of PR257. Please take a look and add a comment if you think
that something else would make sense.

Saw it, commented on it.

> The end goal of this change is to minimize the size of a standard
> distribution of LLVM and make configuration and building easier.

This is *clearly* a useful change :slight_smile:

Especially if you are over dialup or DSL :slight_smile:

At this point, sending in patches is probably the easiest thing to do.
Figuring out how to set up the permissions, deal with the politics of
giving external access to university machines, and other annoying details
is just not something that is very appealing to me, and the actual CVS
work is pretty small.

Unfortunately, I won't be able to do that for this kind of project. The
logistics of trying to keep track of a multitude of patches while at the
same time keeping the build tree clean for other purposes is beyond the
kind of time I have available. The reality of my work schedule is that
I get very few concentrated periods of time to work on things. The way I
handle this is by creating a branch that is separate from other things
where I can work on a related set of changes. Other sets of changes are
in other branches.

I'm willing to help, but it has to be efficient with my time too.

Thanks,

Reid.

4

It makes you wish we were using bitkeeper or arch, doesn't it? :slight_smile:

5

> Yes, definitely. I think the right way to do this is to make it be as
> incremental as possible. In particular, the biggest benefit will be to
> get llvm/test/Programs into a seperate tarball from the main LLVM tree, as
> it is big and will (hopefully!) keep getting bigger. There are several
> tasks that can be done incrementally to help out with this:
>
> 1. Make test/Programs be self contained, including not depending on
> makefiles in directories above it.
> 2. It should have its own (small) configure script, just like the other
> projects do.
> 3. It should "act" as if it were in llvm/projects/TestPrograms or
> something. Before the files are actually moved, this can be simulated
> with a symlink.
>
This is completely opposite the way I work. If I had CVS access, I would
do the whole change in a branch. This would include fixing all
makefiles, moving all files, creating/fixing configure scripts, and
running all regressions to make sure it worked. I would then check in
the branch, merge mainline into it again and re-test. Once I was synched
with mainline and everything checked out, I'd commit again. The branch
would be verified independently by someone else before being merged back
to mainline.

Trying to do a global change in a piecemeal fashion will just create
pain for more people. For example, how would I supply a patch to
autoconf/configure.ac that removes support for, say --enable-spec2000
and that still lets you configure for testing? Attempting to do this
piecemeal (or "incrementally" as you put it :slight_smile: will just lead to
confusion for everyone. Trust me, I've been there and done that.

I guess I don't understand why making these changes is not
straight-forward and incremental. It seems as though #1 is just strictly
local changes to llvm/test/Programs/Makefile.programs, with no other
changes. This can be tested and worked on exactly as you mentioned, and
there should be zero breakage in the process.

Once that is done, the configure script can be split into two pieces,
again, without breaking anything. Once the makefiles and configure script
are changed, it shouldn't matter where the directory lives.

I'm willing to help, but it has to be efficient with my time too.

I definitely appreciate both your help and the time you are able to spend
on this. I certainly don't want to waste your time at all, but I don't
see how this needs to be done as a single monolithic change.

Unfortunately, CVS isn't designed for distributed development (cue the
"switch to subversion" people here :), so I don't see that as an option,
at least in the short term...

-Chris

6

In the interest of good developer/institution relations, all I can say
is: "No comment." :slight_smile:

7

I guess I don't understand why making these changes is not
straight-forward and incremental. It seems as though #1 is just strictly
local changes to llvm/test/Programs/Makefile.programs, with no other
changes. This can be tested and worked on exactly as you mentioned, and
there should be zero breakage in the process.

Not quite. The configure script contains hard coded partial paths from
the root of the tree to all Makefiles, including those in
llvm/test/Programs directory. I can't go changing the locations of files
and the contents of the makefiles without also changing configure.ac. If
I change configure.ac then it affects everyone. If it affects everyone,
I need to do it on a branch. I can't do that without cvs access.

Once that is done, the configure script can be split into two pieces,
again, without breaking anything. Once the makefiles and configure script
are changed, it shouldn't matter where the directory lives.

So, what you're saying is that I provide you with a patch that excises
all of test/Programs/... from the llvm configure.ac and removes all the
related --enable-xyz options. You check this in. Then what? You've now
disabled your testing environment waiting for a subsequent patch from me
that restores it in "some other directory". Or, were you going to check
the configure.ac change into a branch. If so, then I have to work out
of a branch that you're administering and we have synchronization
problems. This is the part that gets time consuming.

I can't retain things in both worlds because autoconf won't configure
anything that isn't in a directory that is a descendant of
${top_srcdir}.

Gack.

Can't we just be subversive and use subversion? :slight_smile:

You invited that comment :slight_smile:

Reid.

8

> I guess I don't understand why making these changes is not
> straight-forward and incremental. It seems as though #1 is just strictly
> local changes to llvm/test/Programs/Makefile.programs, with no other
> changes. This can be tested and worked on exactly as you mentioned, and
> there should be zero breakage in the process.
>
Not quite. The configure script contains hard coded partial paths from
the root of the tree to all Makefiles, including those in
llvm/test/Programs directory. I can't go changing the locations of files
and the contents of the makefiles without also changing configure.ac. If
I change configure.ac then it affects everyone. If it affects everyone,
I need to do it on a branch. I can't do that without cvs access.

By far the *biggest* change that needs to be made is to get the
test/Programs makefile independent of the rest of the makefile stuff.
At least this *certainly* can be done by just changing Makefile.programs.

> Once that is done, the configure script can be split into two pieces,
> again, without breaking anything. Once the makefiles and configure script
> are changed, it shouldn't matter where the directory lives.

So, what you're saying is that I provide you with a patch that excises
all of test/Programs/... from the llvm configure.ac and removes all the
related --enable-xyz options. You check this in.

No, I would check in two patches, the one to remove it, and the one to add
it in the new location.

Then what? You've now disabled your testing environment waiting for a
subsequent patch from me that restores it in "some other directory". Or,

Why not check in both patches at the same time?

were you going to check the configure.ac change into a branch. If so,
then I have to work out of a branch that you're administering and we
have synchronization problems. This is the part that gets time
consuming.

I can't retain things in both worlds because autoconf won't configure
anything that isn't in a directory that is a descendant of
${top_srcdir}.

I have no idea about the configure issues, as I (intentionally :slight_smile: know
very little about it. Just the makefile changes alone, which I do
understand, are clearly a starting point.

Can't we just be subversive and use subversion? :slight_smile:
You invited that comment :slight_smile:

I know. Perhaps someday we can make the change. In fact, there are
CVS->subversion gateways. If there was a CVS->subversion gateway,
couldn't you do your development on a subversion branch from the gateway?

-Chris

9

I know. Perhaps someday we can make the change. In fact, there are
CVS->subversion gateways. If there was a CVS->subversion gateway,
couldn't you do your development on a subversion branch from the gateway?

That's an interesting idea that I'll look into. However, I'm pretty
certain any such gateway would still require me to log n to CVS or it
would be a "read only" gateway which wouldn't be much use.

One thing I *could* do is set up a gateway with subversion so that it
keeps itself synchronized with CVS, do all my changes, etc. and then
when you guys are ready to move to subversion just donate the whole
(new) repository back to the LLVM project. Would you be interested in
that?

Reid.