From here: http://parsec.cs.princeton.edu
It is open-source and redistributable, and I have added LICENSE.TXT files to all the pieces I want to merge. These are blackscholes, canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions. I will disable the tests by default on the initial merge and test thoroughly on Linux and MacOS before enabling them. May I please commit my changes directly to test-suite?
Thanks,
--Patrick
Patrick,
Patrick,
It is open-source and redistributable, and I have added LICENSE.TXT
files to all the pieces I want to merge. These are blackscholes,
canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions. I
will disable the tests by default on the initial merge and test
thoroughly on Linux and MacOS before enabling them. May I please commit
my changes directly to test-suite?
Awesome! Can we see a patch?
The PARSEC webpage mentions that the benchmark suite is quite large.
Does your patch actually commit a specific version to the test-suite,
or is it setup like the externals tests where the user is expected to
download PARSEC on the side if they wish to test it?
I'm not convinced that the PARSEC tests are appropriate for addition the LLVM testsuite, because they are multithreaded tests. The testsuite needs to be low-volatility in terms of performance variation, as well as easy to verify correctness, and easy for the compiler developers to debug miscompilations. I don't think PARSEC is a good match on these fronts.
I don't agree, more tests are always better. We can always invent
infrastructure to ignore the performance of certain tests if that
becomes an issue.
- Daniel
Daniel Dunbar wrote:
Patrick,
It is open-source and redistributable, and I have added LICENSE.TXT
files to all the pieces I want to merge. These are blackscholes,
canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions. I
will disable the tests by default on the initial merge and test
thoroughly on Linux and MacOS before enabling them. May I please commit
my changes directly to test-suite?
Awesome! Can we see a patch?
Hi! I'm the person that asked Patrick to email llvmdev about integrating PARSEC. We needed to make PARSEC work with test-suite because we use test-suite as the foundation of our testing infrastructure, and since we've gone through the trouble to make it work, we might as well let others benefit from our effort.
I advised Patrick to *not* send a patch because it would be very large and mostly contain PARSEC source code. However, I've since realized that we can place it on our web site and send out a URL for it. Patrick, I'll let you know how to do that.
The PARSEC webpage mentions that the benchmark suite is quite large.
Does your patch actually commit a specific version to the test-suite,
or is it setup like the externals tests where the user is expected to
download PARSEC on the side if they wish to test it?
We can integrate PARSEC either as a regular internal test with its source code inside the test suite or as an external test like SPEC, depending on what people think is best. We wanted to know if there were objections to either approach.
I'm not convinced that the PARSEC tests are appropriate for addition the LLVM testsuite, because they are multithreaded tests. The testsuite needs to be low-volatility in terms of performance variation, as well as easy to verify correctness, and easy for the compiler developers to debug miscompilations. I don't think PARSEC is a good match on these fronts.
I don't agree, more tests are always better. We can always invent
infrastructure to ignore the performance of certain tests if that
becomes an issue.
As an aside, I'll offer up my opinion on the purpose of test-suite:
IMHO, the test suite is for the LLVM community. Benchmarking LLVM is one of its uses. However, I think the test suite is also useful for stress testing LLVM's correctness and for research (e.g., we use the SPEC and MultiSource/Benchmark tests in our papers). As long as it does not grow too large, I think test-suite should continue to fill these needs. If it does grow too large (or some tests cause problems), we can split it into several smaller test suites tailored to different subgroups of the community.
So, with that said, are there still objections to integrating it as an internal test? If not, I'll ask Patrick to create a patch, and we'll send a URL for it. If there are still objections, would integrating it as an external test (like SPEC) be okay?
-- John T.
My plan was to commit a specific version to test-suite. The exact programs I want to commit are blackscholes, canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions.
John has mentioned the possibility of my putting a patch on a website somewhere, so I will do that. The patch is really too large to post here.
--Patrick
Daniel Dunbar wrote:
Patrick,
It is open-source and redistributable, and I have added LICENSE.TXT
files to all the pieces I want to merge. These are blackscholes,
canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions. I
will disable the tests by default on the initial merge and test
thoroughly on Linux and MacOS before enabling them. May I please commit
my changes directly to test-suite?Awesome! Can we see a patch?
Hi! I'm the person that asked Patrick to email llvmdev about integrating
PARSEC. We needed to make PARSEC work with test-suite because we use
test-suite as the foundation of our testing infrastructure, and since we've
gone through the trouble to make it work, we might as well let others
benefit from our effort.I advised Patrick to *not* send a patch because it would be very large and
mostly contain PARSEC source code. However, I've since realized that we can
place it on our web site and send out a URL for it. Patrick, I'll let you
know how to do that.The PARSEC webpage mentions that the benchmark suite is quite large.
Does your patch actually commit a specific version to the test-suite,
or is it setup like the externals tests where the user is expected to
download PARSEC on the side if they wish to test it?We can integrate PARSEC either as a regular internal test with its source
code inside the test suite or as an external test like SPEC, depending on
what people think is best. We wanted to know if there were objections to
either approach.
I would probably object to integrating it directly, depending on the
size of code. I don't like the idea of *having* to check out a huge
test suite, just to run the SingleSource tests, for example.
OTOH, I strongly encourage integrating it as an external supplement like SPEC.
I'm not convinced that the PARSEC tests are appropriate for addition the
LLVM testsuite, because they are multithreaded tests. The testsuite needs
to be low-volatility in terms of performance variation, as well as easy to
verify correctness, and easy for the compiler developers to debug
miscompilations. I don't think PARSEC is a good match on these fronts.I don't agree, more tests are always better. We can always invent
infrastructure to ignore the performance of certain tests if that
becomes an issue.As an aside, I'll offer up my opinion on the purpose of test-suite:
IMHO, the test suite is for the LLVM community. Benchmarking LLVM is one of
its uses. However, I think the test suite is also useful for stress testing
LLVM's correctness and for research (e.g., we use the SPEC and
MultiSource/Benchmark tests in our papers). As long as it does not grow too
large, I think test-suite should continue to fill these needs. If it does
grow too large (or some tests cause problems), we can split it into several
smaller test suites tailored to different subgroups of the community.
Yes, I generally agree with this philosophy, with the caveat as above
that I think keeping large bodies of code in external supplements is
easier to manage.
So, with that said, are there still objections to integrating it as an
internal test? If not, I'll ask Patrick to create a patch, and we'll send a
URL for it. If there are still objections, would integrating it as an
external test (like SPEC) be okay?
Can we start by integrating it externally? I think it would be fine to
check the external bits into another part of the LLVM repo, I just
don't want to be required to grab it whenever I (or my buildbot
minions) grab llvm-test-suite.
- Daniel
It's not *that* big. The patch is 634K uncompressed. I'd say it would be annoying to have that attached to an email message, but it's no bigger than existing MultiSource tests.
--Patrick
It's not *that* big. The patch is 634K uncompressed. I'd say it would
be annoying to have that attached to an email message, but it's no
bigger than existing MultiSource tests.
Oh, sure, thats totally fine. I read the "3GB" on the website and got scared. 
- Daniel