MultiSource/Applications/lemon slow in JIT

Hi,

I've been toying around with the testsuite for a bit, and after recompiling
llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually got it to
run and pass most tests.

When running, I noticed a very long runtime on the lemon test. At first I
suspected an infinite loop, but it turned out the test simply needed around
1000 seconds to run. Some investigation turned out that the lemon program does
a lot of forking (it runs every input file 20 times, so that's around 200
forks). However, since the forking happens quite early in the main function,
it happens before most of the program is JITted. So, every one of these forked
process is taking a few seconds to compile the other needed functions, over
and over again.

A solution for this problem seems to be to run the lemon test with the
-no-lazy option, so things get compiled before forking. In my case, this
reduced runtime from 1000+ seconds to only 16 seconds.

Intrestingly enough, I can't seem to find similar behaviour in the test
results on llvm.org/nightlytest (though I only checked two machines [since the
results take ages to load...] of which
only one actually ran the test). Am I the only one that is experiencing this?

Anyhow, the below patch fixes the problem for me, good to commit?

Gr.

Matthijs

Index: MultiSource/Applications/lemon/Makefile

Hi all,

Intrestingly enough, I can't seem to find similar behaviour in the test
results on llvm.org/nightlytest (though I only checked two machines [since the
results take ages to load...] of which
only one actually ran the test). Am I the only one that is experiencing this?

After finally being able to view the full test results of another machine, I
noticed that grue-x86-64 also needs 142 seconds for the test, which is not as
slow as here, but also way too slow...

Gr.

Matthijs

Hi,

I've been toying around with the testsuite for a bit, and after recompiling
llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually got it to
run and pass most tests.

When running, I noticed a very long runtime on the lemon test. At first I
suspected an infinite loop, but it turned out the test simply needed around
1000 seconds to run. Some investigation turned out that the lemon program does
a lot of forking (it runs every input file 20 times, so that's around 200
forks). However, since the forking happens quite early in the main function,
it happens before most of the program is JITted. So, every one of these forked
process is taking a few seconds to compile the other needed functions, over
and over again.

A solution for this problem seems to be to run the lemon test with the
-no-lazy option, so things get compiled before forking. In my case, this
reduced runtime from 1000+ seconds to only 16 seconds.

Intrestingly enough, I can't seem to find similar behaviour in the test
results on llvm.org/nightlytest (though I only checked two machines [since the
results take ages to load...] of which
only one actually ran the test). Am I the only one that is experiencing this?

I have noticed a timeout on lemon JIT, hadn't investigated to see whether looping or just slow, thanks!

It's true it's not failing in the nightly logs, but if the number in the "Today's JIT" column has anything to do with runtime (another Feature of that display is that it nowhere explains what any of the numbers mean), it is taking a very long time, about 6x as long as any other Multisource JIT run. Is the timeout disabled in those runs somehow?

I don't know anything about the test, but your patch works for me.

Hi,

I've been toying around with the testsuite for a bit, and after recompiling
llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually got it to
run and pass most tests.

When running, I noticed a very long runtime on the lemon test. At first I
suspected an infinite loop, but it turned out the test simply needed around
1000 seconds to run. Some investigation turned out that the lemon program does
a lot of forking (it runs every input file 20 times, so that's around 200
forks). However, since the forking happens quite early in the main function,
it happens before most of the program is JITted. So, every one of these forked
process is taking a few seconds to compile the other needed functions, over
and over again.

A solution for this problem seems to be to run the lemon test with the
-no-lazy option, so things get compiled before forking. In my case, this
reduced runtime from 1000+ seconds to only 16 seconds.

Intrestingly enough, I can't seem to find similar behaviour in the test
results on llvm.org/nightlytest (though I only checked two machines [since the
results take ages to load...] of which
only one actually ran the test). Am I the only one that is experiencing this?

Anyhow, the below patch fixes the problem for me, good to commit?

Ouch. I didn't notice this. I don't think passing this option is the right fix though. Any test that forks a lot is simply not a good compiler benchmark. The right thing to do is to eliminate the forking. Is that cannot be easily done. We should remove it from our test suite.

Thanks,

Evan

Evan Cheng wrote:

Hi,

I've been toying around with the testsuite for a bit, and after
recompiling
llvm-gcc a bunch of times and fixing a nasty bashism bug, I actually
got it to
run and pass most tests.

When running, I noticed a very long runtime on the lemon test. At
first I
suspected an infinite loop, but it turned out the test simply needed
around
1000 seconds to run. Some investigation turned out that the lemon
program does
a lot of forking (it runs every input file 20 times, so that's
around 200
forks). However, since the forking happens quite early in the main
function,
it happens before most of the program is JITted. So, every one of
these forked
process is taking a few seconds to compile the other needed
functions, over
and over again.

A solution for this problem seems to be to run the lemon test with the
-no-lazy option, so things get compiled before forking. In my case,
this
reduced runtime from 1000+ seconds to only 16 seconds.

Intrestingly enough, I can't seem to find similar behaviour in the
test
results on llvm.org/nightlytest (though I only checked two machines
[since the
results take ages to load...] of which
only one actually ran the test). Am I the only one that is
experiencing this?

Anyhow, the below patch fixes the problem for me, good to commit?
    
Ouch. I didn't notice this. I don't think passing this option is the
right fix though. Any test that forks a lot is simply not a good
compiler benchmark. The right thing to do is to eliminate the forking.
Is that cannot be easily done. We should remove it from our test suite.
  
The forking was introduced because the runtime was too small; the
program keeps global state, so there is no easy way to reset state.
I would prefer to remove the forking (and get short runtimes again),
rather than removing it completely from the testsuite.
What do you think?

Best regards.
--Edwin

I think this direction is wrong. I'd rather have some infrastructure that can get nanoseconds or tick counters and print them... This then allows for a microbenchmark suite, which I think fits in nicely with the llvm philosophy.

No, this is completely inappropriate for llvm-test.

-Chris

I agree with Evan, that removing the forking is best. Even if lemon isn't useful as a perf test, it would be useful to keep it as a correctness test. For this, a short runtime is ok. I'd rather have it with a short runtime than just removing it entirely.

-Chris