Adding SQLite3 to llvm test-suite

Hi,

SQLite3 has a very simple build system, and it comes with an extensive
testsuite (over 40000 tests).
It hasn't shown any bugs in LLVM, but it is fairly simple to build [even
turn off features via -D], and can be CPU intensive.

I have integrated SQLite3 into llvm-test's build system, it is too large
to send as attachment (784K), you can get it from here:
http://edwintorok.googlepages.com/SQLite_llvm_test.tar.gz

The testsuite is using tcl, and needs tcl headers available at
build-time so I didn't add that [they are mostly I/O bound anyway]
Instead I found a "speedtest" script in the sqlite distribution. I
modified it to generate sql files instead of directly running sqlite.

The results are here, LLVM is ~20% slower than GCC:

Program | GCCAS Bytecode LLC compile LLC-BETA compile JIT
codegen | GCC CBE LLC LLC-BETA JIT | GCC/CBE GCC/LLC
GCC/LLC-BETA LLC/LLC-BETA
lemon/lemon | 0.3833 149404 0.5699 *
0.5166 | 0.04 0.04 0.05 * 0.62 | - -
n/a n/a
sqlite3/sqlite3 | 4.8063 1744588 5.3996 *
3.7497 | 7.09 8.19 8.78 * 13.68 | 0.87 0.81
n/a n/a

There are more speedtest scripts in its testsuite, but the generated sql
is large (>10 Mb), and I think the above shows a clear difference
between LLVM and GCC.

Can somebody please review and commit this to llvm-test?

Best regards,
--Edwin

Hi Edwin,

It works fine for me. Thanks!

We prefer a flat directory structure. Is it possible for you to separate it out to sqlite3 and lemon rather than having them as sub-directories under SQLite? Also, can you increase lemon's test size?

Please file bug about the performance issue. Thanks!

Evan

Evan Cheng wrote:

Hi Edwin,
  
Hi Evan,

It works fine for me. Thanks!

We prefer a flat directory structure. Is it possible for you to
separate it out to sqlite3 and lemon rather than having them as sub-
directories under SQLite?

Ok.
The new package is here:
http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
Anything else I should change?

Also, can you increase lemon's test size?
  
Lemon doesn't have a testsuite (those 40000+ tests are all for SQLite),
however I found some projects that use lemon (its syntax is not
compatible with yacc), and copied their grammar files into lemon's
directory.
I also wrote a wrapper main function for lemon that forks/executes for
each file (lemon keeps some state in global variables, so I have to fork).

The difference isn't much, it now takes 0.12 seconds to run lemon.

Please file bug about the performance issue. Thanks!
  
Done, bug #2174.

Best regards,
--Edwin

Evan Cheng wrote:

Hi Edwin,

Hi Evan,

It works fine for me. Thanks!

We prefer a flat directory structure. Is it possible for you to
separate it out to sqlite3 and lemon rather than having them as sub-
directories under SQLite?

Ok.
The new package is here:
http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
Anything else I should change?

Looks good. Please commit.

Also, can you increase lemon's test size?

Lemon doesn't have a testsuite (those 40000+ tests are all for SQLite),
however I found some projects that use lemon (its syntax is not
compatible with yacc), and copied their grammar files into lemon's
directory.
I also wrote a wrapper main function for lemon that forks/executes for
each file (lemon keeps some state in global variables, so I have to fork).

The difference isn't much, it now takes 0.12 seconds to run lemon.

Still much too small to be really useful as a benchmark. Can you just run it 10 times?

Evan

> The new package is here:
> http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
> Anything else I should change?

Looks good. Please commit.

Should I speak with Chris about commit access?

I made the change below for lemon:
http://edwintorok.googlepages.com/sqlite_lemon_llvmtest_2.tar.gz

>
> The difference isn't much, it now takes 0.12 seconds to run lemon.

Still much too small to be really useful as a benchmark. Can you just
run it 10 times?

Ok, now it takes ~1.3 seconds.

Best regards,
--Edwin

The new package is here:
http://edwintorok.googlepages.com/sqlite_lemon_llvmtest.tar.gz
Anything else I should change?

Looks good. Please commit.

Should I speak with Chris about commit access?

I've committed it for you. Ping Chris about commit access next time.

I made the change below for lemon:
http://edwintorok.googlepages.com/sqlite_lemon_llvmtest_2.tar.gz

The difference isn't much, it now takes 0.12 seconds to run lemon.

Still much too small to be really useful as a benchmark. Can you just
run it 10 times?

Ok, now it takes ~1.3 seconds.

Thanks. I've bumped it up a bit more.

Evan

Evan Cheng wrote: