RFC: LNT/Test-suite support for custom metrics and test parameterization

Hi all,

As we understood great changes will be done in LNT, so we are waiting to new LNT version and stopped our work in LNT.

One more question about using test-suite separately with cmake. Cmake can only build all tests and generate lit tests. After that we can run LIT and get report which is not equal with report (simple) got with make. Cmake test-suite version has no features to run custom metrics and generate other report type, right?

Are these features of make-version of test-suite planned to be added?

Thanks,

Elena.

Hi Elena,

I’m sorry to hear you’re stopping work on LNT for the moment :frowning:

Are these features of make-version of test-suite planned to be added?

The test-suite runner has support for custom metrics. What features in particular (can you be more specific?) about the Make-based test-suite do you miss in the CMake-based one?

Cheers,

James

Hi Chris,

I'm wondering if you managed to figure out exactly what the bottleneck is on loading those 100 graphs?
Is it:
* The number of bytes that need to be sent over the network between server and client - with a low ratio of useful info to useless bytes?
* A poor execution plan of the queries involved on the database side?
* Lots of small queries, resulting in latency of network traffic between client and server being the bottleneck?
* Or there is just a humongous amount of useful data that needs to be sent over the network?

I think that for each of those different causes of inefficiency, different techniques can be used to overcome them.
In a previous life, I used to work on a product doing lots of complex SQL querying from python to huge databases. I've seen lots of techniques to make big queries on relational databases go fast.
Unfortunately, I think it's not possible to use those techniques with SQLAlchemy.

If you would happen to have a bit more insight into underlying bottlenecks causing slowness of getting data from the DB,
I think that would go a long way in being able to make a more informed decision into whether the right way forward is:
* NoSQL DB
* Dropping SqlAlchemy and using SQL queries directly with a few optimization techniques not possible without using SQL queries directly.

Thanks,

Kristof

A slight diversion if you don’t mind, though related.

Currently LNT is not branched along with other components commonly used with LLVM such as ‘clang’ and ‘compiler-rt’. I generally try to synch LNT to the same SVN revision number as a corresponding LLVM branch on the possibly invalid assumption that this is the version of LNT that LLVM was tested with for that branch. Would it be possible to include LNT under the LLVM branch umbrella, especially if significant changes are being made to LNT that might not be backward or forward compatible with various branches and versions of LLVM?

Thanks,

MartinO

The lit test-suite runner supports arbitrary metrics, it already features codesizes for different segments, compiletime, linktime, executable hash, execution time. I designed it be easily extensible with further metrics. Not all of these metrics are understood by LNT yet so they may get lost after submission to an LNT database.

We do not use GenerateReport.pl and friends in the cmake/lit version anymore, as the main data collection and reporting mechanism in llvm has mostly shifted to LNT these days. In any case it should be easy to use “lit -o result.json” and process the resulting json file in your favorite scripting language to generate arbitrary reports.

As far as the TEST.*.Makefile things go: Many of those variants have no equivalent in cmake/lit. But I have the strong feeling that the majority of those is not really used or even broken so there are no plans to adding them to. On the other hand I would love to hear from people that actually use any of those besides TEST.simple.Makefile, it should be possible to transition most of them but I’d first like to hear what they are used for.

- Matthias

I think most of us live on ToT LNT. I think it is rare that the LLVM version has any impact on LNT. The only two exceptions to that are that at one point the new test-suite module needed some new lit features, and once someone broke “—version”. The core parts of LNT are pretty stable, and don’t change much.

I’d like to get people using the Bugzilla more for LNT. If something does not work for you please file a PR!

Hi Matthias,

Thank you for your answer.

But can you answer for some more questions?

First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake?

As I understood lit will run and collect all metrics, but there is no opportunity to make any settings for choosing what metrics I would like to collect. Test reports files allow to choose what report I would like. One time I can use one, second time I can use another. I can do this with cmake only by changing test_modules in file.

So I can’t group some metrics and give them some name.

Am I right?

Thanks,

Elena.

Hi Elena,

First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake?

There are two test drivers in LNT. “lnt runtests nt” uses the old Makefile-based system and “lnt runtests test-suite” uses the new cmake-based system. It sounds like you are using the former and should switch to the latter.

Cheers,

James

Hi James,

Thank you. We use last one, but I don’t know about this option, because I found no information about this in documentation. It may be very useful to add more information about modes of running tests in documentation.

Now I saw source code for test-suite mode.

Thanks,

Elena.

Hi Matthias,

Thank you for your answer.
But can you answer for some more questions?
First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake?

As James already said "lnt runtest test-suite".

As I understood lit will run and collect all metrics, but there is no opportunity to make any settings for choosing what metrics I would like to collect. Test reports files allow to choose what report I would like. One time I can use one, second time I can use another. I can do this with cmake only by changing test_modules in file.
So I can’t group some metrics and give them some name.

The test_modules flag does indeed allow some customisation in what metrics are collected, my main usage for this has been do setup things to just collect codesize but not run the benchmark, a setup in which benchmarks are cross compiled and then run on a remote device through ssh and the "default" mode which just runs all benchmarks and collects as much metrics as possible.
However lit is a tool to run the benchmarks, you still need other tools to filter analyze and present the data afterwards: The typical process we have today is to run the tests with lnt, store the results in lnt and filter/analyze with the lnt webpage. If you need custom reports then you can either add new views to lnt or write some scripts that parse the json output produced by lit. I have some personal scripts I use to compare the results of two test-suite runs because I find that easier to do ad-hoc from the commandline rather than getting lnt running just to compare two runs, I can send you those scripts as an example, however if you want something better looking than commandline output or need to process data from a lot of runs then I recommend extending lnt for your reporting needs.

- Matthias

I understood your modules and I see as them can be used in LNT. But there are some question about old features.

  1. With Makefiles we can write pass and collect some statistics. There was an example of branch pass. Metrics can be collected by

@-$(LOPT) -load dcc888$(SHLIBEXT) -branch-counter -stats \ -time-passes -disable-output $< 2>>$@

in makefile. In report file we write how data should be parsed. Can we do same things now with cmake+lit?

  1. As I saw in LNT code there is no opportunity to compile but not execute tests. Some metrics cab be collected without execution, than run can be faster.

  2. Before each group of metrics has name. Now if we would like to integrate cmake with custom metrics in LNT, we should compare all metrics of each run to make sure if this is the same test-suite or not. And we should give some random names to each group.

These moments are unclear for us now. These question are connected with first described RFC where shouldn’t be some reports which are known before, but there should be flexible system when we don’t know what metrics user would loke to collect. I will be very grateful for your answers.

Thanks,

Elena.

I understood your modules and I see as them can be used in LNT. But there are some question about old features.
1. With Makefiles we can write pass and collect some statistics. There was an example of branch pass. Metrics can be collected by
@-$(LOPT) -load dcc888$(SHLIBEXT) -branch-counter -stats \ -time-passes -disable-output $< 2>>$@
in makefile. In report file we write how data should be parsed. Can we do same things now with cmake+lit?

This question hits several points at once:
- Collecting metrics at compiletime: The Makefiles have definitely been more flexible in this regard. The flip side of the medal however is that the makefiles became so complicated that you typically needed to pass several barely documented flags to them for a simple task like compiling and running the benchmarks without any further actions.
cmake on the other hand restricts us a bit in how much we can intercept and modify the build process of the benchmark (and just to point this out: the running process in lit is no problem and really flexible).
The best worst solution I have come up with so far is using a python wrapper that understands the same arguments as clang and slightly modifies the arguments (adding stats and a providing a new output filename for the stats) before passing it along to clang. I have some personal script running which does this but it is not in a cleaned publishable state yet.

As for the reporting in the .report files: Instead of doing perl magic you can just as well read the json produced by lit and process it in your favorite scripting language. Minimalistic example in python (that just prints all metrics per benchmark):

import json
import sys
data = json.load(open(sys.argv[1]))
for test in data['tests']:
    print "%-10s:" % test['name']
    for name, value in test['metrics'].items():
        print "\t%s: %s" % (name, value)

2. As I saw in LNT code there is no opportunity to compile but not execute tests. Some metrics cab be collected without execution, than run can be faster.

The lnt test module only exposes a limited set of features possible with the cmake/lit testsuite. IMO we should break the odd relationship that lnt drives the running of the benchmark, but should just read the result file after the user has the testsuite in whatever configuration he wanted. I started a patch for this here: http://reviews.llvm.org/D19949 but I need to find some time to rework it.

If you run the test-suite manually: Just remove the "run" module in the config.test_modules list in the lit.site.cfg of your builddir. The benchmarks will no longer be executed but the codesize, hash and compiletime modules will still collect those metrics. We can add a cmake configuration parameter for this if this is a generally useful configuration.

3. Before each group of metrics has name. Now if we would like to integrate cmake with custom metrics in LNT, we should compare all metrics of each run to make sure if this is the same test-suite or not. And we should give some random names to each group.

I don't understand what you mean here, can you give an example/typical use case?

- Matthias

Thank you for answers.

About last question. Now there are compile and nt(simple) tests in LNT. We suggested to add opportunity to run and save in database any test-suite(nightly and etc test-suites described with Makefile+.report file). Each test-suite has name: simple, nightly, etc. And when we run test we choose some test-suite. Now if we use Cmake and test modules we create test-suite as we want and it has no name. Moreover, I can run tests always with different test modules. If we want to save results in LNT database we need to compare all metrics and then understood if we have such test-suite before or not.

And how should we show these test-suites?

Now there are on main page

Compile

Nt

New custom metrics should be shown to and they should have name.

One decision for this problem is to add file with description of test modules and association with this group of test modules some name.

There is no such opportunity now?

Thanks,

Elena.