LNT ClamAV - Sorting output

Hi all,

Looking into ClamAV, it seems that the output is highly asynchronous, so a simple diff is obviously bogus. Since the output is mainly a report of what’s happening, without any logical sequence, I can sort both outputs and diff them, for a similar (though not perfect) result. If you sort clamscan.out-nat and clamscan.out-simple, you’ll see that they’re identical.

I’m not sure if there is an option to run the program in sequential mode (if there is, the test will pass), but if not, the simplest way to fix the diff is to sort the reference_output and the program’s output on execution.

Adding “| sort” to the RUN_OPTIONS didn’t have the desired effect (that was a long shot, anyway), does anyone know of a way to sort the output of a program on those Makefiles? Maybe a rule that can be intermediary of the final rule (I’m guessing $PROGRAM) that I can add a few steps before the diff?

cheers,
–renato

Hi Renato,

What is it that makes the output of the program asynchronous? The output is deterministic on Darwin, so it seems like it should be possible to make it more stable.

I’m not really a fan of doing any more transformations on the output than we have to, because it just pushes complexity into the test suite infrastructure.

  • Daniel

What is it that makes the output of the program asynchronous? The output
is deterministic on Darwin, so it seems like it should be possible to make
it more stable.

This is a virus scan and, AFAICS, depends on the order in which the INODEs
are laid out in the directory. I'm not sure there is a way to sort the
files before, I'll look into that.

I'm not really a fan of doing any more transformations on the output than
we have to, because it just pushes complexity into the test suite
infrastructure.

I agree, that is the last resort.

cheers,
--renato

What is it that makes the output of the program asynchronous? The output
is deterministic on Darwin, so it seems like it should be possible to make
it more stable.

This is a virus scan and, AFAICS, depends on the order in which the INODEs
are laid out in the directory. I'm not sure there is a way to sort the
files before, I'll look into that.

Ok, that seems ideal if possible.

Looping in Edvin who wrote Clamscan. :slight_smile:

- Danil

        What is it that makes the output of the program asynchronous? The output is deterministic on Darwin, so it seems like it should be possible to make it more stable.

    This is a virus scan and, AFAICS, depends on the order in which the INODEs are laid out in the directory. I'm not sure there is a way to sort the files before, I'll look into that.

Ok, that seems ideal if possible.

You can pass all the filenames from the inputs/ directory directly on the command-line, instead of specifying -r inputs/.
That way the order of scanning will be exactly the one specified on the command-line.

Otherwise clamscan just does a readdir, so the order is entirely FS-dependent, similarly with clamd which does a sort by inode.

Looping in Edvin who wrote Clamscan. :slight_smile:

Thats an overstatement, although I wrote a fair amount of libclamav :slight_smile:

Best regards,
--Edwin

Hum, I think I can fix that with Make...

--renato

Hi Torok,

I’ve used a hard-coded list on the input parameter and still got some output (slightly) scrambled between two different bots…

INPUT = $(PROJ_SRC_DIR)/inputs/clam.cab
$(PROJ_SRC_DIR)/inputs/clamdoc.tar.gz
$(PROJ_SRC_DIR)/inputs/clam.exe
$(PROJ_SRC_DIR)/inputs/clam.exe.bz2
$(PROJ_SRC_DIR)/inputs/clam-v2.rar
$(PROJ_SRC_DIR)/inputs/clam-v3.rar
$(PROJ_SRC_DIR)/inputs/clam.zip
$(PROJ_SRC_DIR)/inputs/README
$(PROJ_SRC_DIR)/inputs/rtf-test/Doc11.rtf
$(PROJ_SRC_DIR)/inputs/rtf-test/Doc1.rtf
$(PROJ_SRC_DIR)/inputs/rtf-test/Doc22.rtf
$(PROJ_SRC_DIR)/inputs/rtf-test/Doc2.rtf
$(PROJ_SRC_DIR)/inputs/rtf-test/doc3.rtf
$(PROJ_SRC_DIR)/inputs/rtf-test/docCLAMexe.rtf
$(PROJ_SRC_DIR)/inputs/rtf-test/rtf1.rtf
$(PROJ_SRC_DIR)/inputs/rtf-test/rtf-novirus.rtf

RUN_OPTIONS = --debug --exclude-dir .svn --verbose -d$(PROJ_SRC_DIR)/dbdir -r $(INPUT)

This is what Make generated for me on both machines:

Output/clamscan.simple --debug --exclude-dir .svn --verbose
-d/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/dbdir
-r
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/clam.cab
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/clamdoc.tar.gz
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/clam.exe
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/clam.exe.bz2
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/clam-v2.rar
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/clam-v3.rar
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/clam.zip
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/README
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/Doc11.rtf
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/Doc1.rtf
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/Doc22.rtf
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/Doc2.rtf
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/doc3.rtf
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/docCLAMexe.rtf
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/rtf1.rtf
/home/user/devel/llvm/test/test-suite/MultiSource/Applications/ClamAV/inputs/rtf-test/rtf-novirus.rtf

I though the dbdir could be the culprit, but it has only one file. Attached is the output of both.

cheers,
–renato

ClamAV-output.zip (5.25 KB)

The version of ClamAV in the LLVM test-suite is quite old, and it first unpacks the database file, and then loads each one (newer versions
would load the database directly, without unpacking to a tmpdir):

Loading databases from %s
in cli_cvdload()
MD5(.tar.gz) = %s
in cli_untgz()
cli_untgz: Unpacking %s
cli_untgz: Unpacking %s

As a quick workaround you could remove the daily.cvd from the dbdir and put just a file called: "empty.pdb".
You wouldn't be testing ClamAV's digital signature checking code then, but everything else should still work.

--Edwin

I see. That looks like an important test (at least from a compiler point of
view), so I'll see if I can sort the list of files in the temp dir, too.

cheers,
--renato

Hi Torok,

I've used a hard-coded list on the input parameter and still got some output (slightly) scrambled between two different bots...

I though the dbdir could be the culprit, but it has only one file. Attached is the output of both.

The version of ClamAV in the LLVM test-suite is quite old,

In the interests of having relevant metrics - should we update to a
more recent version?

The reason for adding ClamAV to the test-suite was more for testing
correctness than performance. (there were quite a few GCC bugs triggered
by ClamAV's code for example).
To do a proper performance test for ClamAV you need to run it for at
least half an hour on a large set of representative files, i.e. not
something for LLVM. Otherwise what might seem like a 20% improvement
could very well be just a 0.2% improvement in practice.

You can try to update to a newer version, I think the script that I used
to convert ClamAV's source-code to LLVM-test-suite-friendly source code
is still in the repository, although it may need updating as some new
files probably got added upstream.

Unfortunately I don't have the time to do this update myself at this time.

--Edwin

This is (maybe to a lesser extent) what happens with most of our
benchmarks, and running them 3 times doesn't add that much confidence but
makes it run much slower. I end up treating the test-suite as functionality
and correctness test, rather than useful benchmark data.

I agree it would be great to have a decent benchmark infrastructure for
LLVM, but I'm not sure the test-suite is the appropriate place. Maybe a
different type of run that crank the inputs up to 11 and let the
applications run for longer, to be run once a week or so wouldn't be a bad
idea, though.

cheers,
--renato

A simple benchmark that we run "all the time" is how long it takes for us
to compile ourselves. Do we track this?

-- Sean Silva

The test suite tracks this, but most of the compilation times are in
seconds, and load in the machine can make it very chaotic, so it's only
relevant in a large time scale. Ex:

http://llvm.org/perf/db_default/v4/nts/graph?plot.0=7.178.3&highlight_run=10811

vs

http://llvm.org/perf/db_default/v4/nts/graph?highlight_run=10813&plot.178=10.178.3

--renato

This is (maybe to a lesser extent) what happens with most of our benchmarks, and running them 3 times doesn’t add that much confidence but makes it run much slower. I end up treating the test-suite as functionality and correctness test, rather than

useful benchmark data.

I agree it would be great to have a decent benchmark infrastructure for LLVM, but I’m not sure the test-suite is the appropriate place. Maybe a different type of run that crank the inputs up to 11 and let the applications run for longer, to be run once a

week or so wouldn’t be a bad idea, though.

Performance benchmarking also has a coupling to the memory space layout (both code and run-time data placement) that can make individual comparisons of a single before and single after compile be unrepresentative, eg, see

http://plasma.cs.umass.edu/emery/stabilizer

Addressing this without going quite as far as the stabilizer stuff goes is turning out to be quite challenging.

Cheers,

Dave

Hi,

A simple benchmark that we run "all the time" is how long it takes for us to compile ourselves. Do we track this?

I recently wrote a simple script to help compare compile time against gcc. The script works as a wrapper around clang++ that compiles multiple times and reports the average time.

I use the script by setting CMAKE_CXX_COMPILER=cctime.py and CMAKE_CXX_COMPILER_ARG1=/path/to/clang then run make.

The script could be modified and improved to generate a json file to be submitted to lnt.

Anyway, the script is attached if anyone is interested.

paul

cctime.py (1.99 KB)