Adding a new External Suite to test-suite

Greetings LLVMers,

I would like to add the licensed part of AnghaBench
(http://cuda.dcc.ufmg.br/angha/) as an External Test in the LLVM
test-suite. We have 128,411 files with their original licenses.
(http://www.dcc.ufmg.br/~fernando/coisas/c_files_with_licenses.tar.gz)
Each file is a single function, that compiles as is—no dependencies are
needed. For instance, the commands below work for each individual
file:

$ clang $compile_flags -Xclang -disable-O0-optnone -S -c -emit-llvm
“$file” -o “$individual_bc”
$ opt -mem2reg -O0 ${user_passes[@]} -instcount -stats -S
“$individual_bc” -disable-output 2>&1"

So, these are my questions:

Q1: What do I have to do to start integrating my collection of .c files as an
External Suite? They are all single source, single function,
compilable headerless files extracted from open source repositories.
They are grouped in folders, in a similar way the original project
was.

Q2: I read that using glob might not be a good idea, even though there is a
partial workaround. However, when trying to understand what the other External
suites were doing, I noticed several of them were using it.
May I ignore the note at
https://cmake.org/cmake/help/latest/command/file.html#glob ?

Q3: The C files were not extracted/reconstructed with execution in mind.
For now, we’re mainly interested in collecting stats and testing the
compiler. Since we don’t generate executables, should I call cmake’s
add_library() for each .c file?

Kind regards,

Bruno Kind

I suggest to have a look into the CMakeLists.txt of the test-suite.
These are mostly self-explanatory.

Q1: What do I have to do to start integrating my collection of .c files as an
External Suite? They are all single source, single function,
compilable headerless files extracted from open source repositories.
They are grouped in folders, in a similar way the original project
was.

External tests are in
https://github.com/llvm/llvm-test-suite/tree/master/External.
Tests where each source file is one executable are in
https://github.com/llvm/llvm-test-suite/tree/master/SingleSource
Tests where each source file in a directory compiles to object files
that is linked to an executable are in
https://github.com/llvm/llvm-test-suite/tree/master/MultiSource

Q2: I read that using glob might not be a good idea, even though there is a
partial workaround. However, when trying to understand what the other External
suites were doing, I noticed several of them were using it.
May I ignore the note at
file — CMake 3.24.1 Documentation ?

The typical use case of the test-suite is to configure and build it
from scratch every time with a different compiler executable. There is
no typical edit-compile-debug cycle.

Q3: The C files were not extracted/reconstructed with execution in mind.
For now, we're mainly interested in collecting stats and testing the
compiler. Since we don't generate executables, should I call cmake's
add_library() for each .c file?

See compile time benchmarks in
https://github.com/llvm/llvm-test-suite/tree/master/CTMark

Are the C files intended to compile-time benchmarking as well? If not,
what are they useful for?

Michael

I suggest to have a look into the CMakeLists.txt of the test-suite.
These are mostly self-explanatory.

Q1: What do I have to do to start integrating my collection of .c files as an
External Suite? They are all single source, single function,
compilable headerless files extracted from open source repositories.
They are grouped in folders, in a similar way the original project
was.

External tests are in
https://github.com/llvm/llvm-test-suite/tree/master/External.
Tests where each source file is one executable are in
https://github.com/llvm/llvm-test-suite/tree/master/SingleSource
Tests where each source file in a directory compiles to object files
that is linked to an executable are in
https://github.com/llvm/llvm-test-suite/tree/master/MultiSource

Q2: I read that using glob might not be a good idea, even though there is a
partial workaround. However, when trying to understand what the other External
suites were doing, I noticed several of them were using it.
May I ignore the note at
file — CMake 3.24.1 Documentation ?

The typical use case of the test-suite is to configure and build it
from scratch every time with a different compiler executable. There is
no typical edit-compile-debug cycle.

Q3: The C files were not extracted/reconstructed with execution in mind.
For now, we're mainly interested in collecting stats and testing the
compiler. Since we don't generate executables, should I call cmake's
add_library() for each .c file?

See compile time benchmarks in
https://github.com/llvm/llvm-test-suite/tree/master/CTMark

Are the C files intended to compile-time benchmarking as well? If not,
what are they useful for?

I'd also like to know what the intention here is. What is tested and how?

Looking at a few of these it seems there is not much you can do as it is

little code with a lot of unknown function calls and global symbols.

Cheers,

Johannes