How to write a regression test case?

hi,all:
i really want to how to write a regression test case,but i do not find
out a entire document about it.
By reading LLVM Testing Infrastructure Guide and FileCheck - Flexible
pattern matching file verifier, i only have a idea,but it is not
enough for me to write a test case.i need more detail about how to
write a RUN:lines.

thanks for you scan,wish your letter.

best wishes,

changcheng

Hi, changcheng,

There are following steps to be taken to write a test case:
1. create a directory in test, say "test/XXX"
2. create a file named lit.local.cfg, which is a configuration file.
The straightforward way is to copy an existed one,
    like "test/CodeGen/SPARC/lit.local.cfg". You might need make some
modification on it.
3. create a file, say "example.ll"
    (The name of file does not matter, but the suffix of the file name
DOES matter. See config.suffixes = [ ... ] in lit.local.cfg)
    for more details.
    example.ll includes normal content of LLVM IR, and the ";RUN ..."
directives at the beginning of the file.
4. run all regression test cases with "make check", which includes
above test case newly added.
    Or you might want to run just that case with "llvm-lit test/XXX"
or "llvm-lit test/XXX/example.ll"
    llvm-lit resides in BUILD tree, distinguished from source tree and
install tree. You can find it under
    your-build-tree/Release+Asserts/bin/ or
your-build-tree/Debug+Asserts/bin/ or likewise, which depends on your
build
    configuration.

By the way, in the following ";RUN ..." directive
; RUN: llc < %s -march=x86 | FileCheck %s
%s indicates current file under test. I guess the first %s stands for
the input for llc, while the second and last %s specifies
what to check using "; CHECK ..." directives since you write them down
on exactly CURRENT file.

Hope it work for you.
Regards.

Hi,Yang
thanks for your entire answer,i will do it follow you.
still another question puzzled me:i write a hello.c file like this:

//RUN: llc -march=c < %s | FileCheck %s
#include"stdio.h"

int main() {
  printf ("Helloworld.\n");
  return 0;
}

in fact, i want to thanslate it to a hello.ll file like this:
;RUN: llc -march=c < %s | FileCheck %s
.......
;CHECK.....

but,when i thanslate the hello.c with clang,the sentence "//RUN: llc
-march=c < %s | FileCheck %s" was treated as comment.

it samed the method translating hello.c to hello.ll is wrong,isnot it?

To translate a .c file to a .ll file, you should wirte, for example,
      clang -emit-llvm -S test.c -o test.ll

And the corresponding command for regression test might be:
     ; RUN: clang < %s -emit-llvm -S | FileCheck %s

Regards.

(PS. When you reply to this email, make sure you send a copy to
llvmdev@cs.uiuc.edu)

Hi,Yang
thanks for your entire answer,i will do it follow you.
still another question puzzled me:i write a hello.c file like this:

//RUN: llc -march=c < %s | FileCheck %s
#include"stdio.h"

int main() {
  printf ("Helloworld.\n");
  return 0;
}

in fact, i want to thanslate it to a hello.ll file like this:
;RUN: llc -march=c < %s | FileCheck %s
.......
;CHECK.....

but,when i thanslate the hello.c with clang,the sentence "//RUN: llc
-march=c < %s | FileCheck %s" was treated as comment.

it samed the method translating hello.c to hello.ll is wrong,isnot it?

To translate a .c file to a .ll file, you should wirte, for example,
      clang -emit-llvm -S test.c -o test.ll

And the corresponding command for regression test might be:
     ; RUN: clang < %s -emit-llvm -S | FileCheck %s

Sorry for my carelessness. The method mentioned above does not work at all.
An alternative way is as follows:
First compile .c files into .ll files, then modify .ll files to do
regression test.

You can modify the Makefile under current directory.

Hi,Yang
thanks for your patient answers,i will try just like you mentioned.

Hi,Yang
thanks for your entire answer,i will do it follow you.
still another question puzzled me:i write a hello.c file like this:

//RUN: llc -march=c < %s | FileCheck %s
#include"stdio.h"

int main() {
  printf ("Helloworld.\n");
  return 0;
}

in fact, i want to thanslate it to a hello.ll file like this:
;RUN: llc -march=c < %s | FileCheck %s
.......
;CHECK.....

but,when i thanslate the hello.c with clang,the sentence "//RUN: llc
-march=c < %s | FileCheck %s" was treated as comment.

it samed the method translating hello.c to hello.ll is wrong,isnot it?

To translate a .c file to a .ll file, you should wirte, for example,
      clang -emit-llvm -S test.c -o test.ll

And the corresponding command for regression test might be:
     ; RUN: clang < %s -emit-llvm -S | FileCheck %s

We can write:
// RUN: clang < %s -emit-llvm -S
in a .c file to test clang.

And if we use FileCheck, like:
// RUN: clang < %s -emit-llvm -S | FileCheck %s
make sure there is some "// CHECK:" in that file, or lit will report a failure.

See utils/lit/lit/ExampleTests/Clang for more details.

I did an experiment just now with a test case like:
// RUN: clang %s -S -emit-llvm -o %t.ll
// RUN: llc %t.ll -o %t.s
// RUN: FileCheck < %t.s %s

...

// CHECK: ...

And it worked.

It seems clang does not take stdin as its input, so "clang < %s" fails.
Meanwhile, both "llc < %t.ll" and "llc %t.ll" work.

the example is more like what i need,it is so nice.
but,i am indefinite if "RUN: FileCheck < %t.s %s" can pass,i
understand that t.s was translate from t.ll,%s means read the local
source,are they the same?

the example is more like what i need,it is so nice.
but,i am indefinite if "RUN: FileCheck < %t.s %s" can pass,i
understand that t.s was translate from t.ll,%s means read the local
source,are they the same?

%t indicates a temporary file for output. ".s" is just a suffix I used
conventionly.
You can replace it with any string you like.

See entry "tmp" in http://llvm.org/docs/TestingGuide.html#rtvars

yeah,i see.
i thought FileCheck check if the same string in the two file,is it?
the file t.s maybe do not have the same string with %s,so FileCheck
perhaps give a fail.
i am not sure about it.

hi,Yang:
if you can give me a examle,which check out a string(such as "abcd")
from a file(such as aaa.c) with fileCheck?
i tried to write it but failed.

thanks for your help!

best wished!

changcheng

Wang,
Attachment is a simple example, you may put it in test/CodeGen, and run with
llvm-lit C90

It worked ok in my llvm 3.2 devel. Let know if you come into any trouble.

Regards.

Wang,

Attachment is a simple example, you may put it in test/CodeGen, and run with

llvm-lit C90

It worked ok in my llvm 3.2 devel. Let know if you run into any trouble.

Regards.

C90.zip (582 Bytes)

Sorry, Wang, previous attachment uses "-march=vpu -mcpu=vpu", this is
a mistake. You can replace it with any
backend available. This new attachment fixes it.

C90.zip (580 Bytes)

hi,yang:
thanks for your help!
my net is so poor that i can not donwload your attachment.
would you like to paste here,or send to my other email:200005275@163.com?

yours

changcheng

You are welcome.

/* example.c */
// RUN: clang %s -S -O0 -emit-llvm -o %t.ll
// RUN: llc %t.ll -O0 -march=x86 -o %t.s
// RUN: FileCheck < %t.s %s

// CHECK: fct
unsigned int fct(unsigned int n) {
  if(n>0) return fct(n-1)*n;
  else return 1;
}
/* end example.c */

/* lit.local.cfg */
config.suffixes = ['.c', '.cpp', '.ll']

targets = set(config.root.targets_to_build.split())
if not 'VPU' in targets:
    config.unsupported = True
/* end lit.local.cfg */

hi,yang:
another question,i write a test like this:
/*****test start******/
; RUN: llc -march=c < %s | FileCheck %s

; ModuleID = 'hello.c'
target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-
f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-
n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

@.str = private unnamed_addr constant [12 x i8] c"helloworld\0A\00", align 1

define i32 @main() nounwind uwtable {
entry:
  %retval = alloca i32, align 4
  store i32 0, i32* %retval
  %call = call i32 (i8*, ...)* @printf(i8* getelementptr inbounds ([12 x i8]*
@.str, i32 0, i32 0))
  ret i32 0
}

declare i32 @printf(i8*, ...)
;

;CHECK: main
;CHECK: printf
/******test end*******/
i run the example ok,now i what to check if the file llc ouput contain
a string "helloworld",so i add a sentence ";CHECK: {helloworld}" after
";CHECK: printf".but it failed.why?

yours,
changcheng

hi,yang:
another question£¬i write a test like this:
/*****test start******/
; RUN: llc -march=c < %s | FileCheck %s

; ModuleID = 'hello.c'
target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-
f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-
n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

@.str = private unnamed_addr constant [12 x i8] c"helloworld\0A\00", align 1

define i32 @main() nounwind uwtable {
entry:
  %retval = alloca i32, align 4
  store i32 0, i32* %retval
  %call = call i32 (i8*, ...)* @printf(i8* getelementptr inbounds ([12 x i8]*
@.str, i32 0, i32 0))
  ret i32 0
}

declare i32 @printf(i8*, ...)
;

;CHECK: main
;CHECK: printf
/******test end*******/
i run the example ok,now i what to check if the file llc ouput contain
a string "helloworld",so i add a sentence ";CHECK: {helloworld}" after
";CHECK: printf".but it failed.why?

Why do you check "{helloworld}" rather than "helloworld"?

If you are not sure a string pattern does exist in result file,
compile it manually and check the content before
adding a CHECK to test it.

hi,yang:

thanks,i replaced "{helloworld}" with "helloworld" as you said,and the
test run well on llvm version 3.0,but fail on llvm 162227,is it right?
llvm version 162227 do not support it?

your,
changcheng

aha,i add cBackend into llvm version 162227,my test run passed.
it so hapyy!
thanks for all of you!

still a question:
what does "; PR1028" means?

best wishes,

changcheng