I am using the LLVM 3.8 since it include some of the newer version of alias analysis (cfl-aa) that might be worthwhile to compare against our proposed method
Just wanted to note that CFLAA has had a bug fixed since the 3.8 release, and still isn’t enabled by default because there are probably more bugs lurking in it.
but after I ran it using command line argument such as ./opt -[some]-aa -aa-eval … it always give me a consistent answer despite the fact that I am running it with different analysis
Have you tried the -disable-basicaa flag? Sometimes we’ll use BasicAA by default, so you might be getting answers from that. 
(It seems that was added as a part of the new pass manager transition; if it’s not currently mentioned in our docs, I’ll see if I can add it)
Also, is there a way to objectively quantify (say a baseline) the result of the aa-eval since it only output the value of precision from the analysis? how do I verify that the alias is indeed alias in LLVM?
I’m parsing your question as “how do we determine if two things alias each other?” The answer to that is “alias analysis”.
- what is alias analysis mainly used for?
Looking at BasicAA’s API:
- Getting whether some statement A may be able to modify/reference some memory location B
- Getting whether two memory locations {may,must,can’t,partially} alias each other
- what type of benchmark can I run for the pointer analysis? I know there is a test-suite package but I also want to know if there are other options.
I don’t think there’s a well-defined way to do this. If you only care about execution time, opt gives you the -time-passes flag, so you can try running opt with a simple configuration (e.g. opt -basicaa -aa-eval file.ll). Depending on the size of your benchmarks and what you’re trying to benchmark, you can also probably make an aa-bench pass that hammers the AA you’re interested in. Note that some AA algorithms (specifically, I have CFLAA in mind) build up internal data structures ahead of time, so that queries take little more than a few map lookups. So, if you do end up making your own benchmarking pass, please keep that in mind.