Hey Aart and everyone, I have started a revision that creates a bare minimal build necessary to get started writing actual benchmarks for sparse kernel. The revision is here. The sequence of steps we need to do to implement benchmarks for sparse kernel are outlined by Aart here which I am copying inline (1) investigating the typical way in which LLVM at large integrates benchmarks (2) find interesting sparse kernels to implement and measure (3) integrate such tests in a continuous build (or at least frequently run system) The patch I linked to strives to achieve the first objective here. Once we are done here, we can start talking about what benchmarks to implement and so on. Thank you, Saurabh

Thanks for starting the discussion! Quick first reply: since you are really introducing a benchmarking framework (not just sparse), may I suggest you rename the subject to e.g. just “MLIR Benchmarks”. Then, sparse benchmarks can be a subfolder within this newly proposed framework.

Hey everyone, I am working on introducing benchmarks for MLIR in general and for SparseKernel in particular. So far, I have explored two ways of doing it: Using google benchmark library. Using python’s timeit library. Here’s a summary of the discussion I had with @mehdi_amini and @aartbik on the patch To use google benchmark library, we can write C++ execution tests but we need to improve the runtime to make this approach viable. We can set up and run MLIR IR from python programs. Thus, we can use python libraries to do the execution and report the timings. I believe we might go down the python route. What do people think about this? I am also not sure whether these benchmarks should be included in llvm-lit’s discovery path.

[image] SaurabhJha: I believe we might go down the python route. What do people think about this? Strong +1 on my end, seamless integration into a language/runtime/library ecosystem that provides easy to use libraries is a must for fast iteration, reproducibility and just showing the usefulness of MLIR in general. We have been operating under the assumption that this type of infra is a bridge too far for the MLIR / LLVM repo so we have been building our own out of tree for a few months. This is also the place where the sparse compiler benchmarks started before forking off to their separate place that also brings in TACO. Recently, the winds have shifted, motivated in large part by realizing that this is possible, necessary for reuse and overdue. There are a few orthogonal aspects here that I’ll happily unpack once there is reasonable confidence that this is welcome in-tree. The MVP for me that would show this is possible is having a base reusable harness that we can all contribute to and per-project/dialect entry points. Our harness is built around the concept of a “ProblemDefinition” that makes it easy to create a new benchmark that plays nicely with it. It specifies: how to create and initialize numpy and MLIR tensors (incl. alignment), the compute/bandwidth characteristics of the benchmark a check function that compares the MLIR JIT’ed code with a known correct impl. (e.g. in numpy) an entry point to metaprogram MLIR from python under a given context manager. The ProblemDefinition itself is not ideal: it requires too much human intervention and should be automated; the current state is as far as we’ve been willing to push it so far for our use cases. IMO, a “test for success” of a basic setup in-tree would be that we can just delete the existing code and reuse what would be in-tree. The flow I describe above is tailored atm for a “custom op” programming model and has a minimal runtime/compiler contract to specify what “bufferization” should do as well as various compiler strategies and basic “transformations wrapping infra” that are not appropriate for other projects. So if you go with a model similar to what I described above, let’s be sure to iterate closely so we can surface what is generally reusable and portable vs what is too specific. Below is a snapshot of what you can expect from the basic stuff that exists today: > python -m python.examples.matmul.bench ############################################################### Compile-time problem size {'m': 2040, 'n': 2041, 'k': 2042} Runtime problem size {'m': 2040, 'n': 2041, 'k': 2042} Problem types [ , , ] Compilation expert compilation in 0.2015s xxxxxxxxxx : 10 iters time on 1 threads ------------------------------------------------------------------------------------------------------------------------ slowest p1 p10 p25 p50 p75 p90 p99 fastest unit ------------------------------------------------------------------------------------------------------------------------ 2.3e-01 2.3e-01 2.2e-01 2.2e-01 2.2e-01 2.2e-01 2.1e-01 2.1e-01 2.1e-01 seconds 73.58 73.58 76.24 76.49 78.15 78.76 82.07 82.07 82.07 GFlops/s 0.29 0.29 0.30 0.30 0.31 0.31 0.32 0.32 0.32 GBs/s Compilation expert compilation in 0.322s xxxxxxxxxx : 10 iters time on 1 threads ------------------------------------------------------------------------------------------------------------------------ slowest p1 p10 p25 p50 p75 p90 p99 fastest unit ------------------------------------------------------------------------------------------------------------------------ 1.6e-01 1.6e-01 1.6e-01 1.6e-01 1.6e-01 1.6e-01 1.5e-01 1.5e-01 1.5e-01 seconds 103.70 103.70 103.74 103.78 105.14 106.28 112.44 112.44 112.44 GFlops/s 0.41 0.41 0.41 0.41 0.41 0.42 0.44 0.44 0.44 GBs/s ############################################################### Compile-time problem size {'m': -1, 'n': 2041, 'k': -1} Runtime problem size {'m': 2040, 'n': 2041, 'k': 2042} Problem types [ , , ] Compilation expert compilation in 0.2068s xxxxxxxxxx : 10 iters time on 1 threads ------------------------------------------------------------------------------------------------------------------------ slowest p1 p10 p25 p50 p75 p90 p99 fastest unit ------------------------------------------------------------------------------------------------------------------------ 2.1e-01 2.1e-01 2.0e-01 2.0e-01 1.9e-01 1.9e-01 1.9e-01 1.9e-01 1.9e-01 seconds 82.80 82.80 86.25 86.40 88.16 88.68 89.35 89.35 89.35 GFlops/s 0.32 0.32 0.34 0.34 0.35 0.35 0.35 0.35 0.35 GBs/s Compilation expert compilation in 0.3505s xxxxxxxxxx : 10 iters time on 1 threads ------------------------------------------------------------------------------------------------------------------------ slowest p1 p10 p25 p50 p75 p90 p99 fastest unit ------------------------------------------------------------------------------------------------------------------------ 1.7e-01 1.7e-01 1.7e-01 1.7e-01 1.6e-01 1.5e-01 1.5e-01 1.5e-01 1.5e-01 seconds 101.42 101.42 101.94 102.05 108.79 111.31 113.19 113.19 113.19 GFlops/s 0.40 0.40 0.40 0.40 0.43 0.44 0.44 0.44 0.44 GBs/s ############################################################### Compile-time problem size {'m': -1, 'n': -1, 'k': -1} Runtime problem size {'m': 2040, 'n': 2041, 'k': 2042} Problem types [ , , ] Compilation expert compilation in 0.2182s xxxxxxxxxx : 10 iters time on 1 threads ------------------------------------------------------------------------------------------------------------------------ slowest p1 p10 p25 p50 p75 p90 p99 fastest unit ------------------------------------------------------------------------------------------------------------------------ 2.1e-01 2.1e-01 2.0e-01 2.0e-01 2.0e-01 2.0e-01 1.9e-01 1.9e-01 1.9e-01 seconds 80.91 80.91 83.69 83.76 85.70 87.13 90.24 90.24 90.24 GFlops/s 0.32 0.32 0.33 0.33 0.34 0.34 0.35 0.35 0.35 GBs/s Compilation expert compilation in 0.344s xxxxxxxxxx : 10 iters time on 1 threads ------------------------------------------------------------------------------------------------------------------------ slowest p1 p10 p25 p50 p75 p90 p99 fastest unit ------------------------------------------------------------------------------------------------------------------------ 1.7e-01 1.7e-01 1.7e-01 1.7e-01 1.6e-01 1.6e-01 1.6e-01 1.6e-01 1.6e-01 seconds 100.42 100.42 100.65 100.71 105.42 106.16 108.50 108.50 108.50 GFlops/s 0.39 0.39 0.39 0.39 0.41 0.42 0.43 0.43 0.43 GBs/s

[image] SaurabhJha: Using python’s timeit library. On the specific use of python infra, this can easily get noisy and hard to control, esp. at low latencies. So we built our own where the harness synthesizes a timing loop that calls a lower-level C timing library and fills an array of timing values. When done, we run basic stats on those higher confidence numbers in python. There would be a lot of interest on our end for help plugging-in perf counter harvesting in there and start generating some roofline plots: we have a few cases where we suspect we are bound by the frontend and I was made aware recently that llvm-mca does not model that.

Also: Benchmarking tips — LLVM 16.0.0git documentation

For benchmarking purposes, we have added this wrapper https://github.com/llvm/llvm-project/blob/e90630e5a501fc1ef3b8e56e9d35bb6588c4eea1/mlir/lib/ExecutionEngine/RunnerUtils.cpp#L79 around C++ high-resolution clock. It is used in several downstream projects as follows: take the MLIR function you want to benchmark, emit a “benchmark” function that takes the same arguments as the original function, calls the timer, the original function, and the timer again before computing the difference, return the results of the original function + the measured time. This sounds a bit complicated at a glance, but is really a dozen lines. It is also flexible enough to integrate with whatever code that will be calling this function. I have used it from Python. Regarding running Python tests from lit, it is mostly straightforward and is already in use for Python binding tests. lit itself is driven in Python, and the test discovery is literally a glob by filename extension. The only important thing is to have a local configuration in a directory that marks the tests as unsupported for lit if the user has not requested benchmarking, this will need some cmake/lit communication similar to what is done here https://github.com/llvm/llvm-project/blob/main/mlir/test/python/lit.local.cfg and here https://github.com/llvm/llvm-project/blob/main/mlir/test/lit.site.cfg.py.in#L50.

[image] nicolasvasilache: The flow I describe above is tailored atm for a “custom op” programming model Could you explain more what you mean here? I take it as it is currently targeted at smallish programs that can be meta-programmed at the single op level. What you have is really nice for that level of work. I’ve struggled with the relation to continuity of tooling between that and the larger scale, and we have a lot of gaps there. Currently, we end up reducing such problems down to benchmark in this kind of harness when we have a problem or need to do focused development, but that leaves a lot to be desired. At the larger scale, ime, the primary differences are: You are rarely meta-programming: you are working from programs that something higher level has produced. You may not have a convenient place to just insert a timing loop. Instead, you want some kind of interface to surface and aggregate metrics (which may be coming from different devices). You tend to be more interested in metrics that indicate total system performance that goes beyond single invocations (ie. How much overhead between repeat invocations, compute unit idle time, peak memory usage over time, etc). Amatorization of startup costs. You want to drill down into these lower level metrics when there is a problem or opportunity. In my experience, these are rarely served by the same tooling and development process, but they do benefit from a continuity of tooling and approach – and right now are more disconnected than I think we ultimately want. But I struggle with concrete steps to grow one up and grow the other down towards a common center.

[image] stellaraccident: In my experience, these are rarely served by the same tooling and development process, but they do benefit from a continuity of tooling and approach – and right now are more disconnected than I think we ultimately want. But I struggle with concrete steps to grow one up and grow the other down towards a common center. Yes, this is an instance of alternating between top-down and bottom-up investigations. I am starting to see some glimmers on the horizon / have a few ideas for bridges to get us closer. The steps outlined are indeed for O(1) op but already help reduce ambiguity and surprise. They are by no means sufficient but IMO they are necessary for solving the bigger problem progressively, without surprises. I did take the post as looking at benchmarking at the O(1) ops level since it was mentioning SparseKernel, but I agree the larger space is much more ambiguous: the tooling I personally like in this space is nvprof/nsight but that is definitely a different level.

So many good comments :slightly_smiling_face:. Here’s what I understand from the above discussion: We should call these benchmarks from python. We should include them in llvm-lit with CMake flags to include/exclude them. The directory structure I am thinking about is a top level directory inside something like mlir/benchmark/python that contains all the code for a test harness and then the subdirectories for the benchmarks of different dialects. What we need to decide on is what approach are we taking for test harness. Should we go with the iree-llvm-sandbox suggested by @nicolasvasilache or RunnerUtils suggested by @ftynse. We can probably get started with something lightweight on a toy benchmark and see if it works There were a lot of things I couldn’t understand in the discussion so far like “meta programmed at the op level” or “bufferization”. It will hopefully make more sense to me as I spend more time working on this project.

MLIR benchmarks

MLIR

SaurabhJha December 6, 2021, 7:59pm 3

Sure thing, done!

Topic		Replies	Views
Using Google Benchmark Library LLVM Dev List Archives	7	418	May 30, 2018
Sparse Compiler Starter Projects MLIR	10	1219	December 14, 2021
Check-mlir test suite times: Sparse tensor integration tests MLIR	10	807	June 12, 2023
Benchmarks for LLVM-generated Binaries LLVM Dev List Archives	20	472	September 16, 2016
Check-mlir times: `Examples/standalone` testing time MLIR	28	1698	April 16, 2022

MLIR benchmarks

Related topics