What should we do about simulation tests, like this one?
Verilator is the only open source simulator-like software which supports SystemVerilog (to my knowledge) but it doesn’t support simulation constructs. So in terms of running the simulations in an open pipeline, I think we’re SOL – right? Does anyone know how EDA Playground got a license?
Yeah, I’m not sure how to handle this. How much of this do you plan to build out, and are there alternative approaches that dont’ require simulation constructs?
Do you mean the RTL or the lit driver. In both cases, as little as necessary. We will, I suspect, want/need to run RTL level tests (with simulation constructs) for the integration testing for all the flows (e.g. handshake → firrtl → rtl/sv).
Yes and no. There are ways to run simulations with only verilator but they are very inconvenient for some classes of tests.
SystemVerilog is very good at (and originally designed for) cycle level simulation. It can easily handle driving signals and evaluating the outputs in parallel. With Verilator, all that needs to be done via the C++ API in software. And said testing driver software not portable to any other simulator. For any tests which need to have cycle-level awareness, Verilog is inconvenient. As you get higher up in functionality and abstraction, Verilator becomes more convenient.
When you get to the point where you’re only testing functionality, ESI’s cosimulation will come in handy. It provides an untimed, high-level interface to modules so you ignore the cycle-level interactions at the edges. It will eventually allow you to integrate into the application which is actually using the accelerator using the same API.
You could also imagine some hybrid of the two: using simulation constructs at the low-level to check the cycle-level signaling and cosim to drive the simulation and verify the results. Looks like verilator is beginning to support asserts which could cover this use-case. I don’t know how much we’d do this in the circt integration tests though – this is more something users could want to do.
Either way, we’re definitely going to want a way to run SystemVerilog tests with simulation constructs.
Aldec does have a free student edition which we could contact them about using in our pipelines. We’d need permissions to redistribute, though, in order to add it to our docker image.
Does anyone have a contact there?
Just to be clear, I know I’m asking half baked questions that are difficult to answer, the path isn’t clear here 
Sure that makes sense. I guess I’m asking about “what is the plan in the context with circt in general”. We have the opportunity/need to define specific testing tools that can be driven from lit tests and that work with filecheck, so talking to Verilator C++ API seems reasonable for such a tool to do.
We also don’t want large numbers of “unit test” style verilog tests. It can be helpful to have integration tests, but we might instead decide to follow the path of LLVM’s test-suite library, which is a separately maintained corpus of code that has optional external dependencies (e.g. SPEC) and depend on significant infra in some ways.
Figuring out the big picture here would help us make better local decisions.
-Chris
I’ve been wondering this for a while, and I figure it’s relevant to this discussion. How does LLHD fit in here? I’ve been very interested in the work on llhd-sim, and wondering how that will integrate with the rest of CIRCT.
Seems to me like we already have an open-source, cycle-accurate RTL simulator, right here in CIRCT. And it’s already being used for lit tests with filecheck.
In terms of hooking it up, I took a look just now, and it seems like the FIRRTLToLLHD conversion is the only way in-tree to get into LLHD land, and that basically just supports modules and connects at the moment. Long term, it would be really cool if more of low-FIRRTL (or maybe just RTL/SV) could be converted into LLHD for simulation.
Another thought I had related to LLHD was that we could perhaps depend on the Moore compiler like we depend on Verilator for integration testing. Then we might be able to compile that SV primitive and test case @jdd wrote with Moore, and simulate it with the llhd-sim built in tree.
CC @fabianschuiki, do you have any thoughts on how LLHD might fit in here?
Given that we have support for Verilator in the tests now, I took the liberty to put together a PR which adds Questa in an identical way: [lit] Questa support by teqdruid · Pull Request #334 · llvm/circt (github.com). This seemed like the simplest thing to do without affecting anybody who doesn’t want it. I expect this PR to spark some debate *ducks*.
Agreed. RTL simulations are slow. I think, however, for tests which are actually unit tests (e.g. testing primitives written in SystemVerilog) we should have some fast simulation tests. We should also have some quick sanity check simulations as integration tests. As for longer tests, they should probably exist in a separate repo since I could see them having some pretty big external dependencies as inputs.
@mikeurbach We certainly could (and should and are) build our own simulators (both cycle-accurate and not). But we should probably have a ground truth simulations – Verilator and at least one commercial.
I think LLHD and Moore would be an excellent fit for this. It has been designed to cover exactly this area of the hardware design process: cycle-accurate RTL simulation, including timing and all the additional stuff that comes with SV.
I see LLHD as the “least common denominator” to get an RTL simulation going. So running cycle-accurate simulations of FIRRTL could in a first step involve lowering FIRRTL to the RTL dialect, then lowering the RTL dialect to LLHD, to simulate in llhd-sim.
As a second step we could then rename llhd-sim to circt-sim, and start adding support for the RTL dialect directly, where it allows us to get better simulation performance (e.g. easier reasoning about signal dependencies, skipping the event queue where possible). An additional “LLHD-to-RTL” hoisting pass could then try to simplify LLHD to the higher-level RTL dialect where possible, to give designs that come into the CIRCT framework as LLHD (e.g. SV/VHDL) the ability to benefit from the speed gains. And suddenly we’re only a few steps away from a CIRCT alternative to Verilator
Yes please 
! I would be super excited to see Moore tied into CIRCT more closely as an alternative to Verilator for testing SV snippets. As @clattner says, we may not want a lot of SV unit tests in CIRCT directly, but building a larger body of tests in a separate repository would be amazing.
This would also further the cause of making CIRCT fully self-sustained in the sense that it also provides a complete (and probably fast, looking at a JIT-based circt-sim) flow for SV.
Fair enough. In that light, I think the PR to add Questa makes sense since Verilator isn’t sufficient.
I don’t mean to derail the discussion at hand so I’ve started a new thread about the using the LLHD simulator.
Here’s my philosophy: frequently test things that are likely to break. Sanity test things that are not. Be careful about testing time since it affects project scalability.
In this case, static blobs of verilog aren’t likely to break unless someone changes the blob, so testing for it can be “integration test” instead of “check-circt”. They just don’t have the same interdependencies that C++ libraries do.
-Chris
I think the right solution is incremental testing – only testing the stuff which changed and all the things which rely on it. I think Pytest has something like this…? I’m not pushing for this in CIRCT, just mentioning it as an interesting direction in testing. An FPGA group in Microsoft actually uses this approach for their PR gate simulations – they use my build tool to get the dependencies for each simulation + Pytest and only run simulations which may have been affected by the file changes. It’s very neat. I don’t know how much it cuts down on simulation time.
Good point.
I agree that incremental testing is the correct technical solution to this, but such a thing is a massive change to how the whole LLVM build system would have to work, and there is a long line of history there with llvmbuild, cmake, etc that is still getting the basics sorted out.
As such, I think we should build CIRCT assuming its absence, just like LLVM and the other subprojects do. If it ever appears, then we can all be overjoyed at the productivity boost, but I’m not going to hold my breath. 
-Chris