Clang has a few different ways to output JSON, but our current model for testing the output is brittle and fairly unreadable (both in source and the output from FileCheck). Aaron Ballman and I have been chatting about using a dedicated tool such as jd, which can show the delta fairly well, and only needs assistance if we’re in need of regular expressions (though I’m looking into that).
I’m not sure how to go about adding this dependency to llvm-zorg, because I’ve never had to interact with the buildbots. It looks like there are a few Dockerfiles that I can use to install the tool; are there any other places where I might need to do this?
As a required tool for running the test suite, or as an optional tool for nicer diffs on failure? I would have my concerns about growing a dependency on a tool written in Go given the language is less portable than our current language dependencies (C, C++ and Python).
This would be as a required tool. FileCheck isn’t particularly suitable for testing JSON, so we need something new to test this kind of output.
Is there a platform that we test on that doesn’t support Go? My understanding is that LIT has facilities to skip tests on platforms that aren’t supported, so portability might already be accounted for.
“Real” Go is missing support for GNU/Hurd, Motorola 68000 (“m68k”), 32-bit PowerPC (“powerpc”), 64-bit SPARC (“sparc64”) and ILP32 64-bit x86 (“x32”), looking at Debian’s architecture list. Some, but not all, of those have GCC Go ports. Those all ignore downstreams, too, that add support for additional architectures to LLVM (e.g. CHERI architectures in our research) not supported by any Go implementation.
A Go-based tool only has to support environments where we host LLVM, not everything where we target LLVM. I suspect there aren’t any m68k hosts out there that build/test Clang/LLVM, for instance. But still that list doesn’t look complete enough.
Thank you for bringing this up! I think we’re to the point where we need such a tool, and I’d rather use one off-the-shelf instead of developing something ad hoc for our needs. We have testing needs around compilation databases, -ast-dump=json, SARIF diagnostics in the static analyzer, and soon SARIF diagnostics in the main compiler. All of these have some pretty gnarly tests to deal with the JSON output, so I think there’s plenty of times when we could make use of a better diffing tool.
In terms of zorg questions, I’d defer to @gkistanova
I think this depends on how large of a subset we’re talking about. If we’re missing testing on one or maybe two host architectures that don’t support Go-based tools, that might be reasonable, but if it’s quite a few architectures, that’d be a problem. If there are unsupported architectures with whatever tool we decide to go with, the behavior should degrade gracefully on those architectures (might need a new kind of REQUIRES clause for lit so we can expectedly fail those tests that can’t be run on some arch).
jd happens to be the tool I knew about, but if there are other tools out there that meet our needs but don’t have the Go dependency issues, we could consider those alternatives.
Upon contrast, I’m partial to jd. It looks more mature, has substantially better documentation, a nicer API, and supports unordered diffs, which are a thing that we need. I can keep looking, but I’m not sure we’re going to find anything else that’s quite as powerful out of the box.
I’d be interested to know if GCC Go targets all our supported host.
It may be possible for us to sidestep this issue by compiling jd to WASM and then leveraging that on all systems. Is there a list of host architectures that we care about so I can cross-reference that with WASM interpreters?
Clang-based tools are able to emit JSON for various reasons, but we currently don’t have a good test framework for JSON. While there exist third-party JSON diff tools, they’re unfit either due to dependencies that can’t be installed or don’t meet our needs.
Clang needs a tool that can:
perform diffs, sometimes without regard for the order of JSON objects, and
produce output that is machine-parseable, human-readable, and resilient to change.
Of the tools surveyed, only jd seems to meet these requirements, but can’t be added due to depending on Go. It appears that adding Go as a dependency will be very tricky, so we should consider reimplementing the functionality as an LLVM utility.
Alternatives considered
In addition to surveying a few diff tools, we also considered compiling jd to WASM and:
running the WASM on an interpreter such as wasm3, and
compiling the WASM to C using wasm2c, and then compiling that to an executable.
The first alternative likely has dependency issues too, and the latter produced a C file that’s 4.5M lines, making it an unreasonable task to review.
It seems like a replacement for jd is in the same spirit, but perhaps not the same amount of effort. I’m not opposed though, as this is a testing need we’ve hit several times already. I think SARIF testing is going to be sufficiently important to need a tool (we could get away with hand waving around testing -ast-dump=json being mostly an internal tool for our own needs, but SARIF is user-facing).
Here’s a thought. IIUC we already have libraries that can inhale and exhale JSON. Rather than building a JSON diff, would it work to build a JSON canonicalizer that can emit a FileCheck-friendly format?
Not entirely sure that I follow, but it sounds to me like you’re suggesting that the diff functionality get put into this util.py (or nearby), as opposed to being a fully fledged binary?
Anything that gets built out will definitely be using the JSON libraries we already have. I don’t think FileCheck is sufficient because it seems to process everything in order of appearance, which is something that we need to avoid in certain places.
Sorry for being unclear! I don’t think it needs to be in util.py, a full-fledged binary may be more appropriate. I was more pointing out that we have some precedent for “existing tools aren’t up to snuff despite existing, so we’ll make our own replacements”.
FileCheck mostly processes things in order, but not everything. You can cause it to look at subsets of the input, if you have a textual marker that can delimit the subsets (using the CHECK-LABEL directive). Within a subset, things don’t necessarily have to be in order, although this is relatively limited (the CHECK-DAG directive). I’ve also seen FileCheck run multiple times on the same input with different sets of checks, if the checks don’t rely on each other.
The out-of-order requirement is why I was suggesting the tool could reorder the JSON to make it more FileCheck-friendly. I have little familiarity with JSON or how it’s used in the cases you’re thinking of, which makes it hard to discuss specifics, I’m just pointing out a potential alternative for composing with an existing tool rather than writing a differ from scratch. (LLVM has not done that to date.)
jd happens to be the tool I knew about, but if there are other tools out there that meet our needs but don’t have the Go dependency issues, we could consider those alternatives.
If it solves the problems you need it to solve, or you think it can be extended to solve them, feel free to slap the LLVM license on it and land it in-tree.