[RFC] Add a --wtt-output format for reporting lit results to Windows test infrastructure

LLVM Project
1

Summary

Lit has built-in result reporters (xUnit XML, ResultDB) but no way to produce WTT (Windows Test Technology) log files. This RFC proposes adding a --wtt-output flag that writes a .wtl file consumable by Windows lab infrastructure.

Motivation

We run LLVM Offload tests as part of Windows hardware certification (HLK). The HLK consumes test results through WTT log files. WTT log files have a UTF-16 XML format with specific elements for test start/end, error messages, and pass/fail results.

Lit already has precedent for platform-specific reporters, such as --resultdb-output, which produces ResultDB JSON.

Proposal

A new flag:

llvm-lit --wtt-output=results.wtl tests/

This produces a UTF-16 XML file conforming to the WTT Logger schema:

<?xml version="1.0" encoding="utf-16"?>
<WTT-Logger>
  <RTI ID="1" Machine="hostname" ProcessName="lit" ProcessID="1234" BaseTime="2026:6:23 17:0:0:0" Frequency="1" />
  <CTX ID="1" Current="WTTLOG" Parent="ROOT" />
  <StartTest Title="suite :: path/to/test.test" TUID="">
  </StartTest>
  <Err UserText="error output truncated to 4KB">
  </Err>
  <EndTest Title="suite :: path/to/test.test" TUID="" Result="Fail" Repro="">
  </EndTest>
  <PFRollup Total="50" Passed="48" Failed="2" Blocked="0" Warned="0" Skipped="0" />
</WTT-Logger>

Result code mapping

lit result WTT Result Notes
PASS Pass
XFAIL Pass Expected failure = success
FAIL Fail Error output included in <Err>
XPASS Fail Unexpected pass = failure
TIMEOUT Fail
UNRESOLVED Fail
EXCLUDED (omitted) Not reported
SKIPPED (omitted) Counted in summary
UNSUPPORTED (omitted) Counted in summary

Proposed Code Changes

  • lit/reports.py
    • Add WttReport class alongside the existing ResultDBReport. Inherits from Report.
    • Overrides write_results to force UTF-16 encoding (WTT requirement).
    • Iterates over tests, writes <StartTest>/<EndTest> pairs for each executed test, includes <Err> with truncated output (4KB) for failures, and writes a <PFRollup> summary at the end.
    • Includes two helper functions: _wtt_timestamp() for the WTT time format and _wtt_sanitize_attr() for XML attribute escaping.
  • lit/cl_arguments.py
    • Register the --wtt-output flag in the Output Format group, using type=lit.reports.WttReport (same --*-output naming pattern as --xunit-xml-output and --resultdb-output).

Design Decisions

  • UTF-16 encoding: WTT Logger spec requires it. The Report base class defaults to UTF-8, so WttReport overrides write_results to open the file with encoding="utf-16".
  • Output truncation: Error output is capped at 4KB per test, pass messages at 1KB. The WTT log viewer has display limits.
  • No new dependencies: Uses xml.sax.saxutils which reports.py already imports for the xUnit reporter.
  • Skipped tests omitted: Tests that didn’t execute (UNSUPPORTED, EXCLUDED, SKIPPED) don’t appear as <StartTest>/<EndTest> entries. A summary message notes how many were skipped.

Backward compatibility

Purely additive, and no output produced unless the flag is explicitly passed. Also, no new dependencies.

Thanks!

2

I guess given this can be abstracted pretty easily, I’m probably not opposed to this being added.

But this feature also seems like it would have a single user and could also easily be written as an external script that transforms JUnit/another format that’s currently emitted into WTT. Upstreaming this generation means the community is somewhat taking on the maintenance burden for this feature when there is likely only a single downstream user. Although given the nature of this feature and the relevant abstractions already being in place, the maintenance burden is likely quite small.

3

I understand the concern that this has a small audience, but as you point out this is self contained code that implements an existing abstraction. I think the maintenance burden should be small enough that doing this in tree where the logic can be simple and direct is worthwhile.

4

As mentioned, the abstraction makes this a small maintenance burden. I do want to point out that while it is one user, it is a rather larger user and impacts IHVs trying to get their drivers certified for Windows. It would be good to get more of this out of the walled garden of Windows.

5

Looks like there is agreement on this, I am working on a PR for this and will be posting it here once its published.

6

We discussed this in the @infra-area-team and we agreed this seems reasonable and well-scoped, FWIW.

1 Like
7

PR is ready for review! [lit] Add --wtt-output option to report results in WTT (.wtl) format by yassermkhan · Pull Request #211066 · llvm/llvm-project · GitHub