Description of the project: LLVM’s main test running tool, lit (LLVM’s integrated tester), is a key part of LLVM’s testing practices for end-to-end testing of the compiler and related tools. lit is a tool that can parse and execute RUN: commands added in source file comments. Chiefly, these commands are shell commands that invoke tools that are being tested—like clang or lld—and whose output can be programmatically checked in conjunction with other tooling, like LLVM’s FileCheck utility.
Over the last couple years, we’ve worked towards improving lit’s internal shell enough, so that it could be the default mode for running LLVM tests (see [RFC] Enabling the Lit Internal Shell by Default). Today, lit already uses the internal shell by default in LLVM’s CI, which has improved test performance and increased consistency across platforms. However, there are still many aspects of the current implementation that could be improved, though.
Modernize the lit Python implementation: Today, lit’s current implementation is, to a large extent, a direct translation of the old Python 2.7 implementation to Python 3. There are numerous language improvements that would allow us to simplify the code and follow best practices. This would require an audit of the current implementation and subsequent refactoring. Note that we’re not advocating a complete rewrite, but targeted well scoped changes to bring the current implementation into line with more modern Python standards.
Reduced process overhead: Using asyncio for subprocess handling is likely a significant improvement in terms of performance, since lit spawns many subprocesses and waits for their results when it could frequently take care of unrelated work. Further, some of the out-of-process builtins should be moved back so that they can run in-process (see [RFC] Reducing process creation overhead in LLVM regression tests).
Improve shell utilities: A few of the internal shell replacements for common programs have some issues with common usage. For instance, the built-in env has some problems with pipelines (see #115578), and the ergonomics around readfile and environment variables in some compiler-rt tests are problematic.
Performance tuning: lit and the various pieces of its internal shell implementation have never been properly profiled and optimized. For instance, commands like echo and cat, while functional, employ suboptimal algorithms and data structures. There is a big opportunity here to leverage profiling information, specifically via Scalene, to pinpoint inefficiencies, such as contention on the Global Interpreter Lock (GIL), excessive memory usage, and redundant data copies.
Expected result: The goal of this project is to improve lit’s performance, usability, and maintainability. Successful proposals should provide a detailed plan for improving lit’s implementation. Applicants are encouraged to look more deeply into these areas, and provide concrete plans for what they propose to change, over only vague descriptions of improvements.
Skills: Python
Project size: Either medium or large
Difficulty: Medium
Confirmed Mentor: @boomanaiden154-1, @ilovepi, @petrhosek
