Status of debuginfo-tests

What is the status of debuginfo-tests? Is it actively supported? How do you run it? It doesn’t appear to be based on lit, any particular reason? Why is it its own repo instead of being part of llvm repo?

I’d like improve this to support CodeView and PDB, such that it would only run on Windows and only if a suitable debugger was found (probably WinDbg). WinDbg supports a JavaScript-based scripting model, similar to how LLDB supports a Python based model, so my thoughts were to have a lit-based runner that scans for .js files that contain a test script alongside some source, then build the program, run it in WinDbg with some script that does various things, and exits the debugger, moving on to the next test.

Anything I should be aware of / careful of when messing around in here? And any reason it can’t be moved to llvm/tests and ported to lit?

It’s used, but not a huge repository of things, as you can see. I think I’ve run it once or twice, but a long time ago. It was introduced for/by Apple/LLDB stuff, so it’s not something I’ve paid much attention to.

It’s probably not suitable as part of llvm tests directly. Those tests are designed to be shorter/narrower/more focussed than full integration tests (we don’t execute any compiled programs under test there, for example).

Porting to lit seems probably fine/good.

Even if it were under llvm, that doesn’t mean it would have to be part of ninja check-all. Currently we have lldb/test and lldb/unittests, what about a third lldb/dbgtests? Then you could run it as ninja check-dbgtest.

Although now that I think about it, we would want it to be able to invoke clang-cl directly, so I guess that does necessitate it not being under llvm.

As executable tests, they fail spectacularly in a cross-build environment. And you need some sort of debugger with GDB-like commands and output (or have some front end to your debugger that imitates that) in order to run them. I think they would need to stay in a separate project because of those requirements.

–paulr

To be clear, the tests I’m proposing will have no resemblance whatsoever to GDB, so I would be intentionally forking the set of tests in this regards. So there would very clearly be a paradigm shift in writing CodeView debug info tests (which would be written in JavaScript using WinDbg specific debugger commands) and DWARF debug info tests (which would be written in whatever / using GDB style commands)

Even if it were under llvm, that doesn’t mean it would have to be part of ninja check-all. Currently we have lldb/test and lldb/unittests, what about a third lldb/dbgtests? Then you could run it as ninja check-dbgtest.

Having them in lldb seems like an interesting question - not sure what the value is in having them in a separate repo compared to having them in the LLDB project. (I mean, long term, I think LLDB testing should probably have fewer of these end-to-end tests (ideally few tests that actually run a compiler/execute a resulting program (avoiding that so the tests are more portable, reliable, etc)) - but it’s reasonable to have some & probably OK that they live in the LLDB repo (for LLVM these sort of tests live separately in the test-suite repo)

Maybe because it makes the tests more likely to be run, more reliable (so they don’t fail due to LLDB changes - makes it easier for Clang/llvm developers to act on regressions, etc)

What is the status of debuginfo-tests? Is it actively supported?

Yes absolutely. For example, all jobs on green dragon run it.

  How do you run it?

The repository is supposed to be cloned into llvm/tools/clang/test and will then appear as part of check-clang.

  It doesn't appear to be based on lit, any particular reason?

It is most definitely based on lit.

  Why is it its own repo instead of being part of llvm repo?

Presumably (this decision predates me) so it can be easily disabled.

I'd like improve this to support CodeView and PDB, such that it would only run on Windows and only if a suitable debugger was found (probably WinDbg). WinDbg supports a JavaScript-based scripting model, similar to how LLDB supports a Python based model, so my thoughts were to have a lit-based runner that scans for .js files that contain a test script alongside some source, then build the program, run it in WinDbg with some script that does various things, and exits the debugger, moving on to the next test.

The repository already supports running with gdb and lldb (by implementing the gdb commands used in the tests using the lldb python API), so I'm sure a similar approach should work for Windows as well.

Anything I should be aware of / careful of when messing around in here? And any reason it can't be moved to llvm/tests and ported to lit?

As I said above, it is based on lit and considered part of the clang testsuite.

-- adrian

It’s used, but not a huge repository of things, as you can see. I think I’ve run it once or twice, but a long time ago. It was introduced for/by Apple/LLDB stuff, so it’s not something I’ve paid much attention to.

To clarify, they have been introduces by Apple as an end-to-end test for clang’s debug info under GDB and I later added a wrapper script so they also work with LLDB.

It’s probably not suitable as part of llvm tests directly. Those tests are designed to be shorter/narrower/more focussed than full integration tests (we don’t execute any compiled programs under test there, for example).

Porting to lit seems probably fine/good.

As mentioned in another reply, they already use lit.

Sorry, I had a bad typo. I meant we have llvm/test, llvm/unittests, so what about llvm/dbgtests.

I definitely woudln’t want them in LLDB, as my entire motivation is to write tests against a non-LLDB debugger.

Even if it were under llvm, that doesn’t mean it would have to be part of ninja check-all. Currently we have lldb/test and lldb/unittests, what about a third lldb/dbgtests? Then you could run it as ninja check-dbgtest.

They are not LLDB tests. They end-to-end test that the debug info produced by clang works with the host debugger, which can be gdb or lldb. They are part of the CFE tests for this reason.

– adrian

Hmm, I’m using a mono-repo and it appears under a top-level debuginfo-tests folder that doesn’t have any lit configuration files.

Yea, sorry, see above. I meant llvm, not lldb.

To be clear, the tests I’m proposing will have no resemblance whatsoever to GDB, so I would be intentionally forking the set of tests in this regards. So there would very clearly be a paradigm shift in writing CodeView debug info tests (which would be written in JavaScript using WinDbg specific debugger commands) and DWARF debug info tests (which would be written in whatever / using GDB style commands)

That sounds like an unfortunate direction. Can you explain why it wouldn’t be possible to write a wrapper (in JavaScript) that interprets the 3ish gdb commands used by the tests in terms of WinDbg? Similar to how LLDB is supported?

– adrian

That’s a bug in the monorepo :slight_smile:
More seriously: This is probably a good idea. This way people who want to run them, can symlink them into llvm/tools/clang/test.
They don’t have a lit configuration because they expect to be checked out into clang’s test directory.

– adrian

They don’t have a lit configuration because they expect to be checked out into clang’s test directory.

It might be nice if it had one that returns Unsupported when target != host.

Not sure how to express that in lit-speak though.

–paulr

I can think of a couple of reasons:

  1. We’re already going to need entirely different runlines. clang and clang-cl don’t use the same command line options, or for that matter even styles.

  2. The output format is going to be different. whereas the current tests look for things like

// CHECK: ${{[0-9]+}} =
// CHECK: Data ={{.}} 0x0{{(0)}}
// CHECK: Kind = 2142

In WinDbg this is going to be more like:

Local var @ 0x6ffa50 Type SVal

   +0x000 Data             : (null) 
   +0x004 Kind             : 0x85e

So we're also going to need different check lines.
At this point, the only similarities in the PDB / DWARF tests is going to be the source code, as I don't see an easy way to automatically translate command lines, input commands, and output text.

cmake -DCLANG_ENABLE_DEBUGINFO_TESTS=ON

?

To be clear, the tests I’m proposing will have no resemblance whatsoever to GDB, so I would be intentionally forking the set of tests in this regards. So there would very clearly be a paradigm shift in writing CodeView debug info tests (which would be written in JavaScript using WinDbg specific debugger commands) and DWARF debug info tests (which would be written in whatever / using GDB style commands)

That sounds like an unfortunate direction. Can you explain why it wouldn’t be possible to write a wrapper (in JavaScript) that interprets the 3ish gdb commands used by the tests in terms of WinDbg? Similar to how LLDB is supported?

I can think of a couple of reasons:

  1. We’re already going to need entirely different runlines. clang and clang-cl don’t use the same command line options, or for that matter even styles.

I’m not familiar with clang-cl so please bear with me if this is a stupid question: Is clang-cl just an MSVC compatible driver? Can you not produce the same result by using the “standard” clang driver and a windows target triple?

  1. The output format is going to be different. whereas the current tests look for things like

If you are going to write a wrapper script for the debugger commands, that wrapper script could also transform the output to look more like the GDB output that the CHECK-lines are looking for.

// CHECK: ${{[0-9]+}} =
// CHECK: Data ={{.}} 0x0{{(0)}}
// CHECK: Kind = 2142

In WinDbg this is going to be more like:

Local var @ 0x6ffa50 Type SVal

   +0x000 Data             : (null) 
   +0x004 Kind             : 0x85e

So we're also going to need different check lines.
At this point, the only similarities in the PDB / DWARF tests is going to be the source code, as I don't see an easy way to automatically translate command lines, input commands, and output text.

By the way, I’m fine with adding, e.g., a PDB subdirectory with a lit filter that contains tests that will only work under Windows, but it would be very unfortunate if there was no common set of tests, since then the Windows folks wouldn’t benefit from new tests being added for GDB and vice versa.

– adrian

That’s a bug in the monorepo :slight_smile:
More seriously: This is probably a good idea. This way people who want to run them, can symlink them into llvm/tools/clang/test.

cmake -DCLANG_ENABLE_DEBUGINFO_TESTS=ON

Sounds reasonable. Unfortunately, I don’t know enough CMake or LIT to make this happen…

To be clear, the tests I’m proposing will have no resemblance whatsoever to GDB, so I would be intentionally forking the set of tests in this regards. So there would very clearly be a paradigm shift in writing CodeView debug info tests (which would be written in JavaScript using WinDbg specific debugger commands) and DWARF debug info tests (which would be written in whatever / using GDB style commands)

That sounds like an unfortunate direction. Can you explain why it wouldn’t be possible to write a wrapper (in JavaScript) that interprets the 3ish gdb commands used by the tests in terms of WinDbg? Similar to how LLDB is supported?

I can think of a couple of reasons:

  1. We’re already going to need entirely different runlines. clang and clang-cl don’t use the same command line options, or for that matter even styles.

I’m not familiar with clang-cl so please bear with me if this is a stupid question: Is clang-cl just an MSVC compatible driver? Can you not produce the same result by using the “standard” clang driver and a windows target triple?

  1. The output format is going to be different. whereas the current tests look for things like

If you are going to write a wrapper script for the debugger commands, that wrapper script could also transform the output to look more like the GDB output that the CHECK-lines are looking for.

// CHECK: ${{[0-9]+}} =
// CHECK: Data ={{.}} 0x0{{(0)}}
// CHECK: Kind = 2142

In WinDbg this is going to be more like:

Local var @ 0x6ffa50 Type SVal

   +0x000 Data             : (null) 
   +0x004 Kind             : 0x85e

Alternatively, this is still similar enough that we could relax the check to look like
CHECK: Data{{.[:=].(0x0|.null.)}}
CHECK: Kind{{.[:=].(2142|0x85e)}}
without much loss.

– adrian