Proposal: Using LLD in tests

Hello all,

currently we have a couple of tests, in-tree or under review, which are
very close to being host-independent. The only part they are missing is the
ability to link a intermediate object file:
- the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
resolve relocations in the debug info (*)
- the PDB tests under lit/SymbolFile/PDB need a linker to produce the
program database.

I think it would be great if everyone were able to run these tests and
verify they don't regress them before they actually push a patch.

Apart from that, I have started looking at writing some non-execution debug
info (**) tests as a part of adding DWARF v5 accelerator table support to
lldb (both to test the new implementation, and to make sure I don't regress
existing ones). Ideally I'd like to make sure that everyone is able to run
them, regardless of their primary (or only) development platform. For this,
I also need a linker capable of running everywhere (*)

To achieve these goals, I'd like to propose that we add LLD as a (optional,
but strongly recommended) dependency for running tests and start using it
in the tests I mention. Doing this would optional in the sense that the
tests would be marked "REQUIRED: lld", and simply skipped if lld is not
available (so the tests would still be green). I say "strongly recommended"
because not having lld checked out should not be an excuse for breaking the
test, and the patch author should pro-actively revert a patch which breaks
such tests and investigate.

I hope this proposal is not too controversial. LLD is already required on
windows to run dotest tests. Also, all monorepo users likely have it
already, or it is very easy for them to enable it. For non-monorepo users
it should be a matter of checking out one extra repository. Please Let me
know what you think.

pavel

(*) our ELF parser has very limited support for applying debug info
relocations -- it only works for x86, and only a couple of relocations are
currently implemented. It would be possible to remove the linker dependency
by implementing these (essentially, doing the link ourselves -- this is
what llvm does), but given the large number of architectures and relocation
types, combined with the long term goal of reusing the llvm's ELF parser,
this does not seem like a worthwhile goal. Also, it does not help the
windows situation, as in the PDB model it's the linker who produces the
pdb's.

(**) I'll write a separate email about this, but what I'm essentially
thinking of is producing a stand-alone module (either from .yaml, .s, .ll,
or .c), hitting it with various FindXXX methods, and dumping the results.

The last I knew LLD doesn't work on mach-o very well, so be sure to not require LLD for linking any Darwin executables.

Yes, I considered the lld+Mach-O situation. I understand that it does not
work very well, but I don't know what exactly that means.

However, I am not sure that we even need a linker for Mach-O. As I
understand it, in a darwin world, the linker does not even touch the debug
info, and lldb is already capable of reading the debug info from .o files.
So at least for non dsym-case it seems to me we should be good. It may be
possible we need a linker for the dSYM (i don't know exactly how that
works), in which case we may just not be able to test dsym this way, but
even that will be better than what we have now.

PS: I am not proposing to do anything to the existing dotest tests (they
need access to a running process anyway, so playing with the linker will
not help). I am just trying to make new debug-info specific tests
universally available.

The last I knew LLD doesn't work on mach-o very well, so be sure to not

require LLD for linking any Darwin executables.

>
> Hello all,
>
> currently we have a couple of tests, in-tree or under review, which are
> very close to being host-independent. The only part they are missing is

the

> ability to link a intermediate object file:
> - the ppc64 test in https://reviews.llvm.org/D44437 needs a linker to
> resolve relocations in the debug info (*)
> - the PDB tests under lit/SymbolFile/PDB need a linker to produce the
> program database.
>
> I think it would be great if everyone were able to run these tests and
> verify they don't regress them before they actually push a patch.
>
> Apart from that, I have started looking at writing some non-execution

debug

> info (**) tests as a part of adding DWARF v5 accelerator table support

to

> lldb (both to test the new implementation, and to make sure I don't

regress

> existing ones). Ideally I'd like to make sure that everyone is able to

run

> them, regardless of their primary (or only) development platform. For

this,

> I also need a linker capable of running everywhere (*)
>
> To achieve these goals, I'd like to propose that we add LLD as a

(optional,

> but strongly recommended) dependency for running tests and start using

it

> in the tests I mention. Doing this would optional in the sense that the
> tests would be marked "REQUIRED: lld", and simply skipped if lld is not
> available (so the tests would still be green). I say "strongly

recommended"

> because not having lld checked out should not be an excuse for breaking

the

> test, and the patch author should pro-actively revert a patch which

breaks

> such tests and investigate.
>
> I hope this proposal is not too controversial. LLD is already required

on

> windows to run dotest tests. Also, all monorepo users likely have it
> already, or it is very easy for them to enable it. For non-monorepo

users

> it should be a matter of checking out one extra repository. Please Let

me

> know what you think.
>
> pavel
>
> (*) our ELF parser has very limited support for applying debug info
> relocations -- it only works for x86, and only a couple of relocations

are

> currently implemented. It would be possible to remove the linker

dependency

> by implementing these (essentially, doing the link ourselves -- this is
> what llvm does), but given the large number of architectures and

relocation

> types, combined with the long term goal of reusing the llvm's ELF

parser,

> this does not seem like a worthwhile goal. Also, it does not help the
> windows situation, as in the PDB model it's the linker who produces the
> pdb's.
>
> (**) I'll write a separate email about this, but what I'm essentially
> thinking of is producing a stand-alone module (either from .yaml, .s,

.ll,

> or .c), hitting it with various FindXXX methods, and dumping the

results.

Yes, I considered the lld+Mach-O situation. I understand that it does not
work very well, but I don't know what exactly that means.

However, I am not sure that we even need a linker for Mach-O. As I
understand it, in a darwin world, the linker does not even touch the debug
info, and lldb is already capable of reading the debug info from .o files.
So at least for non dsym-case it seems to me we should be good. It may be
possible we need a linker for the dSYM (i don't know exactly how that
works), in which case we may just not be able to test dsym this way, but
even that will be better than what we have now.

The problem is the linker must produce a debug map in the symbol table that contains the addresses of all linked items. So using LLD won't work unless that is fully supported as LLDB won't be able to like the debug info on the fly, nor will dsymutil have a debug map to use in order to make a dSYM file.

PS: I am not proposing to do anything to the existing dotest tests (they
need access to a running process anyway, so playing with the linker will
not help). I am just trying to make new debug-info specific tests
universally available.

What I was trying to say is this is fine if we always compile/link as an ELF file with some triple that uses ELF for these tests. They just probably shouldn't be mach-o.

Greg

the PDB tests under lit/SymbolFile/PDB need a linker to produce the program database

With this proposal, would we preserve any coverage for MSVC produced debug information?

We already don’t really have any.

All of the makefiles that build the inferiors use a gcc style command line syntax, and the way we made that work on Windows is to use clang for the compiler. So MSVC is already out of the equation.

There may be one or two instances where we check in a PDB file, but that doesn’t really scale.

I’ve been working a lot on the PDB support in LLVM and we have some good coverage there that our PDBs are “as good as” Microsoft PDBs, and in the future we have plans to have a debug info test suite that tests LLD-generated PDBs with Microsoft debuggers. If we have coverage of that, I think that should be sufficient and maybe not worth complicating the LLDB test suite matrix over trying to test debugging with MSVC generated PDBs.

   the PDB tests under lit/SymbolFile/PDB need a linker to produce the

program database

With this proposal, would we preserve any coverage for MSVC produced

debug information?

Well.. the question there is what are you trying to test? Is it the fact
your debugger works with a particular compiler+linker
combination (note that those tests already compile with clang-cl), or that
your pdb-parsing code is sane. (integration vs. regression test).

Historically we've only had the former kind of tests (dotest), and we've
had the ability (and used it) to run those tests against different kinds of
compilers. This is all nice, but it means that a specific test will be
testing a different thing for every person who runs it. That's why I would
like to build up a suite of more regression-like tests (*). I would say
that the tests under lit/*** should be regression tests and our goal should
be to remove as many system dependencies as possible, and leave the job of
testing integration with a specific toolchain to "dotest" tests (**).

Technically, the answer to your question is "no", because currently dotest
tests don't know how to work with cl+link. Making that work would be an
interesting project (although a bit annoying as the Makefiles are full of
gcc-isms). However, I don't think that should stop us here.

(*) Ideally I would like to leave even the compiler out of the equation for
these tests, and make it so that the tests always run on the exact same set
of bytes. I am hoping I will be able to write at least some tests using .s
files. However, I don't think I will do that for all of them, because these
files can be long/verbose/tedious to write.

(**) However, even "dotest" tests should have a "default" mode which is as
hermetic as possible.

we have some good coverage there that our PDBs are “as good as” Microsoft PDBs, and in the future we have plans to have a debug info test suite that tests LLD-generated PDBs with Microsoft debuggers.

Thanks Zach. What I was asking is exactly the other half of this equation. Testing LLDB with MSVC produced PDBs should be complementary and I doubt we can get full coverage by just testing from the other direction (it’s plausible that even with equivalent semantic information, certain patterns may only occur in MSVC produced debug information)

IMO reliable support for MSVC generated binaries and debug information is critical since even with a LLVM/LLD toolchain you’ll still have system/3rd party modules, right?

Anyway, I was curious if any of this is in scope for Pavel’s proposal and the answer seems to be no, thanks everyone.

Our Windows buildbots use msys for gnuisms. The makefiles in the test suite run fine with minimal modifications (just the object delete hack Zach put in to use del instead of rm; msys make doesn't accept cmd syntax while Cygwin make does). Now, that's using clang to build Hexagon binaries, but teaching the makefile to use cl syntax shouldn't be too hard. I've seen it done before; same makefile for windows and various unix derivatives, detect what OS you were running on and set CFLAGS/CXXFLAGS/LDFLAGS accordingly.

Ted

I have created a patch <https://reviews.llvm.org/D46318>, which extends
lldb-test to support more precise dumping of the symbol information in a
module. It uses lld to make sure the tests can run on any system (which has
lld checked out) and to avoid the tests being affected by the environment.
Let me know what you think of it.

I've also tried using lld for the PDB tests. The lld part worked fine, but
unfortunately, it seems lldb still depends on the microsoft pdb reader to
get the symbol information (it seems the "native" pdb reading apis in llvm
are not all implemented).

On the bright side, it looks like it should be able to produce working (and
debuggable) MachO binaries using lld. It probably does not support all the
fancy features that the native darwin linker does, but it seemed to work
fine for my hello world examples (the only issue I saw was that it is not
possible to convince it to *not* require the dyld_stub_binder symbol, but
this can be worked around). I am going to continue experimenting here.

Our Windows buildbots use msys for gnuisms. The makefiles in the test

suite run fine with minimal modifications (just the object delete hack Zach
put in to use del instead of rm; msys make doesn't accept cmd syntax while
Cygwin make does). Now, that's using clang to build Hexagon binaries, but
teaching the makefile to use cl syntax shouldn't be too hard. I've seen it
done before; same makefile for windows and various unix derivatives, detect
what OS you were running on and set CFLAGS/CXXFLAGS/LDFLAGS accordingly.

Ted

--
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a

Linux Foundation Collaborative Project

> From: lldb-dev [mailto:lldb-dev-bounces@lists.llvm.org] On Behalf Of

Pavel

> Labath via lldb-dev
> Sent: Thursday, April 19, 2018 12:45 PM
> To: Leonard Mosescu <mosescu@google.com>
> Cc: aaron.lee.smith@gmail.com; LLDB <lldb-dev@lists.llvm.org>
> Subject: Re: [lldb-dev] Proposal: Using LLD in tests
>
>
> >> the PDB tests under lit/SymbolFile/PDB need a linker to produce
> >> the
> program database
>
>
> > With this proposal, would we preserve any coverage for MSVC produced
> debug information?
>
>
> Well.. the question there is what are you trying to test? Is it the

fact your

> debugger works with a particular compiler+linker combination (note that

those

> tests already compile with clang-cl), or that your pdb-parsing code is

sane.

> (integration vs. regression test).
>
> Historically we've only had the former kind of tests (dotest), and

we've had the

> ability (and used it) to run those tests against different kinds of

compilers. This

> is all nice, but it means that a specific test will be testing a

different thing for

> every person who runs it. That's why I would like to build up a suite

of more

> regression-like tests (*). I would say that the tests under lit/***

should be

> regression tests and our goal should be to remove as many system
> dependencies as possible, and leave the job of testing integration with

a

> specific toolchain to "dotest" tests (**).
>
> Technically, the answer to your question is "no", because currently

dotest tests

> don't know how to work with cl+link. Making that work would be an

interesting

> project (although a bit annoying as the Makefiles are full of gcc-isms).
> However, I don't think that should stop us here.
>
> (*) Ideally I would like to leave even the compiler out of the equation

for these

> tests, and make it so that the tests always run on the exact same set

of bytes. I

> am hoping I will be able to write at least some tests using .s files.

However, I

> don't think I will do that for all of them, because these files can be
> long/verbose/tedious to write.
>
> (**) However, even "dotest" tests should have a "default" mode which is

as

(it seems the “native” pdb reading apis in llvm are not all implemented)

Sorry, I missed this thread earlier. That’s true, the native PDB reading APIs have been started but are not complete. I’m working on that, mostly in the context of llvm-pdbutil rather than lldb.