[DebugInfo][DWARFv5][LLD] .debug_names with fdebug-type-sections

LLVM Project
,
1

Hello

I am trying to enable debug names acceleration table with fdebug-types-sections. One part I am not sure about is the local TU list. It contains an offset into .debug_info section. All the entries have an index entry that points to the local TU list. DIEs within entry offsets are relative to the TU entry.

Linker de-duplicates Type Units using COMDAT. So final result will have less type units. So Local Type Unit List will be invalid.
Am I missing something linker, specifically LLD, will need to be aware of context of .debug_names sections when it de-duplicates type sections?

There is also Foreign TU List, which has signature instead of offset, but that is for split dwarf. Maybe this is he way to go to implement type units?

Thanks.
@dblaikie @MaskRay

2

The more I look at it the more I think the best way to implement support is to always use the foreign type units. The Entry will have a Type hash. If it points to just TU then entry is in current object file, if it points to CU also then this is a split dwarf case. Not sure if this is abusing the spec.

" When an index entry refers to a foreign type unit, it may have attributes for both CU and (foreign) TU. For such entries, the CU attribute gives the consumer a reference to the CU that may be used to locate a split DWARF object file that contains the type unit."

Either that or abandon the whole idea and do it in post build step: llvm-dwarfutil has partial support for debug_names accelerator table. Although considering it lifts everything to IR, might be a bit slow for production purposes.

@clayborg WDYT form LLDB perspective?

3

I think the opposite direction might be needed/suitable - that .debug_names should refer to the type stubs in .debug_info.

The reason is this:

Types in type units are canonical, but don’t contain all the features of a type - for instance, they can’t contain, consistently, any nested types (those types might not be defined in every translation unit the type is defined in), member function templates (similarly, you might instantiate some instances of such a template in some translation units, and some in others), and implicit special members (similarly, they may be instantiated only in some translation units)

So if a DWARF consumer wants to reconstruct the totality of a type (eg, they want to evaluate TypeName::func<int>()) - then they’ll need to search for all type stubs.

This also means it’s simpler/the same whether using type units or not.

4

(it does mean the index is bigger, since it’ll need to mention every place a type appears - whereas if it referenced the type unit directly, indeed, there would only need to be one result for the given term in the index)

5

Either that or abandon the whole idea and do it in post build step: llvm-dwarfutil has partial support for debug_names accelerator table. Although considering it lifts everything to IR, might be a bit slow for production purposes.

Yeah, that’s no good for us (Google) - we don’t want to have to parse all the DWARF again to make an index to get acceptable debugger performance. We’re likely interested in something like the existing .debug_gnu_pubnames/types → .gdb_index solution - where individual .debug_names tables are put in .o files, then the linker can do a smart/DWARF-aware merge of these indexes, or it could even do a non-aware traditional linking and get something like the performance of Apple’s situation with .o files (where each .o has its own little index - except in this case allr those little indexes would be side-by-side in the executable, saving time/avoiding having to go out and pull all the .o files over a network filesystem before queries could be run).

6

Not sure what you mean by stubs in .debug_info. Can you elaborate please?
I asked @clayborg to give his perspective from LLDB point of view.
I agree post build tool that parses all of debug info is not good. Ideally linker can do something.
I guess that would be question for @MaskRay (anyone else?) on what level of involvement of LLD should have in supporting .debug_names.

7

(here’s the thread I was talking about, @cmtice )

8

@dblaikie
When you refer to “type stubs” are you referring to these kind:

0x0000004e:   DW_TAG_structure_type [10]   (0x0000000c)
                DW_AT_declaration [DW_FORM_flag_present]  (true)
                DW_AT_signature [DW_FORM_ref_sig8]  (0x104ec427d2ebea6f)
9

Right, sorry for the delay - been swamped with email after getting back from vacation and working through at least some of the pull request migration fallout.

Yes, those are the stubs I’m thinking of.

So breaking this down:

Non-Split DWARF:

  • Yes, the local type unit list would be affected by linking in one of two ways:
    • local type unit references could become dead (like references to linker-gc’d functions)
    • local type units could end up shared (sometimes this happens to comdat functions, if their definitions are identical)

I think either of these things are acceptable, and should happen without linker-special casing. The dead case should be handled by the usual debug info tombstoning rules - a relocation to a gc’d section gets replaced with a tombstone value & consumers should be ready to handle that.

For Split DWARF it’s more problematic because, as you say, you need a CU and a TU to find a type unit when a dwp isn’t built (& at compile/link time you don’t know, so you’ve got to err on the side of caution & make it usable even if the user is debugging from dwo files directly). But, yeah I think that could be addressed by using DW_IDX_type_unit and DW_IDX_compile_unit on any type unit references into Split DWARF. The foreign TU list wouldn’t get cleaned up - it’d continue to contain all the hashes, none would be tombstoned, etc (because the linker doesn’t observe the DWARF linking happening in the dwp, if it happens at all)

But, yes, the question of how to reference type units is a valid one - I think for Clang’s debug info, at least, we always produce a stub type description, which may have extra entities in it, etc. So we might as well, whenever there’s a type stub (which is always for LLVM), reference that - and not use the type unit mechanisms of .debug_names at all. This’ll make the index a bit larger than if we referenced type units directly (well, when the .debug_names is merged by a smart linker/or otherwise unified index is produced) - but is more comprehensive/useful for consumers to find all these possible features of a type. If a consumer finds a type unit, it could then ignore all the other versions of a type if it wanted to.

(I guess, arguably to find all these extra bonus features that aren’t in the canonical type, for Clang’s debug info (with type homing, -fno-standalone-debug) you’d need to include type declarations with features to be in the index, which isn’t valid according to the spec - so I guess we’ll ignore that case)

If we didn’t want to reference the type stubs - then, for non-split DWARF, I think referencing the type unit, using a relocation (same as we must be doing for CUs) for the TU offset, and letting that become tombstoned by the linker if the linker isn’t doing a DWARF-aware .debug_names merge. And for split DWARF, no relocations required, just using foreign TU hashes and merged on that basis.

Does that make sense?

10

No problem.
I saw you reply [Dwarf-discuss] [DWARF5] .debug_names + fdebug-types-sections also.
Let’s just continue discussion here, unless someone else jumps in there. I pointed @clayborg to this one also.

If I understand your post correctly there are three options:
Assertion: Clang always produces type stubs with type unit hash from which consumers can find correct TU. Checking locally it also ends up in .debug_names.
For monolithic:

  1. Just not output type units at all into .debug_names. Consumers can parse all of type units and use type_stubs in .debug_names to find correct type information. Question for LLDB folks is how much will it degrade the usefulness of .debug_names for them.
  2. Each module will have type unit information. When linker de-duplicates type units using COMDAT it can put a tombstone value in to the TU list for equivalent TUs (same hash), or the address of section kept if bit identical. If it is tombstone value, how would LLDB handle such a case when it searches for type units? Would it have to re-parse everything anyway?
  3. Smart linker approach. We put TU entries into their own section and make it depend on the TU section (like relocation section tied to section it is applied to). When linker drops TU section it also drops corresponding .debug_names.tu section. From remaining it will construct TU list. Although not sure how this will work with whole bucket concept, so we might end up with one TU per module.

For split-dwarf:

  1. Use DW_IDX_type_unit and DW_IDX_compile_unit on assumption it will all remain in .dwo files. Consumers will need to ignore it when DWP is present. Kind of like what they do now anyway.

From BOLT perspective, once I add support, I was thinking output would look like the output from step 3. There is much more flexibility here because BOLT re-writes all of debug information anyway, and has access to all of it.

Is that a good summary?

P.S. I have local modifications that output TUs per module with a reloc. Looks like right now with LLD tombstone value will be 0. Which is the same one as start of TU.

11

Speaking of referring to type stubs, we would end up referring to a type tag that has DW_AT_declaration set to true. The .debug_names spec states:

All non-defining declarations (that is, debugging information entries with a DW_AT_declaration attribute) are excluded.

The .debug_names section should point us to the real declaration and if it can’t, just don’t add it to the .debug_names.

Just not output type units at all into .debug_names. Consumers can parse all of type units and use type_stubs in .debug_names to find correct type information. Question for LLDB folks is how much will it degrade the usefulness of .debug_names for them.

LLDB keeps track of which DWARF units are included in .debug_names by checking the CUs, TUs and anything in the debug info that aren’t mentioned in any .debug_names tables will get indexed manually. If it is really hard to add correct and full indexing of type units, it would be a good start to just emit a .debug_names table for the CUs for now and leave any and all indexing of TUs (local and foreign) out for now, but it would be great to at least get something when type units are enabled.

LLDB wants full indexing of information within the type units, so what we don’t want is end up with single .debug_names entry for a type that only points to the main type in the type unit. If we have end up with a solution like this, then LLDB will detect we have an entry for the TU and it won’t manually index it to discover the contained types within the type in the type unit. For example all STL collection classes make many defines inside of the class itself like “iterator”, “const_iterator”, “reverse_iterator”, etc. If no one ever creates a local variable that uses any of these types, then the only definitions for them can be contained only inside the type unit, so we really need to be able to find these types when evaluating expressions, so the the type units need to be fully indexed.

So I would vote to avoid adding any entries into .debug_names that point the type stubs in the compile unit output as these violate the definition of what is to be contained in the .debug_names table and it isn’t really useful as LLDB will still need to manually index the type unit and we will end up ignoring these entries anyway.

Each module will have type unit information. When linker de-duplicates type units using COMDAT it can put a tombstone value in to the TU list for equivalent TUs (same hash), or the address of section kept if bit identical. If it is tombstone value, how would LLDB handle such a case when it searches for type units? Would it have to re-parse everything anyway?

What would actually get tombstoned here? The TU offset in the TUs table in the header of the .debug_names table? Or the DIE offset in the actual entry itself? LLDB can be taught to ignore either of these entries, so it shouldn’t be a problem, it will just cause lookups to be slower as we will need to sift through dead entries, but we already do this for functions, so it can easily be added if needed.

LLDB will actually probably still end up manually indexing any type units that don’t have full indexes.

Smart linker approach. We put TU entries into their own section and make it depend on the TU section (like relocation section tied to section it is applied to). When linker drops TU section it also drops corresponding .debug_names.tu section. From remaining it will construct TU list. Although not sure how this will work with whole bucket concept, so we might end up with one TU per module.

Could we create a separate fully indexed .debug_names section for each TU that references the current TU in the .o file and tie this section to the COMDAT section for the TU contents and don’t emit either of them if if should be removed?

I like any solution that is a bit smarter about what we emit over a solution that just concatenates and relocates/tombstones things.

For split-dwarf:

  1. Use DW_IDX_type_unit and DW_IDX_compile_unit on assumption it will all remain in .dwo files. Consumers will need to ignore it when DWP is present. Kind of like what they do now anyway.

Shouldn’t there a .debug_names section in the main executable that has everything all relocated and correct for the contents of the .dwp file? Or is there a .debug_names section in the main executable and each CU or TU has offsets that are only valid from the original .dwo file? Not sure how this works.

12

Yeah, that doesn’t seem great.

I don’t see any reason it’d have to reparse anything.
If we’re dealing with a dumb link - so lots of tiny .debug_names contributions, one per .o file linked into the binary, then a consumer looking for type information would do a query, find maybe the first result in some contribution - look up the local TU the entry references and find it’s tombstoned. It could then repeat the query until it finds a non-tombstone result. That’s probably still quicker than parsing any DWARF.
If you’re dealing with a smart-linked .debug_names unified hash table - it’s not a problem, such a smart link wouldn’t leave index entries that reference tombstoned type units. It could remove them entirely in favor of the entries describing the non-tombstoned version of the type.

This I’d still classify as a dumb linker (a smart linker wouldn’t need separate sections - it’d do semantic-aware .debug_names merging) but a smart compile-time encoding to get a better result from a dumb linker. Though “better” would be a bit subjective - it wouldn’t leave around dead/tombstoned entries - but it’d mean many more, smaller, indexes for the consumer to probe - which would hurt performance (making the search more linear, less constant-time).

Yep, you’d mentioned before that maybe this was a bit dodgy/not-entirely-to-spec, but it’s actually called out in the spec: “When an index entry refers to a foreign type unit, it may have attributes for both CU and (foreign) TU. For such entries, the CU attribute gives the consumer a reference to the CU that may be used to locate a split DWARF object file that contains the type unit.” - top of page doc page 142 (or physical/pdf page 160). So it seems like a solid way to go.

Fair enough - though the ability to find unqualified names inside types (where C++ doesn’t allow ADL or other forms of unqualified lookup anyway) seems of less value to me? But, sure, if the intent is to index all the members, then pointing to the stubs doesn’t help. The fact that they’re DW_AT_declarations seems a bit incidental, they still describe (indirectly) the whole type, so I could see myself to not applying the spec wording to this situation. shrug

13

OK. Let me try the tombstone approach for local TU list. I think I mostly got LLVM part done…
To clarify tombstone will be on the offset in the Local TU correct? I don’t think there is any way to tombstone index in entries.

For
" Use DW_IDX_type_unit and DW_IDX_compile_unit on assumption it will all remain in .dwo files. Consumers will need to ignore it when DWP is present. Kind of like what they do now anyway."

I originally suggested to use it in all cases not just for split dwarf thus the “dodgy/not-entirely-to-spec” part. For split dwarf it’s totally within spec, as you said.

14

Yep, the offset of the Local TU entries should use a relocation (as the CU offsets would use relocations too) & the linker should tombstone those if it resolves them to a non-identical copy, if I understand the linker semantics correctly.

Ah, right right - yeah, don’t think we need to do that.

15

LLVM part. Hopefully I did this correctly. Still getting used to new flow. :slight_smile:

16

@dblaikie @clayborg

So what do you think would be approach for DWP?

I think there are couple paths forward:

  1. Proposed by David of expanding the TU-Index table to include column for CU. That way tools, LLDB, that use .debug_names can see from which CU the TU came and use use the right one. Put this into a new section .llvm.tu.index or something for backward compatability and for tools that don’t use .debug_names.

  2. Something I cooked up maintain one same section but have two entries for each TU. One as it is now where type signagure is hashed to get the row. Another that combines DWO ID + TU into one hash, which is then used to access the row.

17

Have you got some data on % size of the tu_index (like, how bad would it be if we produce two - one for backcompat, one with a new/prototype format that supports encoding the original CU for each entry in debug_names, etc) - if it’s not much, then it seems like a nicer way to go since it sets us up for the future/means where won’t be much work to do once it’s standardized (just read the same format from the original section/with a standard version)

Otherwise I’m not totally opposed to jamming the the CU+TU hash into the table too… hmm, I guess that amounts to nearly doubling the size of the table anyway? So maybe the former’s still better?

18

I think former is better also. Specially if we can start brining other experimental features from DWARF6 (64bit support… ).

19

Yeah, fair. So we’ll need to sit down and spec out what we think a DWARFv6 cu_index/tu_index looks like, beyond the DWARF64 functionality that’s already been approved for DWARFv6.

I think some small abbrev-style record to describe a row in the index.

But I haven’t figured out how that should work exactly - should it have its own new attribute types (like the line table’s pseudo-abbreviation, I think? the DW_LNCT things) - then we could have some paired encoding (could abuse addr_offset for the offset+length encoding of the existing columns… oh, except those use ULEBs, so they aren’t fixed size records, which would be unfortunate)

So we could be more bespoke, where these DW_IDX “attributes” might have a special kind that’s a “DW_IDX_section” which would have a section identifier inline (like DW_FORM_implicit_const) followed by two forms for the offset and then length.
Then we’d have a DW_IDX_compile_unit_signature for the CU reference from the tu index.
And we could have a DW_IDX_user_lo/hi for future extension spaces.

The DW_IDX_section one would be pretty special (an inline constant value and two forms) but not totally unreasonable.

20

The main issue with .dwp files is we don’t know which type unit will be kept from the list of .dwo files for a given type. One idea was to put a new attribute in the DW_TAG_type_unit die that points to the .dwo file it came from maybe with a DW_AT_dwo_id attribute. This attribute would only be enabled when using split DWARF. Then if we find a .debug_names entry from a table, and if that table only has one CU (which is pretty much how all .debug_names from each .o file that also has a .dwo file are represented), then we can ignore any type unit entries when the DW_AT_dwo_id doesn’t match. The bad part is we would need to parse the first DIE of any type units to know if we should ignore the .debug_names entry, but it would be only the first DIE of a type unit, we wouldn’t need to parse all DIEs. That allows llvm-dwp to pick any type unit as the one representation, and then the name entries will be consistent.

I think is what @dblaikie was saying above with the DW_IDX_compile_unit_signature, but in this case we would just add the backlink to the DWO ID of the compile unit inside the type unit itself.