Gdb < 10.1 can't read clang's DWARF v5

cc @dblaikie @dyung

Using clang built at 13252160c3984b52a210bfa6ec64b9be4c911920, which includes the switch from default DWARF version from v4 to v5:

Build:

$ cat test.c
int main() { return 0; }

$ clang test.c -g -o test

Open in gdb 9.2 (error):

$ ~/dev/gdb/gdb-9.2-build/gdb/gdb test
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from test...
Dwarf Error: DW_FORM_strx1 found in non-DWO CU [in module /home/och/dev/llvm/llvm-project/test]
(No debugging symbols found in test)

Open in gdb-10.1 (ok):

$ ~/dev/gdb/gdb-10.1-build/gdb/gdb test
GNU gdb (GDB) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from test...

We noticed this when tests in cross-project-tests/debuginfo-tests/llgdb, which use whichever gdb is installed, started failing on an linux machine. The latest gdb version you get using apt on ubuntu-20.04 appears to be 9.2.

Do we need to mention this somewhere in docs or release notes?

Assuming it is desirable to keep the llgdb tests building using clang’s default dwarf version, is there an easy way to make the tests unsupported on machines with gdb versions less than 10.1?

My guess is this is just a gdb bug. DW_FORM_strx1 is not constrained to .dwo sections.
I’m not finding an option to emit DWARF v5 but without a .debug_str_offsets section. If there isn’t one, then it seems like the failing tests would have to use -gdwarf-4.

@adrian.prantl - do you own these tests? Any preference on how this works - either making the tests unsupported on earlier GDB versions, or changing them all to use -gdwarf-4?

Yeah, I’m responsible for these tests. These tests check that the platform debugger can debug code produced with the just-built clang, so in a way they work as intended. If you have a platform that ships an older version of GDB, ideally you would modify clang to not emit DWARF v5 on that platform. Now since you are talking about Ubuntu Linux, I don’t know if we have any precedence for checking for specific linux distributions/versions in Clang driver code. But if you are working on packaging Clang for that distribution, you should probably patch Clang to not default to DWARF 5.

I talked with a gdb developer who thinks this same problem will affect other tools too. I tried to do a quick test with debugedit 5.0 (latest version in Fedora), but I ran into a different error:

debugedit: test: Unknown DWARF DW_FORM_0x25

At least in Fedora, we may have to default to dwarf-4 until more of the tooling is able to parse clang’s output.

Form code 0x25 is DW_FORM_strx1, the same form that GDB didn’t like.

It could well be the case that gcc/gdb originally assumed that this form and its friends would be seen only with split DWARF, and that assumption percolated into other utilities. But, by the time split DWARF was standardized, these forms were all made applicable to non-split DWARF as well.

@dblaikie @tstellar I’d say there is a decision to make here, whether to revert the change in default until after the release branch (next week?) and then put it back in place. That will hopefully give distros more time to catch up with the DWARF v5 realities?
Just a thought,
–paulr

I don’t feel especially strongly either way - I didn’t commit this aiming to hit/not hit any particular release, so if it’s the case there’s still significant breakage on current versions of mainstream DWARF consumers, could make sense to switch back. At least if you’re releasing latest LLVM presumably that goes with latest gdb which does support DWARFv5 - I’m not sure about debugedit (never heard of/used it myself), whether it has enough users to mean that Clang’s defaults have to be lowered to be acceptable by debugedit in any platform that ships both?

Anyway - really don’t mind too much, however other folks feel about the compiler default compared to folks downstream tweaking the default (want a CMake flag for that, or happy enough to patch it in?), especially @tstellar 's opinion.

Be nice if the failure were a bit more obvious (“GDB doesn’t support this feature of DWARFv5, try using DWARFv4”), but no such luck - at least the failure’s pretty up-front, so if we put some documentation somewhere saying “ify ou see this error, use this flag” that wouldn’t be /super/ inconvenient. (compared to failing halfway through a debugging session)

@adrian.prantl, maybe I’m misunderstanding what you mean here (and if so I apologize), but with the change to clang, a developer working on linux with the latest Ubuntu LTS release (currently 20.04), the dwarf generated by the ToT compiler would be unusable with the currently latest available version of gdb (currently 9.2) for the system from Ubuntu. This scenario is not someone specifically working on creating a package for a distribution, but rather someone working on LLVM itself.

More concretely, I am working on getting our public linux bot running the cross-project-tests, and since it is running on Ubuntu 20.04, it will hit this problem unless we manually upgrade to a custom version of GDB on the machine. Perhaps we should wait until more distributions ship with a compatible debugger before enabling it as @pogo59 suggested.

I have no opinion on what the right default for Linux should be. I would not be opposed to make DWARF 4 the default on Linux if we agree that that’s the sensible decision.

This is correct, from what I’ve been told gdb assumes this will only be seen with split DWARF.

I would lean towards keeping dwarf-5 the default upstream and for the LLVM-14 release. I think this is the best way to get the GNU tools to support clang’s DWARF output better. It’s not that hard for distros to roll it back if they want.

I’m not a fan of adding CMake flags for things like this (even though as a distro packager it would make my life a little easier), because It just adds another testing configuration that all the lit tests need to support. This is really straight-forward for distro to patch out, and also users can easily drop back to dwarf-4 with the -fdebug-default-version option.

I’ve put up a patch for review that lets us XFAIL the tests llgdb-tests here ⚙ D118468 [cross-project-tests] XFAIL llgdb-tests when gdb can't read clang's DWARF.

Hi I’m running clang 14.0.0 release, and even when I pass in -gdwarf-4 I’m getting this error with gdb on Ubuntu 20.04. This didn’t happen with 13.0.1.

Figured it out. It was because libclangrt-builtins, had dwarf5 in it. I need to rebuild all the runtimes with dwarf4.

It would be nice to make this a CMake option.

Looks like this has been fixed in GDB with an update Ubuntu – Details of package gdb in focal-updates.

I’m still seeing the issue with that gdb version on Ubuntu 20.04.