[RFC] Address spaces support in LLDB

Address space support in LLDB has been a long-desired feature. This RFC revives an earlier discussion from @JDevlieghere and @jasonmolenda from 2020.
link: [RFC] Segmented Address Space Support in LLDB

TL;DR

Several targets debugged through LLDB have more than one address space: the same numeric address means different memory depending on which space you name. LLDB today assumes a memory location is a single addr_t (uint64_t), so it cannot represent or read these locations.

This RFC proposes address-space support built around three ideas:

  • The target reports its address spaces dynamically at runtime over gdb-remote (jAddressSpacesInfo). And no object/symbol format has to encode them.

  • A value type, AddressSpec (an address plus an optional address-space name/id), carries “where” through the code that needs it, and an AddressSpec with no space behaves exactly like today’s addr_t, i.e. single-address-space targets and all existing APIs are unaffected.

  • Rollout is incremental : new overloads that fall back to the plain path in the default space and error when a space is requested but unsupported.

A working end-to-end implementation exists downstream (memory reads, CLI, DWARF expression evaluation, variable display, TLS, unwinding, and a GPU plugin); this RFC describes that design and proposes to upstream it in phases, starting with runtime memory reads.

Full implementation is available in @clayborg 's clayborg/llvm-server-plugins branch.

Motivation and use cases

Address spaces are needed by a range of targets:

  • GPUs / accelerators (the immediate driver for Meta). A GPU exposes distinct memories (eg: generic, global, local, private (per-lane / per-wave), region etc.) where the same numeric address refers to different storage, and several are thread/wave-specific. AMD’s DWARF address spaces are a concrete example (see User Guide for AMDGPU Backend — LLVM 23.0.0git documentation).

From the original RFC

  • Harvard architectures: separate instruction/data memories reusing addresses, e.g. Motorola DSP56300 p:/x:/y: where p:100, x:100, y:100 are three different locations.

  • Low-level / JTAG debugging : viewing memory as “virtual via TLB”, “virtual == physical via core”, “physical via SoC, uncached”.

  • WebAssembly : no fixed number of address spaces — every module gets its own — and the DWARF does not say which space a location is in; the dynamic loader decides.

  • Embedded targets: that encode address-space info in ELF.

Background: how LLDB represents addresses today

LLDB has three kinds of uint64_t address (as clarified by Greg Clayborg on the other RFC):

  • file address: This is meaningful only to its Module; carried as lldb_private::Address (section + offset) and it resolves to a load address if the section is mapped (needs a Target).

  • load address : This is unique in a process that has no segments; the transient addr_t used for user input and inferior memory read/write.

  • host address: a pointer into LLDB’s own process (expression results).

The load-address assumption is that “it is unique in a process with no segments”, which breaks for the targets above.

Open Questions from 2020 RFC

  • Bigger scope of the Refactor: give Address a segment discriminator, introduce ProcessAddress (uint64_t + discriminator) to replace addr_t throughout, do the substitution with a clang tool as one large NFC patch, then add behavior, then SB overloads. Greg and Pavel argued that it is not entirely NFC.
  • Where does address-space info come from? No Agreement on this one, options are DWARF, ELF, or dynamically retrieved from the target etc.

Design

Key Design Principles

  • Target is the source of truth for the set of live address spaces (jAddressSpacesInfo gdb packet): The running target inherently knows its own memory layout, so it’s the authority.
  • DWARF supplies static locations spaces via the standard LLVM extension DW_OP_LLVM_form_aspace_address, so variable reads land in the right space.
  • Additive and incremental with small blast radius. Existing addr_t/Address APIs untouched i.e. default space behaves as today and unsupported spaces error rather than silently read the wrong memory.

Address space discovery (jAddressSpacesInfo)

The client asks the target which address spaces exist by sending a jAddressSpacesInfo packet, and each one is described by an AddressSpaceInfo:

struct AddressSpaceInfo {
  std::string name;        // e.g. "generic", "local"
  uint64_t    value;       // numeric id used on the wire / in DWARF
};
send:  jAddressSpacesInfo
reply: \[{"name":"global","value":1},{"name":"local","value":2}\]

jAddressSpacesInfo returns the list as JSON; the client caches it on the Process. Empty list implies single default address space (the common case), everything behaves as today. Advertised via qSupported (address-spaces+); if not advertised, the client never sends address-space packets.

value is the numeric address space id used both on the wire and in DWARF.

Specifying an address location (AddressSpec)

This is meant to be generic and expressive for every scenario. It is constructed differently depending on what is known, and an empty space means the default space:

  class AddressSpec {
    lldb::addr_t               m_addr;
    std::optional<std::string> m_space_name;  // set if addressed by name, or
    std::optional<uint64_t>    m_space_id;    // set if addressed by id, else default

  public:
    AddressSpec(lldb::addr_t addr) : m_addr(addr) {}                        // default space
    AddressSpec(lldb::addr_t addr, uint64_t space_id)                       // by id
        : m_addr(addr), m_space_id(space_id) {}
    AddressSpec(lldb::addr_t addr, llvm::StringRef space_name)              // by name
        : m_addr(addr), m_space_name(space_name.str()) {}

    lldb::addr_t GetValue() const { return m_addr; }
    bool IsInDefaultAddressSpace() const {
      return !m_space_name && !m_space_id;
    }

    /// Resolve to the target's AddressSpaceInfo (by id or name), or an error.
    llvm::Expected<AddressSpaceInfo> GetAddressSpaceInfo(Process &) const;
  };

introducing a new packet for mem read (qMemRead )

send:  qMemRead:addr:1000;space:2;length:4;
reply: <hex bytes>          # e.g. 11223344
       E<xx>                # error
       (empty)              # unsupported

Reading memory

Process gets an AddressSpec overload; the default space falls back to the existing path, and unsupported spaces error:

size_t Process::ReadMemory(const AddressSpec &, void \*buf, size_t, Status &);
//   default space -> resolve to plain addr_t -> ReadMemory(addr_t)
//   otherwise     -> GetAddressSpaceInfo -> DoReadMemory(AddressSpec, info)

virtual size_t Process::DoReadMemory(const AddressSpec &, const AddressSpaceInfo &, void \*buf, size_t, Status &);
//   base: "address space memory reading is not supported"; ProcessGDBRemote overrides.

On the wire (ProcessGDBRemote):

qMemRead:addr:<hex>;space:<hex id>;length:<hex>;

Note: we started with reads first.

Expression evaluation and variables (DWARF)

When debug info places a variable in an address space (via the standard LLVM DWARF extension DW_OP_LLVM_form_aspace_address), LLDB reads that variable and its members from the correct space automatically.

Command line

memory read gains a --space <name> option.

Public API

class SBAddressSpec {
  SBAddressSpec(lldb::addr_t load_addr);                                 // default
  SBAddressSpec(lldb::addr_t addr, const char \*name);                   // by name
};

SBProcess gets a new API, and existing ReadMemory(addr_t, ...) is not touched

size_t SBProcess::ReadMemoryFromSpec(SBAddressSpec, void \*buf, size_t, SBError &);

Unwinding

RegisterContext / RegisterContextUnwind consult the address space when reading during unwind, so stacks in non-default spaces unwind correctly.

References

Please review this and provide feedback @JDevlieghere , @jasonmolenda , @labath , @zdenek-prikryl

Something to consider - address spaces can be used for other things, too, not just distinct memories. If you have a system that allows you to look at the same memory in different ways, you could use different memory spaces for things like memory as the core sees it, uncached memory, memory accessed through the SoC, memory accessed through the core, etc.

Is this related to the GPU support mentioned in Upstreaming basic support for accelerators?

I believe there’s been some interest in formalizing address spaces further as well https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3795.pdf.

Is this related to the GPU support mentioned in Upstreaming basic support for accelerators ?

Yes, exactly. While our (Meta’s) immediate driver is GPU debugging, @clayborg specifically designed this as a generic infrastructure to support any of the various use cases that rely on multiple address spaces.

I believe there’s been some interest in formalizing address spaces further as well https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3795.pdf.

Thank you for sharing that link! It’s great to see address spaces becoming first-class citizens in the language standard.

From what I can tell, AMD’s GDB solution hands off all the work to an AMD debugging library. If that means it only works for local debugging, that means they’ve not got any gdb remote protocol extensions.

Nvidia has its own GDB port but I couldn’t tell whether they had any protocol extensions.

Does that match your understanding?

I was going to say address spaces fit neatly into GDB Remote Protocol Extensions - 🐛 LLDB because it already uses the key:value format.

Except it doesn’t quite because a range is just a begin and end. We could have parallel values where the number of ranges == the number of address space IDs, that’s not too hard to parse.

An alternative to new packets (plural) would be a single packet to set the address space to be used for future packets. This requires caching what you said last, and clearing that cache and so on, so I don’t like it personally.

The root of the issue is a lot of the original packets not being extensible enough. So I like that your qMemRead would be extendable beyond address spaces.

Addresses appear in other packets:

  • c can be c<address>, though I suspect we hardly ever use it in that way.
  • z type,addr,kind, how will you know what address space to use here, will the debug server default to the selected thread? Perhaps there is only one “code” address space for some targets, but it sounds like WASM might have one per module.

Hi,

The proposed RFC roughly matches our downstream implementation that we are maintaining for our products. So, we have extended the address representation pretty much the same as the proposal has. We use the same DWARF extension. Besides the target providing info about the address space, we also have info in ELF (section that indicates to which address space it belongs), since we need to work with the binaries as well (different kinds of converters, binutils, …). But this is irrelevant from the LLDB perspective.

So, we are supportive of this proposal.

Thanks and best regards.

CUDA-GDB does have NVIDIA-specific remote-protocol extensions, but its architecture is somewhat orthogonal to what we are considering for LLDB. The client intercepts CUDA-aware memory accesses, uses the CUDA address class and current GPU focus to select the appropriate CUDA Debugger API operation, and falls back to ordinary host-memory access when appropriate.

For a remote target, it proxies that API request to cuda-gdbserver over NVIDIA-specific RSP packets and the server services it through the target-side CUDA debugger machinery. In other words, it tunnels the CUDA Debugger API rather than adding address space semantics to the standard memory packets.

Because accesses can alternate among address spaces, I would prefer carrying the address space ID on each operation rather than introducing a global “current address space.” Persistent state would add selection round trips and cache, reset, and desynchronization rules.

I’d also strongly suggest to carry the address space ID on each operation.

Your lldb-server can return as many address spaces as it wants and the space integer identifier has a name associated with it so that the memory read --space <addr-space-name> works. If you specify an incorrect address space name, then it will specify all valid values.

Is this related to the GPU support mentioned in Upstreaming basic support for accelerators ?

yes it is.

I believe there’s been some interest in formalizing address spaces further as well https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3795.pdf.

If DWARF supports address spaces, then we need to be able to use the same integer identifiers as the DWARF when the LLDB server plug-in for GPUs returns address spaces. Likewise, if an ELF file supports specifying that a section should be from an address space, we need to support that in our lldb_private::Section class.

Correct. We implement the lldb-server extensions in our lldb-server, so this won’t contribute anything back to GDB unless they want to support our GDB servers as a way to debug. GDB also combines both processes for native and GPU into a single process. We don’t do that. So GDB would need to manage two connections and combine them if they wanted to use our lldb-server binaries, so I don’t think that is anything that is desired from the GDB folks.

They don’t AFAIK. They have a GDB branch where the add all of the support they need and I don’t believe they support GDB remote connections.

I don’t like any sequences of packets for GDB remote as it is always latency bound and then we need to track what the current address space is and make sure we are always setting to the correct value. So each memory read could require 2 GDB remote packets.

If we end of being able to tell from looking at an object file (ELF, Macho or COFF) and can have the ObjectFile objects create sections that have predefined address spaces, we can extend the GDB remote packet for setting breakpoints much like we did for qMemRead. We haven’t needed it so far for neither AMD or NVidia.

Part of the problem with making address spaces work transparently in lldb was that we had a structure for representing addresses (Address) but then when we got to places in lldb where we were “sure” the address’s meaning was unambiguous, we converted it to a scalar value and handed it the rest of the way like that.

Because there was lots of code that handled scalar addresses, we balked at the size of the change that adding address spaces to lldb and pushing them everywhere.

This change works around that by adding another high level entity, AddressSpec, and then adding a parallel (to the Address/SBAddress) access using AddressSpec.

It would seem more natural to add Address Space information to Address, and using that in the places where AddressSpec. That wouldn’t be the perhaps overly ambitious plan to use the structured address everywhere lldb uses addresses, but it would avoid adding two not quite parallel systems for “smarter addresses” and since we’re starting from the outside and just working it as far in as needs to go, it seems like we could do this with Addresses just as well as with the pair Address/AddressSpec.

Why was this more Occam-friendly approach seem like it wouldn’t work?

On the tiny scale, why does AddressSpec need to have both names and indices? From what you were saying, we are always getting a mapping “name->ID” from the target. So we can always unambiguously go from name ↔ ID. If that’s true then it seems wasteful to keep the optional<std::string> in the AddressSpec.

If we are going to add address spaces to lldb_private::Address there are two places we might need to add it: in the lldb_private::Section and if we don’t have a section then we need to be able to manually specify it. AddressSpec allows both to work with out requiring either one to support address spaces. We don’t have any current binaries in GPU land that have sections that are address space specific, but that doesn’t mean we won’t have them in the future.

The current patch isn’t including everything that is in the GPU branch. I am not sure why the extra functionality was left out:

/// \class AddressSpec Process.h "lldb/Target/Process.h"
/// An address specification structure powerful enough to represent any memory
/// reads from any kind of address space. This struct needs to handle all of the
/// arguments needed to read from:
/// - A load address to read from default address space.
/// - A load address + address space for reading from different address spaces.
/// - A file address + module for a file specific address in a module.
/// - lldb::addr_t + module + thread ID for TLS (Thread Local Storage)
class AddressSpec {
  /// Load address, file address or offset. If this optional has no value, then
  /// a constructor was called that failed to resolve.
  lldb::addr_t m_value; 
  /// The name of the address space to read from if the value is not empty.
  std::optional<std::string> m_addr_space_name;
  /// The address space integer if this has a value.
  std::optional<uint64_t> m_addr_space_id;
  /// A module might be required for thread local data for a module, or for
  /// address spaces that are thread specific.
  lldb::ModuleWP m_module_wp;
  /// A thread is needed for thread local data accesses where the m_value is
  /// an offset in the thread specific section for a module that is maintained
  /// by the dynamic loader, or for address spaces that are thread specific.
  lldb::ThreadWP m_thread_wp;
public:
  /// Construct an AddressSpec from a load address.
  explicit AddressSpec(lldb::addr_t load_addr) : m_value(load_addr) {}

  /// Construct an AddressSpec from an address and address space integer
  /// idenentifier and an optional thread.
  ///
  /// This method should be used by clients that parse debug info or runtime
  /// information that contains the address space integer identifier in the
  /// serialized format.
  explicit AddressSpec(lldb::addr_t load_addr, std::optional<uint64_t> addr_space_id,
                       lldb::ThreadSP thread_sp = {}) : 
      m_value(load_addr), 
      m_addr_space_id(addr_space_id), 
      m_thread_wp(thread_sp) {}

  /// Construct an AddressSpec from an address and address space and an 
  /// optional thread.
  ///
  /// This method should be used by users when specifying options or address
  /// spaces by name as the users should not know about the numbering schemes
  /// for address spaces.
  AddressSpec(lldb::addr_t addr, 
              llvm::StringRef addr_space,
              lldb::ThreadSP thread_sp = {}) : 
      m_value(addr), 
      m_addr_space_name(addr_space.str()), 
      m_thread_wp(thread_sp) {}

  /// Construct an AddressSpec from a file address and its module.
  AddressSpec(lldb::addr_t file_addr, 
              lldb::ModuleSP module_sp) : 
      m_value(file_addr), m_module_wp(module_sp) {}

  /// Construct and AddressSpec from an offset + module + thread ID which 
  /// represents a threaad local storage address.
  AddressSpec(lldb::addr_t offset, 
              lldb::ModuleSP module_sp, 
              lldb::ThreadSP thread_sp) : 
      m_value(offset), m_module_wp(module_sp), m_thread_wp(thread_sp) {}
  
  /// Check if this address specification describes an address that is mapped
  /// into the default address space. The default address space indicates we
  /// should read memory from the process using the old APIs that take a 
  /// lldb::addr_t as a load address with no address space.
  bool IsInDefaultAddressSpace() const {
    // If we have an address space, then this address can't be in the default
    // address space.
    return !(m_addr_space_name.has_value() || m_addr_space_id.has_value());
  }

  /// See if this address spec can be converted a load adderess in the default
  /// address space. The call will resolve any AddressSpec that doesn't have
  /// an address space into a load address. If the address has an address space, 
  /// an error will be returned. Otherwise this method will resolve the address
  /// or return an error detailing why the address failed to resolve.
  ///
  /// \param[in] process The process to use when resolving this address.
  ///
  /// \param[out] load_addr The resolved load address to fill in. 
  ///
  /// \return
  ///     An error object. If the error is in a success state, \a load_addr was
  ///     successfully resolved. Otherwise the error states why the address 
  ///     couldn't be resolved. 

  llvm::Expected<lldb::addr_t> ResolveAddressInDefaultAddressSpace(
      lldb_private::Process &process) const;

  /// \return
  ///     The address value, which can be a load address, file address, or
  ///     offset depending on how this AddressSpec was constructed.
  uint64_t GetValue() const { return m_value; }

  /// \return
  ///     A StringRef to the address space name if it was specified via the
  ///     string-based constructor, or an empty StringRef if no name is set.
  llvm::StringRef GetSpaceName() const { 
    if (m_addr_space_name.has_value()) 
      return *m_addr_space_name;
    return llvm::StringRef();
  }

  /// \return
  ///     An optional containing the address space ID if this AddressSpec was
  ///     constructed with a numeric address space identifier, or std::nullopt
  ///     if no numeric ID is set.
  std::optional<uint64_t> GetSpaceId() const { return m_addr_space_id; }

  llvm::Expected<AddressSpaceInfo> GetAddressSpaceInfo(
      lldb_private::Process &process) const;

  // Return an error if this is module specific and the module has expired, 
  // otherwise return a ModuleSP, even if it is empty.
  llvm::Expected<lldb::ModuleSP> GetModule() const;

  // Return an error if this is thread specific and the thread has expired, 
  // otherwise return a ThreadSP, even if it is empty.
  llvm::Expected<lldb::ThreadSP> GetThread() const;
};

If we add all of these these instance variables inside of lldb_private::Address that might add too much size as we create a lot of these. And a lot of this information really belongs in a Section, but we don’t want sections to be required when users want to specify things manually, so we would need to also duplicate all information in the Address object as well and that didn’t seem like a clean design.

Users always specify things by name, and LLDB source code, DWARF, and unwind will use the numbers. If things are specified by name this allows an API call to read memory to be made across different processes with the same SBAddressSpec object without having to create an SBAddressSpec each time you switch processes. Users shouldn’t ever need to know about the address space number as the names make more sense. So this was more to make the API usable and not have weird bugs over API calls if the user was required to find the address space integer ID from a call to SBProcess just so they can use SBAddressSpec. So this decision was for usability. These objects are transient as the process has the address spec vector of AddressSpaceInfo objects, and the user always uses an address space name (if they get the name wrong, all errors say either “this process does not support address spaces” or “Incorrect address space name ‘foo’, valid values are ‘global’, ‘local’”), and the API in the code uses integers.

I wasn’t proposing that users would dial up address spaces by number. But since you always have an address-space name → address space number table IRL, why do you need to store both IN the AddressSpec object? Instead, keep a side-table of name->ID and when the user asks for an address space by name, consult your table and record the number in the AddressSpec. There’s no need to keep both in the AddressSpec.

The thing that bugs me here is that Address-s w/o sections are a thing, and can and often do represent load addresses that aren’t in sections just by leaving out the section info. Now we have another structure that can either represent load addresses in address spaces, or just plain addresses if they have no load addresses.

So now I have to decide whether to use an Address or an AddressSpec to represent some address in a running program. That seems likely to get confusing.

I understand your thinking, but as I was creating this I ran into cases where I thought it would be best to leave it in string mode as API calls can have different results if you make an SBAddressSpec with address 0x1000 and memory space “local”. “local” can mean 1 thing to process A, and another to process B. Not a huge deal, but from an API perspective I would want the address spec to stay in string format. Something like:

SBAddressSpec addr_spec(0x1000, "local");

process1.ReadMemory(addr_spec, 512, ...);

process2.ReadMemory(addr_spec, 512, ...);

If we resolved the address spec to a number namespace with:

ABAddressSpec addr_spec(0x1000, process1, "local");

process1.ReadMemory(addr_spec, 512, ...);

process2.ReadMemory(addr_spec, 512, ...);

Not we might read memory from a completely different address space as we would have resolved the address space to a number that makes sense for process1 but not process2.

So it can be done like you suggest, but it makes our API a bit less safe unless the address spec saves the process inside of it in case it tries to read from another process and then returns an error.

It is really designed to replace the lldb::addr_t load_addr variants without changing our entire source base.

Another option is to remove the read memory call that takes a addr_t and replace it with the AddressSpec variant and have an implicit constructor for AddressSpec that takes a single addr_t. This would allow all existing call sites to still function correctly without changing all of our code.

I look at it more as 99% of the people are going to be doing a simple memory read with addr_t load_addr and the very few percentage of clients will use the AddressSpec variant.

What do you think of switching to just a read/write memory that uses an AddressSpec instead of a addr_t and having an implicit constructor that converts a addr_t into an AddressSpec automatically?