RFC: LLDB Telemetry/metrics

Subprojects LLDB
14

P.S: Please see the updated design proposal below.

Objective

  • Fine-grained understanding of LLDB and LLDB clients (lldb-dap, et al) performance, resource usage and bottlenecks.
  • Better understanding of debugger usage

=> Help drive possible improvements to LLDB.

Overview

A telemetry framework will be added to upstream LLDB to collect data on various events, such as debugger startup/exit, process(debugged targets) startup/exit, platform info, and command executions, etc.

Collected data will then be sent to vendor-specific plugins which can decide what to do with the data. It is up to the vendor to enable or disable telemetry altogether. Tthe framework will also allow vendors to optionally collect some additional data, as needed. Specifically, a vendor can:

  • Provide concrete definition for lldb_private::GetTelemetryConfig():
    • This specifies whether Telemetry is enabled. (“disabled” by default)
    • This also specifies a set of Destination to forward the data to. (STDOUT by default)
  • Provide one or more concrete definitions for Destination
    • Note: vendors must define their own Destination. Upstream LLDB does not provide this (perhaps except for a dummy STDOUT Destination - see Privacy&Security section below for rationale)
  • (Optional) Provide a custom (subclass) TelemetryLogger which collects additional data.

If a vendor does not do any of the above, then Telemetry will be disabled.

Detailed design.

in_1k_words
in_1k_words1181×843 66.4 KB

DebuggerInfoEntry

(A session = when an LLDB instance starts until it quits.)

This represents the debugger (and also indirectly a session).

  • Important subfields include:
    • username: the user running LLDB
    • lldb_git_sha: the git version from which this LLDB was built
    • lldb_path: path to this LLDB
    • cwd: current working directory
  • A DebuggerInfoEntry is emitted when LLDB starts
    • timestamp when LLDB starts
    • Optional: how long the startup process takes
  • Another DebuggerInfoEntry is emitted when LLDB shuts down.
    • timestamp when LLDB shuts down and error status, if any.
    • Optional: how long the shutdown process took

This helps answering

  • Who are running LLDB?
  • What versions of LLDB are used?
  • How long each session typically takes?
  • How often LLDB quits with exit 0 vs error?
  • Usages of LLDB command line vs via DAP (deducible based on the lldb_path (argv[0]) )

TargetInfoEntry

This represents a main executable (and also indirectly a process).

  • Subfields include:
    • binary_path: Path to this executable
    • binary_size: The size (in bytes).
    • other aux info
  • A TargetInfoEntry is emitted when a main executable is loaded (that is, whenever Target::SetExecutable() is called).
    • timestamp when the load started
    • timestamp when it finished.
      • Since LLDB can do things lazily - it’s possible we don’t have accurate time measurement for the main-executable load here.
    • Optionally: auxiliary data about the main target can also be collected here (vendor-specific)
  • Another TargetInfoEntry is emitted when a process exits.
    • timestamp when it exits
    • Exit code / error msg if available.

This helps answering:

  • Which target was debugged with LLDB?
  • How long it took to load it? (Does the time correlate with its size?)
    • Note: as mentioned, LLDB’s laziness can affect the load time measurement here?
  • Binary/process exit status

CommandInfoEntry

This represents a single command

  • Subfields include:
    • target_uuid: available if this command can be associated with a target
    • command_name: (eg., breakpoint set)
    • original_command: The command as typed by user
    • args: arguments to the command.
  • A CommandInfoEntry is emitted when a command starts executing
  • Another CommandInfoEntry is emitted after the command finished

This helps answering

  • Which commands were used?
  • How long did it take?
  • Success vs failure?

ClientTelemetryInfo

This represents a request from a client

  • subfields include:
    • request_name: name of the request
    • error_msg: any error fro
  • A ClientTelemetryInfo is emitted for each client request.

MiscInfoEntry

This is a catch-all entry which allows vendors to collect and record any additional specific to their organizations.

TelemetryConfig

  • Vendor can define a set of default configigurations.
  • LLDB users can also supply their own configs via: ~/.lldb/lldb_telemetry_config
    • Note:
      • We decided to use a separate file instead of the existing settings file because that file is parsed too late in the process and by then there might have been lots of telemetry-entries that need to be sent already. This approach avoid losing entries if LLDB crashes during init.
      • Alternative considered $XDG_CONFIG_HOME/.lldb_telemetry_config. (XDG is not really a thing on windows, hence was rejected)
    • lldb_telemetry_config shall have simple syntax: (<field_name><colon><value>)*
      • Eg.,
        • enable_logging:true
        • destination:stdout
        • destination:stderr
        • destination:/path/to/some/file (<= Vendors’ decision whether to allow this)

Destination
As much as possible, the implementations of this interface should do all the data forwarding in a separate thread to avoid slowing down developers workflow.

Privacy and security concerns

Since different organizations can have different privacy models, the upstream Telemetry framework will not make a decision on which piece of data is PII-risk. It will also not store the data directly anywhere.

Its responsibility is simply to instrument and collect the data, as specified above, then forward it to these Destination classes, which are defined by vendors.