DS2 is a debug server originally written at Facebook. Some years ago, Facebook stopped investing in it and archived the project at GitHub - facebookarchive/ds2: Debug server for lldb. · GitHub . Facebook has agreed to donate the code to the LLVM project. I’d like to propose incubating the project within the LLVM organisation.
This code is pretty portable, and the codebase has been known to run on at least (with patches):
macOS (x86_64)
Linux (aarch64, armv7, i686, riscv64, x86_64)
Windows (armv7, i686, x86_64)
FreeBSD (aarch64, x86_64)
Tizen (armv7, i686)
Android (aarch64, armv7, i686, x86_64)
Feature-wise, it was sufficiently complete to enable remote debugging workflows, supporting several LLDB and GDB extensions. When it was under active development, it was tested against the GDB and LLDB test suites to ensure correctness.
Dependency-wise, it requires flex/bison for building and a C++ runtime (libc++/msvcprt/libstdc++ are known to work). The compiled binary on Windows, IIRC was on the order of ~1MiB.
While lldb-gdbserver exists, the LLVM dependencies cause significant size bloat. Especially with any external language support added to LLVM (e.g. Swift), this binary ends up being quite dramatically larger—IIRC, O(150)MiB in release mode with DCE and significant patching.
I also have several patches in a fork ( GitHub - compnerd/ds2: DebugServer for LLDB · GitHub ) which I would bring over to update the code, fix bugs, and add features (such as RISCV support and pipe support on Windows).
The lean debug server should also enable remote debugging for other environments (embedded, GPU).
I agree with the motivation and that this is a problem worth solving. The binary size is a long standing issue and something I’ve seen you and others (like @bulbazord) work on for what must now be almost a decade. The layering and the dependency graph is a real problem and I’m well aware that it’s even worse downstream on the Swift fork. I also agree that the size of the debug stub is important, especially for constrained targets. Besides the fact that debugserver predates LLDB, it’s also part of the reason we never adopted it on Darwin.
However, I am hesitant about incubating DS2. The proposed solution spends the community’s energy on the symptom while leaving the cause in place. The size bloat comes from lldb-server’s coupling to the rest of LLDB. If we (continue to) invest in decoupling that, we solve the size problem for everyone rather than for the subset of users who adopt a second server. A parallel implementation also carries a lasting cost of its own: someone has to commit to maintaining not just the stub, but the bots, pre-commit CI, and the ongoing parity for things like protocol extension that keeps it in step with the rest of LLDB.
Before we consider a parallel implementation, I’d like to understand (1) what’s the smallest lldb-server we can build and (2) what it would take to get there. If the answer is that the current design makes a small debug stub intractable, then maybe we should reconsider lldb-server’s design and start fresh and focus the community’s energy there. I’m not familiar with DS2’s implementation, but in that case DS2 might be a good starting point. But I’d want to reach that conclusion before adopting a parallel implementation rather than after.
Bringing this codebase in to the project (even if we don’t use it) is low cost (and highly recommended) since that requires Meta’s involvement to donate.
Deciding between the debug stubs.
For DS2, it might even be interesting to see if we can trim it further. I do agree that we want to reduce the size for everyone. I think that if we find that DS2 is able to serve all the users, it would be a reasonable path to adopt that as the debug server that we use rather than maintain 2 separate debug servers.
As to understanding the limits of lldb-server, I would say that the past decade of work should hint towards that. There is the other aspect of lldb-server is not as easy to work with for other environments (e.g. Windows). Does lldb-server already support NT pipes and the like? Adding that level of target customization is relatively easy in DS2.
I’m concerned that hinding the donation on the full exploration of the space might mean that we would lose the DS2 donation.
Disclosure: The majority of my job for the last few years has been keeping lldb-server up to date with AArch64 Linux features.
I’ve also spent a lot of time explaining when and where to use the 2 servers we have, so the idea of a third doesn’t immediately appeal to me. Though I do understand that incubation does not make ds2 the third in-tree server. Just my initial thought when I saw this.
I see that the original code is under the same license as LLVM. So could you explain the difference between simply importing this and it being “donated”?
I see that your fork is not under the same license. So I assume donation would include your fork, maybe other changes from Meta employees, and we’re less likely to need to talk to a lawyer if the original copyright holder is on board?
I don’t understand the point you are making here, but I don’t know if I’m parsing the sentence correctly.
It reads like “highly recommended since that requires Meta’s involvement to donate”.
However:
Why does it require Meta’s involvement? Are there trademarks and other non-code things that must be transferred?
Is there a limited time window to do this transfer? Sounds like you’re saying we better import it now before something expires.
Covering all users does not have to be a condition of inclusion, but I do want to make it clear that “all users” is currently defined in a broader way than you might expect.
I think we will have to be more strict on that topic eventually, but that’s unrelated to whether lldb-server or ds2 is the future. So I would not use the possibility of a ds2 only future as a reason to incubate it. Could happen, but so many unknowns on that path.
Can you be explicit what this hint is hinting at? Not all of us have this context.
Without context, I can flip the argument: the first commit to ds2 was 2014, that’s 12 years ago and it hasn’t taken over the world either. Maybe it took over Meta’s world, and if so I’d like to hear more about how successful that was.
I am currently refactoring AArch64 Linux code in lldb-server because it was hard to extend, so I agree there are limits in lldb-server, and fresh ideas are welcome.
I cannot explain that honestly, this was done at the request of Apple due to some objection around copyright rather than anything else. The donation is to simply resolve that.
It is under the same license - it was forked, I did not change the license, merely made improvements. Yes, I would bring my changes over as well. But, yes, the idea is that because the original copyright holder is onboard, there should be no issue about that.
I’m concerned about the time to respond - Meta is a large company and people move about. If we were to lose time debating things and might lose the attention of the people involved to make this happen. At that point, we would have to then consider either forking OR rebuilding this from scratch. I’m suggesting that we accept the donation - and then can decide whether to persue the use or not separately.
That is fair. I think that the point is that this is something that is valuable enough to people that we should consider incubating the project for now. We don’t need to make a decision this moment.
Sure, I was hinting at the fact that the dependency problem today and the use of static linking has resulted in the debug server being both difficult to evolve (i.e. there is a desire to use the LLVM APIs for portability but that pulls in more code) and shrink at the same time.
I think that this states it plainly: there are limits that we keep running up against. This might be a good option to also entertain.
The idea is that we can incubate the project and compare the two practically. If we find that it doesn’t work we can simply write it off. But if we find that it really does work well for sufficiently many people, we can try to consolidate efforts.
I wasn’t involved in this and I’m not a lawyer, but looking at the original repository, I see there was an CLA which I imagine complicates things. On the topic of licensing, did we run this by the LLVM Foundation yet, or are we holding off on that until we’ve reached consensus here?
Do we need to incubate before we can do this comparison? It should be fairly straightforward to swap out the stub and run our test suite against it on say Linux and Windows and compare the results. I think that would help make a stronger technical case.
I tried to avoid expressing an opinion as to the future of lldb-server in my previous reply in order not to derail the discussion, but I’m genuinely interested in your opinion @DavidSpickett as you’re by far the most active contributor to it these days.
With the caveat that I’m basing all of this on second hand experience, it’s my understanding has been that the code-sharing idea between LLDB and the stub didn’t work out as well as we hoped. I’m sure there’s a lot of “data” (think things like register definitions) that is worth sharing between the two, but maybe not so much code/logic (assuming we’ve given up on the “in-process” debugging support that was envision originally).
The main thing I’m trying to avoid is the equivalent of the standards xkcd for debug stubs in tree. I’m all for supporting as many stubs as possible, just not maintaining them.
I have not, but, the code was already re-licensed for donation to Apache 2 and removed the CLA.
No, I don’t think that we need to do that - in fact the exact opposite: the “incubation” is more to have a destination for the code rather than the process itself. I am just trying ot hedge against it getting lost in the normal day-to-day operation of large corporations.
But, yes, everything that we need should be there. We were actually running the LLDB test suite against it when in development. Recently, we’ve not had as much time to really dig into the test coverage, but there were some runs a few months ago on my fork for a small subset.
I’m not involved in this effort, but I’d like to support Saleem’s proposal. Essentially, it gives us a chance to take ownership of ds2, resolve potential non-technical complications while time permits, and then decide its fate. ds2 might be a good starting point.
I would be happy to see ds2 continue development. I’m biased though since I’m a former contributor.
Given the discussion so far, I have a few questions:
It’s an anti-goal to maintain 3 debug servers. We need to go back to 2 at the very least. If your goal is to eventually replace lldb-server, how will you achieve that? What kind of timeline are you thinking about? I assume the answer to this will change over time but what are your expectations going into this?
If incubated, what are the expectations of maintaining it? For example, if a change in lldb inadvertently breaks ds2, who is expected to fix that? Will there be CI or automated testing? Who will provide those if so? This question is primarily aimed at incubation time. If ds2 replaces lldb-server, I imagine that the expectations of now would carry over.
Suppose ds2 is incubated but there is not enough consensus or momentum to switch over to it after the proposed timeline has elapsed. Would you find it acceptable for ds2 to be removed from the tree?
There is definitely a strong consensus here. Jonas and I both basically said the same thing: we need to reduce the number of debug servers. I think that the timeline is definitely going to be a bit longer side. In order to replace lldb-server, we first need to ensure that DS2 is able to work on the current places that users are actively using it. If we can get it stable and complete enough to switch to by default, we can start defaulting it on platforms. I’m thinking Windows first, but Linux and Android are likely good next targets. At a certain point I imagine that the bulk of users would be DS2 and we could switch the default.
Yes, we would need to figure out CI. Fortunately, the build time for DS2 is pretty low, its testing that takes a while. I believe that I should be able to get a host or two for this through The Browser Company. For incubation, I believe that the general expectation is that it is not expected to be the responsibility of the LLDB developer to repair the issue (though it is always appreciated).
I think that this needs to be a requirement - we cannot proliferate the debug servers.
The goal is not to grow the number of debug servers, the goal is to try to come up with a solution to the problem of platform heterogeneity and debug server size. I think that having one that has a track record for both gives us a good way to compare and make an informed choice.
But regardless, if your point is there are other issues and history to this, that would be solved during the donation process, that makes sense to me and I understand the relative urgency of the proposal.
It is a thing I can and will solve in lldb-server, for AArch64 Linux. However due to time, employer priorities and lack of access to hardware, I’m not going to roll it out to all variants of Linux, never mind the other Unixes.
So it is a good candidate to be done in a side project that has fewer things to support at first.
Specifically ds2 seems to have a data driven definition of registers. Do I think that might fall apart when I pile on all the AArch64 extensions? Probably, but having a place to find out would be cool.
I think it’s a question of how many ideas can we try within ds2. We have a couple stated already:
Binary size / layering / dependencies.
Extensibility re. adding a new platform or extensions to that platform.
I see hints of register configuration being data driven which is where I have been heading anyway, I’ll look deeper.
I cannot make commitments to working on ds2 myself right now, as Arm’s priority is going to be lldb-server as it’s “the” debug server for our use cases. That’s not necessarily a bad thing though because I wouldn’t mind a fresh look from someone else at things like scalable vectors.
Is there anyone who wants to declare an intention to work on an incubated ds2 and state what they think they will / would like to work on there?
(priorities change I know, but just roughly)
I know there’s some time limit here, but I don’t think the foundation is going to take kindly to being a host “just in case” someone wants to work on it later.
That’s why I was asking about the licensing, because it looked like we could just fork it any time we wanted, and it seems I was wrong about that.
As long as I’ve been here it seemed to me that in process debugging was never a goal of lldb. It was always to use a separate server. On Windows it seemed like an accident of history that it was in process.
It does seem that more data is shared than logic. Signals, registers, but not the handling of them. Packet definitions probably should be shared, but they have bespoke handling on either side. Maybe another thing we could experiment with.
I think we “solve” this by not making this a goal of incubation, and stating that incubation does not support or detract from that possibility.
That means that a condition of incubation has to be that people will actually do experiments with ds2, otherwise what’s the point beyond preservation.
Multiple years I would expect. Given that we have some platform support that is entirely written by the platform vendor or by one person from a small community. So again it should not be a goal of incubation, but important to set expectations about a timeline.
(in theory if ds2 got this far, the work would be less than the work required on lldb-server, but I think the bigger issue is persuading companies to allocate staff)
I do think it’s worth doing this but do remember there is a tension between the ability of ds2 to be a good experimental thing, and it adhering to the existing tests. Nice if you can have both, but you usually cannot.
And see that SVE/TLS/PAC/non-address bits/watchpoint tests are all skipped (the latter maybe because virtual machine limitations though).
This would make replacing lldb-server at this time an absolute no, but for incubation as a place to try new ideas, I think it’s actually a good thing. These features add significant complexity that has to be tackled eventually, but it shouldn’t get in the way of other things in the meantime.
I believe that they want the donation to at least be in a different repo; so I would say that we should just keep it there and once we have a final decision we can merge it into the LLVM repository (or if we find that its more convenient outside of it, outside).
Sharing my findings on the size of the lldb-server.exe stub. The following numbers are for release builds:
LLVM
In release builds, lldb-server.exe is 16.2 MB.
With some refactoring, I was able to shrink it down to 8 MB.
Swiftlang
In release builds, lldb-server.exe is 123 MB.
With some refactoring I was able to shrink it down to 19MB.
Stacking that with the refactoring in LLVM, that should bring it down to ~10 MB.
Those refactors are not merged yet and are still a work in progress. I’m confident that we can find more ways to reduce the size of lldb-server on Windows both in LLVM and in Swiftlang.
I think that levels the playing field between DS2 and lldb-server on Windows.
It’s also not clear to me what the cost of bringing DS2 up to par with lldb-server.exe is. I assume it’s meant as a drop-in replacement but how much of the lldb test suite passes with DS2 instead of lldb-server.exe? If it takes more effort to fix tests with DS2 compared to shrinking lldb-server.exe, I don’t (currently) see a strong motivation for adopting DS2.
Given that, this RFC is about incubating DS2. I think incubating it is low cost for us. Accepting the donation does not mean we adopt it and make it the default stub. I’m ok with accepting the donation in the llvm codebase, however we should start another RFC thread to further decide if we want to work on it and adopt it.
I don’t have experience incubating codebases: I want to get others’ approval before we move forward.
Active community: I see people who would like to have this, but not necessarily help maintain it.
Needs an active maintainer: @compnerd are you volunteering? If not, then who?
Reasonable documentation how it works?
Reasonable CI? Automated from LLVM commits?
Generally, I’m worried about the “donation” spin. To me this feels like code dump, and those are never good. “Free stuff” are only worth accepting if it brings value that outweighs the cost (CI, maintenance, unrelated test crashes, etc.).
So, if there’s an active community, active maintainers, actually trying to merge this into LLVM and be an alternative to lldb-server (especially reusing / maintainer core infrastructure around both), then this should be a win for LLVM. Otherwise, this will cost more than any potential benefit.
).