>
> > Hi David,
> >
> > The way I imagined that we might want to extend the MCStreamer API
(this
> > was motivated by DIEData) is by allowing clients to move bytes and
fixups
> > into the MC layer.
> >
> > This is the sort of API that I was imagining:
> >
> > void MoveAndEmitFragment(SmallVectorImpl<char> &&Data,
> > SmallVectorImpl<MCFixup> &&Fixups);
> >
> > Note that this mirrors the fields
> > MCEncodedFragmentWithContents::Contents and
> > MCEncodedFragmentWithFixups::Fixups
> > and the arguments could be directly moved into the fields of a newly
> > created
> > MCDataFragment.
> >
> > Would that work for your use case?
> >
>
> Not quite, unfortunately - the issue is that we're doing a task that is
> essentially "linking + a bit" - so imagine linking a bunch of files
> together, the final, say, debug_info.dwo section is made up of the
> concatenation of all the debug_info.dwo sections of the inputs. So it's
> fragmented and it's already available, memory mapped, never in a
> SmallVector, etc.
I see. I guess there's a couple of ways you can go with llvm-dwp:
1) Extend MC with optional ownership as you are doing in your patch.
2) Modify llvm-dwp to write object files directly.
2 is what lld does (with the help of libObject) and might not be such a bad
choice, but it would be adding a lot of machinery for a very specific task
that MC already needs to know how to do in a roughly target-independent
way,
so maybe it would be overkill.
Yeah, if lld's code for doing this were more reusable that might be an
option, but I assume it isn't. (alternatively, could move dwp tool to be a
subproject of lld itself, a "dwp" driver that would just enable the special
treatment of cu/tu_index sections) At least for my needs, the modifications
to MC seem sufficiently unobtrusive & potentially generally useful
(eventually LLVM might care about the memory impact of MC - perhaps for
especially weird inputs (large amounts of static data, for example)).
I reckon that in most cases MC clients aren't going to be copying large
amounts of unowned data, they're most likely going to be creating that
data. So perhaps the implementation should reflect that somehow.
Yeah, even for that, though - they may not want to keep it all in memory.
For example one of the next largest memory costs in dwp is the str_offsets
section, where we emit a bunch of ints created by processing the input. I
know how big the output will be, but I don't want/need to allocate a vector
of all of them, if I could stream them out instead.
So in theory we could generalize more aggressively, rather than narrow down
the usage - if I could pass a thing that could be queried for size and
could write bytes to the underlying entity I could save that memory too. So
could LLVM - for example, type units wouldn't need to all be stored in
bytes before writing any part of them out, we could stream it out to disk.
(there's some buffering in Clang that adds another layer to get through -
so it's actually buffered twice, the MC changes only remove one layer, we'd
have to change clang (& change MC to not require pwrite to patch the
header) to avoid the buffering entirely)
Specifically what I had in mind was that you could add some other derived
class
of MCFragment that would store a StringRef (or maybe a vector of StringRefs
if that proves useful), and that would be unrelated to MCDataFragment.
(needs multiple StringRefs - the output section consists of the
concatenation of all the input sections - so in the simple case it's a
StringRef from each input)
But yeah, could possibly narrow down the usage. I haven't looked closely at
how the MCFragments are created/used/manipulated (some of this conversation
may be better placed in the Differential review thread, perhaps - I do
really appreciate your perspective)