The best way of generating a good representation for an array with header?

Project Infrastructure LLVM Dev List Archives
1

Yes, we’re actually viewing the struct at an offset.

So basically it’s a struct like this:

typedef struct {
  uint32_t size;
  uint32_t capacity;
  int array[0];
} Foo;

The whole thing is malloc:ed with extra bytes at the end, and capacity is set to that same number of extra bytes.

What’s then passed around is actually the int pointer at an offset: &(foo->array)

Using the that pointer we can obviously in a simple way recover the pointer to the struct, but can it be done so that LLVM and DWARF can identify the pointer as a pointer to a struct member for a certain struct?

std::vector is as far as I know wrapping a pointer or two.

The advantage of a stretchy buffer is that its length is recoverable even if stored as a pointer. It’s also incredibly thin, only taking up the same size as a pointer – as opposed to std::vector which is likely 2 pointers long.

Best Regards,

Christoffer

2

Yes, we’re actually viewing the struct at an offset.

So basically it’s a struct like this:

typedef struct {
uint32_t size;
uint32_t capacity;
int array[0];
} Foo;

The whole thing is malloc:ed with extra bytes at the end, and capacity is set to that same number of extra bytes.

What’s then passed around is actually the int pointer at an offset: &(foo->array)

Using the that pointer we can obviously in a simple way recover the pointer to the struct, but can it be done so that LLVM and DWARF can identify the pointer as a pointer to a struct member for a certain struct?

std::vector is as far as I know wrapping a pointer or two.

The advantage of a stretchy buffer is that its length is recoverable even if stored as a pointer.

What’s the advantage compared to a pointer to the struct, rather than a pointer to the array? (a pointer to this first element of the array would still have to be tagged differently from a pointer to an arbitrary int (either a singular int or an int somewhere in the array) to indicate that you can backtrack to find the length - so it’s not like you get to generalize all int pointers) - I wouldn’t expect (but don’t know that much) that the extra constant offset on array indexing would be particularly expensive/observable?

But yeah, I think you’d probably have some trouble getting DWARF consumers to handle the idea that the parameter type to a function is more than the type itself, or that pointers to that type actually point into the middle of the object instead of the start.

Not insurmountable, but seems a bit expensive/complicated to try to make that work - but don’t know what your other constraints/data are.

3

The advantages:

  1. A pointer to the struct offset can be converted to a pointer without any cost.
  2. A nullpointer to a stretchy buffer can be treated as a zero length array. Consequently no actual struct allocation is needed to represent a zero length array.
  3. A reference to the array is the same size as to a pointer.
  4. It can be converted to and back from an pointer without losing any information about the size & capacity.

The downsides are what we discuss. But it looks like I have to accept that I can only represent it as a pointer with unknown length in DWARF then?

Best Regards,

Christoffer

4

The advantages:

  1. A pointer to the struct offset can be converted to a pointer without any cost.
  2. A nullpointer to a stretchy buffer can be treated as a zero length array. Consequently no actual struct allocation is needed to represent a zero length array.

(2) could probably be done as well with the other representation (with the prefix data representation you’d still have to special case the null test before going backwards from the pointer to find the size - because you wouldn’t want to go backwards from null and try to read bytes there to find the size).

  1. A reference to the array is the same size as to a pointer.

(3) Would be true with either representation I’m picturing. (pointing to the start of the struct you’ve described, rather tahn pointing to the trailing array and walking backwards to find the rest)

  1. It can be converted to and back from an pointer without losing any information about the size & capacity.

Converting back you have to know it’s the start of an array, though, right? (& you could still do that in the other representation - by subtraction, but yes, wouldn’t be free/zero-cost)

The downsides are what we discuss. But it looks like I have to accept that I can only represent it as a pointer with unknown length in DWARF then?

I imagine it’d be difficult to describe the calling convention for passing this array? Are you going to have instances of this on the stack and passed by value? or only ever individually dynamically allocated & pointed to from other places?

If they were individually dynamically allocated & only ever pointed to the trailing array - the DWARF for that’d be pretty simple - basically just a special pointer type. The pretty printer for it would know it was allowed to walk backwards to find some extra things (as would any member functions, if needed, be implemented in terms of walking the poniter back).

  • dave
5

The advantages:

  1. A pointer to the struct offset can be converted to a pointer without any cost.
  2. A nullpointer to a stretchy buffer can be treated as a zero length array. Consequently no actual struct allocation is needed to represent a zero length array.

(2) could probably be done as well with the other representation (with the prefix data representation you’d still have to special case the null test before going backwards from the pointer to find the size - because you wouldn’t want to go backwards from null and try to read bytes there to find the size).

But that runs into (1).

In the stretchy buffer, converting to a pointer is essentially a NOP.

For the case you suggest, with the solution in (2) would be the following:

  1. Is it NULL? If so return NULL.
  2. Otherwise return current pointer + size of header as the pointer to the data.

There is then the question of what happens to indexing.

If we do the above conversion to pointer then we are doing an addition comparison and add.

For deref we can optimize away the NULL check for indexing though (assuming UB to index an array out of bounds), for x86 I think the add is included in indexing, but for ARM it’s not.

It might be a small cost to pay.

  1. A reference to the array is the same size as to a pointer.

(3) Would be true with either representation I’m picturing. (pointing to the start of the struct you’ve described, rather tahn pointing to the trailing array and walking backwards to find the rest)

Compared to implementations like std::vector I meant.

  1. It can be converted to and back from an pointer without losing any information about the size & capacity.

Converting back you have to know it’s the start of an array, though, right? (& you could still do that in the other representation - by subtraction, but yes, wouldn’t be free/zero-cost)

Here I’m again mostly thinking about std::vector.

The downsides are what we discuss. But it looks like I have to accept that I can only represent it as a pointer with unknown length in DWARF then?

I imagine it’d be difficult to describe the calling convention for passing this array? Are you going to have instances of this on the stack and passed by value? or only ever individually dynamically allocated & pointed to from other places?

The array is always passed as reference, but could potentially be put on the stack but would still always be used as a reference.

If they were individually dynamically allocated & only ever pointed to the trailing array - the DWARF for that’d be pretty simple - basically just a special pointer type. The pretty printer for it would know it was allowed to walk backwards to find some extra things (as would any member functions, if needed, be implemented in terms of walking the poniter back).

Alright, so then maybe this is solvable after all? Is there any docs on how to do this the proper way in LLVM? And perhaps also where to patch this into a fork of LLDB?

Best Regards,

Christoffer

6

The advantages:

  1. A pointer to the struct offset can be converted to a pointer without any cost.
  2. A nullpointer to a stretchy buffer can be treated as a zero length array. Consequently no actual struct allocation is needed to represent a zero length array.

(2) could probably be done as well with the other representation (with the prefix data representation you’d still have to special case the null test before going backwards from the pointer to find the size - because you wouldn’t want to go backwards from null and try to read bytes there to find the size).

But that runs into (1).

In the stretchy buffer, converting to a pointer is essentially a NOP.

For the case you suggest, with the solution in (2) would be the following:

  1. Is it NULL? If so return NULL.
  2. Otherwise return current pointer + size of header as the pointer to the data.

There is then the question of what happens to indexing.

If we do the above conversion to pointer then we are doing an addition comparison and add.

For deref we can optimize away the NULL check for indexing though (assuming UB to index an array out of bounds), for x86 I think the add is included in indexing, but for ARM it’s not.

It might be a small cost to pay.

  1. A reference to the array is the same size as to a pointer.

(3) Would be true with either representation I’m picturing. (pointing to the start of the struct you’ve described, rather tahn pointing to the trailing array and walking backwards to find the rest)

Compared to implementations like std::vector I meant.

  1. It can be converted to and back from an pointer without losing any information about the size & capacity.

Converting back you have to know it’s the start of an array, though, right? (& you could still do that in the other representation - by subtraction, but yes, wouldn’t be free/zero-cost)

Here I’m again mostly thinking about std::vector.

The downsides are what we discuss. But it looks like I have to accept that I can only represent it as a pointer with unknown length in DWARF then?

I imagine it’d be difficult to describe the calling convention for passing this array? Are you going to have instances of this on the stack and passed by value? or only ever individually dynamically allocated & pointed to from other places?

The array is always passed as reference, but could potentially be put on the stack but would still always be used as a reference.

If they were individually dynamically allocated & only ever pointed to the trailing array - the DWARF for that’d be pretty simple - basically just a special pointer type. The pretty printer for it would know it was allowed to walk backwards to find some extra things (as would any member functions, if needed, be implemented in terms of walking the poniter back).

Alright, so then maybe this is solvable after all? Is there any docs on how to do this the proper way in LLVM?

I’d model this after writing the equivalent C code - allocating the array, taking a pointer to the trailing portion. Passing that around as needed.

And perhaps also where to patch this into a fork of LLDB?

I don’t know much about how LLDB’s pretty printers work - shouldn’t require a fork, as pretty printers should be an extension point with python or the like & the pretty printer would be for the specific type you use for your “pointer to array with prefix data” & the pretty printer would be able to walk back the pointer and find data there.

So long as the debugger isn’t expected to be able to create new arrays of this type, or copy them (which it wouldn’t if they’re always dynamically allocated, never embedded within other objects, never passed by value, etc), I don’t think it’ll need to know anything much about them.