ppc64le and 32-bit LE userland compatibility

Hey all, a couple of us over in #talos-workstation on freenode have been
working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
designated for this (unless you count the patchset from a decade ago [1]), so
it's pretty much uncharted territory as far as Linux is concerned. We want to
sync up with libc and the relevant kernel folks to establish the best path
forward.

The practical application that drove these early developments (as you might
expect) is x86 emulation. The box86 project [2] implements a translation layer
for ia32 library calls to native architecture ones; this way, emulation
overhead is significantly reduced by relying on native libraries where
possible (libc, libGL, etc.) instead of emulating an entire x86 userspace.
box86 is primarily targeted at ARM, but it can be adapted to other
architectures—so long as they match ia32's 32-bit, little-endian nature. Hence
the need for a ppcle userland; modern POWER brought ppc64le as a supported
configuration, but without a 32-bit equivalent there is no option for a 32/64
multilib environment, as seen with ppc/ppc64 and arm/aarch64.

Surprisingly, beyond minor patching of gcc to get crosscompile going,
bootstrapping the initial userland was not much of a problem. The work has
been done on top of the Void Linux PowerPC project [3], and much of that is
now present in its source package tree [4].

The first issue with running the userland came from the ppc32 signal handler
forcing BE in the MSR, causing any 32LE process receiving a signal (such as a
shell receiving SIGCHLD) to terminate with SIGILL. This was trivially patched,
along with enabling the 32-bit vDSO on ppc64le kernels [5]. (Given that this
behavior has been in place since 2006, I don't think anyone has been using the
kernel in this state to run ppcle userlands.)

The next problem concerns the ABI more directly. The failure mode was `file`
surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
garbage value for `pos`, which didn't appear to be caused by anything in
`file`. Initially it seemed as though the 32-bit components of the arg were
getting swapped, and we made hacky fixes to glibc and musl to put them in the
"right order"; however, we weren't sure if that was the correct approach, or
if there were knock-on effects we didn't know about. So we found the relevant
compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
there exists this comment:

/*
* long long munging:
* The 32 bit ABI passes long longs in an odd even register pair.
*/

It seems that the opposite is true in LE mode, and something is expecting long
longs to start on an even register. I realized this after I tried swapping hi/
lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
switches which syscalls use padding arguments depending on endianness, while
hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
variable names/types so that the macro could be consistent.)

This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
though, and there is still concern over the right way to approach this,
whether it should live in the kernel or libc, etc. Frankly, I don't know the
ABI structure enough to understand why the register padding has to be
different in this case, or what lower-level component is responsible for it.
For comparison, I had a look at the mips tree, since it's bi-endian and has a
similar 32/64 situation. There is a macro conditional upon endianness that is
responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead
of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that.
(It also simply swaps registers for LE, unlike what I did for ppc.)

Also worth noting is the one other outstanding bug, where the time-related
syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
endian bug to me, and it doesn't affect standard syscalls (which is why if you
run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm
all ears.

Again, I'd appreciate feedback on the approach to take here, in order to
touch/special-case only the minimum necessary, while keeping the kernel/libc
folks happy.

Cheers,
Will [she/her]

(p.s. there is ancillary interest in a ppcle-native kernel as well; that's a
good deal more work and not the focus of this message at all, but it is a
topic of interest)

[1]: https://lwn.net/Articles/408845/
[2]: https://github.com/ptitSeb/box86
[3]: https://voidlinux-ppc.org/
[4]: https://github.com/void-ppc/void-packages
[5]: https://gist.github.com/eerykitty/01707dc6bca2be32b4c5e30d15d15dcf
[6]: https://gist.github.com/Skirmisher/02891c1a8cafa0ff18b2460933ef4f3c

The next problem concerns the ABI more directly. The failure mode was `file`
surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
garbage value for `pos`, which didn't appear to be caused by anything in
`file`. Initially it seemed as though the 32-bit components of the arg were
getting swapped, and we made hacky fixes to glibc and musl to put them in the
"right order"; however, we weren't sure if that was the correct approach, or
if there were knock-on effects we didn't know about. So we found the relevant
compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
there exists this comment:

> /*
> * long long munging:
> * The 32 bit ABI passes long longs in an odd even register pair.
> */

It seems that the opposite is true in LE mode, and something is expecting long
longs to start on an even register. I realized this after I tried swapping hi/
lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
switches which syscalls use padding arguments depending on endianness, while
hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
variable names/types so that the macro could be consistent.)

The argument passing for pread/pwrite is historically a mess and
differs between archs. musl has a dedicated macro that archs can
define to override it. But it looks like it should match regardless of
BE vs LE, and musl already defines it for powerpc with the default
definition, adding a zero arg to start on an even arg-slot index,
which is an odd register (since ppc32 args start with an odd one, r3).

[6]: https://gist.github.com/Skirmisher/02891c1a8cafa0ff18b2460933ef4f3c

I don't think this is correct, but I'm confused about where it's
getting messed up because it looks like it should already be right.

This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
though, and there is still concern over the right way to approach this,
whether it should live in the kernel or libc, etc. Frankly, I don't know the
ABI structure enough to understand why the register padding has to be
different in this case, or what lower-level component is responsible for it..
For comparison, I had a look at the mips tree, since it's bi-endian and has a
similar 32/64 situation. There is a macro conditional upon endianness that is
responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead
of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that.
(It also simply swaps registers for LE, unlike what I did for ppc.)

Indeed the problem is probably that you need to swap registers for LE,
not remove the padding slot. Did you check what happens if you pass a
value larger than 32 bits?

If so, the right way to fix this on the kernel side would be to
construct the value as a union rather than by bitwise ops so it's
endian-agnostic:

  (union { u32 parts[2]; u64 val; }){{ arg1, arg2 }}.val

But the kernel folks might prefer endian ifdefs for some odd reason...

Also worth noting is the one other outstanding bug, where the time-related
syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
endian bug to me, and it doesn't affect standard syscalls (which is why if you
run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm
all ears.

Not sure about this. Worst-case, just leave it disabled until someone
finds a fix.

Rich

[...]

Also worth noting is the one other outstanding bug, where the time-related
syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
endian bug to me, and it doesn't affect standard syscalls (which is why if you
run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm
all ears.

There is a series at https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231 to switch powerpc to the Generic C VDSO.

Can you try and see whether it fixes your issue ?

Christophe

Hi!

Hey all, a couple of us over in #talos-workstation on freenode have been
working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
designated for this

https://www.polyomino.org.uk/publications/2011/Power-Arch-32-bit-ABI-supp-1.0-Embedded.pdf

(unless you count the patchset from a decade ago [1]), so

The original sysv PowerPC supplement
http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
supports LE as well, and most powerpcle ports use that. But, the
big-endian Linux ABI differs in quite a few places, and it of course
makes a lot better sense if powerpcle-linux follows that.

it's pretty much uncharted territory as far as Linux is concerned. We want to
sync up with libc and the relevant kernel folks to establish the best path
forward.

The practical application that drove these early developments (as you might
expect) is x86 emulation. The box86 project [2] implements a translation layer
for ia32 library calls to native architecture ones; this way, emulation
overhead is significantly reduced by relying on native libraries where
possible (libc, libGL, etc.) instead of emulating an entire x86 userspace.
box86 is primarily targeted at ARM, but it can be adapted to other
architectures—so long as they match ia32's 32-bit, little-endian nature. Hence
the need for a ppcle userland; modern POWER brought ppc64le as a supported
configuration, but without a 32-bit equivalent there is no option for a 32/64
multilib environment, as seen with ppc/ppc64 and arm/aarch64.

Surprisingly, beyond minor patching of gcc to get crosscompile going,

What patches did you need? I regularly build >30 cross compilers (on
both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
in the past those worked fine as well). I also cross-built
powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
from various x86).

bootstrapping the initial userland was not much of a problem. The work has
been done on top of the Void Linux PowerPC project [3], and much of that is
now present in its source package tree [4].

The first issue with running the userland came from the ppc32 signal handler
forcing BE in the MSR, causing any 32LE process receiving a signal (such as a
shell receiving SIGCHLD) to terminate with SIGILL. This was trivially patched,

Heh :slight_smile:

along with enabling the 32-bit vDSO on ppc64le kernels [5]. (Given that this
behavior has been in place since 2006, I don't think anyone has been using the
kernel in this state to run ppcle userlands.)

Almost no project that used 32-bit PowerPC in LE mode has sent patches
to the upstreams.

The next problem concerns the ABI more directly. The failure mode was `file`
surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
garbage value for `pos`, which didn't appear to be caused by anything in
`file`. Initially it seemed as though the 32-bit components of the arg were
getting swapped, and we made hacky fixes to glibc and musl to put them in the
"right order"; however, we weren't sure if that was the correct approach, or
if there were knock-on effects we didn't know about. So we found the relevant
compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
there exists this comment:

> /*
> * long long munging:
> * The 32 bit ABI passes long longs in an odd even register pair.
> */

It seems that the opposite is true in LE mode, and something is expecting long
longs to start on an even register. I realized this after I tried swapping hi/
lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
switches which syscalls use padding arguments depending on endianness, while
hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
variable names/types so that the macro could be consistent.)

The ABI says long longs are passed in the same order in registers as it
would be in memory; so the high part and the low part are swapped between
BE and LE. Which registers make up a pair is exactly the same between
the two. (You can verify this with an existing powerpcle-* compiler, too;
I did, and we implement it correctly as far as I can see).

This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
though, and there is still concern over the right way to approach this,
whether it should live in the kernel or libc, etc. Frankly, I don't know the
ABI structure enough to understand why the register padding has to be
different in this case, or what lower-level component is responsible for it.
For comparison, I had a look at the mips tree, since it's bi-endian and has a
similar 32/64 situation. There is a macro conditional upon endianness that is
responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead
of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that.
(It also simply swaps registers for LE, unlike what I did for ppc.)

But you should :slight_smile:

Also worth noting is the one other outstanding bug, where the time-related
syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
endian bug to me, and it doesn't affect standard syscalls (which is why if you
run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm
all ears.

Again, I'd appreciate feedback on the approach to take here, in order to
touch/special-case only the minimum necessary, while keeping the kernel/libc
folks happy.

A huge factor in having good GCC support for powerpcle-linux (or anything
else) is someone needs to regularly test it, and share test results with
us (via gcc-testresults@). Hint hint hint :slight_smile:

That way we know it is in good shape, know when we are regressing it,
know there is interest in it.

gl;hf,

Segher

Sure thing, I spotted that after making the initial post. Will report back
with results.

Will [she/her]

The argument passing for pread/pwrite is historically a mess and
differs between archs. musl has a dedicated macro that archs can
define to override it. But it looks like it should match regardless of
BE vs LE, and musl already defines it for powerpc with the default
definition, adding a zero arg to start on an even arg-slot index,
which is an odd register (since ppc32 args start with an odd one, r3).

> [6]:
> https://gist.github.com/Skirmisher/02891c1a8cafa0ff18b2460933ef4f3c
I don't think this is correct, but I'm confused about where it's
getting messed up because it looks like it should already be right.

Hmm, interesting. Will have to go back to it I guess...

> This was enough to fix up the `file` bug. I'm no seasoned kernel
> hacker, though, and there is still concern over the right way to
> approach this, whether it should live in the kernel or libc, etc.
> Frankly, I don't know the ABI structure enough to understand why the
> register padding has to be different in this case, or what
> lower-level component is responsible for it.. For comparison, I had a
> look at the mips tree, since it's bi-endian and has a similar 32/64
> situation. There is a macro conditional upon endianness that is
> responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__
> instead of an if/else on the generic __LITTLE_ENDIAN__. Not sure what
> to make of that. (It also simply swaps registers for LE, unlike what
> I did for ppc.)
Indeed the problem is probably that you need to swap registers for LE,
not remove the padding slot. Did you check what happens if you pass a
value larger than 32 bits?

If so, the right way to fix this on the kernel side would be to
construct the value as a union rather than by bitwise ops so it's
endian-agnostic:

  (union { u32 parts[2]; u64 val; }){{ arg1, arg2 }}.val

But the kernel folks might prefer endian ifdefs for some odd reason...

You are right, this does seem odd considering what the other archs do.
It's quite possible I made a silly mistake, of course...

I haven't tested with values outside the 32-bit range yet; again, this is
new territory for me, so I haven't exactly done exhaustive tests on
everything. I'll give it a closer look.

> Also worth noting is the one other outstanding bug, where the
> time-related syscalls in the 32-bit vDSO seem to return garbage. It
> doesn't look like an endian bug to me, and it doesn't affect standard
> syscalls (which is why if you run `date` on musl it prints the
> correct time, unlike on glibc). The vDSO time functions are
> implemented in ppc asm (arch/powerpc/kernel/vdso32/ gettimeofday.S),
> and I've never touched the stuff, so if anyone has a clue I'm all
> ears.
Not sure about this. Worst-case, just leave it disabled until someone
finds a fix.

Apparently these asm implementations are being replaced by the generic C
ones [1], so it may be this fixes itself on its own.

Thanks,
Will [she/her]

[1]: https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231

The original sysv PowerPC supplement
http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
supports LE as well, and most powerpcle ports use that. But, the
big-endian Linux ABI differs in quite a few places, and it of course
makes a lot better sense if powerpcle-linux follows that.

Right, I should have clarified I was talking about Linux ABIs
specifically.

What patches did you need? I regularly build >30 cross compilers (on
both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
in the past those worked fine as well). I also cross-built
powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
from various x86).

There was just an assumption that LE == powerpc64le in libgo, spotted by
q66 (daniel@ on the CC). I just pushed the patch to [1].

Almost no project that used 32-bit PowerPC in LE mode has sent patches
to the upstreams.

Right, but I have heard concerns from at least one person familiar with
the ppc kernel about breaking existing users of this arch-endianness
combo, if any. It seems likely that none of those use upstream, though ^^;

The ABI says long longs are passed in the same order in registers as it
would be in memory; so the high part and the low part are swapped
between BE and LE. Which registers make up a pair is exactly the same
between the two. (You can verify this with an existing powerpcle-*
compiler, too; I did, and we implement it correctly as far as I can
see).

I'll give it a closer look. This is my first time poking at this sort of
thing in depth, so excuse my unfamiliarity!

A huge factor in having good GCC support for powerpcle-linux (or
anything else) is someone needs to regularly test it, and share test
results with us (via gcc-testresults@). Hint hint hint :slight_smile:

That way we know it is in good shape, know when we are regressing it,
know there is interest in it.

Once I have more of a bootstrapped userland than a barely-functional
cross chroot, I'll get back to you on that :slight_smile:

gl;hf,

Segher

Thanks,
Will [she/her]

[1]: https://github.com/Skirmisher/void-packages/blob/master/srcpkgs/gcc/patches/libgo-ppcle.patch

> The original sysv PowerPC supplement
> http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
> supports LE as well, and most powerpcle ports use that. But, the
> big-endian Linux ABI differs in quite a few places, and it of course
> makes a lot better sense if powerpcle-linux follows that.

Right, I should have clarified I was talking about Linux ABIs
specifically.

That was the link you deleted.

> What patches did you need? I regularly build >30 cross compilers (on
> both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
> in the past those worked fine as well). I also cross-built
> powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
> from various x86).

There was just an assumption that LE == powerpc64le in libgo, spotted by
q66 (daniel@ on the CC). I just pushed the patch to [1].

Please send GCC patches to gcc-patches@ ?

> Almost no project that used 32-bit PowerPC in LE mode has sent patches
> to the upstreams.

Right, but I have heard concerns from at least one person familiar with
the ppc kernel about breaking existing users of this arch-endianness
combo, if any. It seems likely that none of those use upstream, though ^^;

So we don't care, because we *cannot* care.

> A huge factor in having good GCC support for powerpcle-linux (or
> anything else) is someone needs to regularly test it, and share test
> results with us (via gcc-testresults@). Hint hint hint :slight_smile:
>
> That way we know it is in good shape, know when we are regressing it,
> know there is interest in it.

Once I have more of a bootstrapped userland than a barely-functional
cross chroot, I'll get back to you on that :slight_smile:

Cool! Looking forward to it.

Thanks,

Segher

> > The original sysv PowerPC supplement
> > http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
> > supports LE as well, and most powerpcle ports use that. But, the
> > big-endian Linux ABI differs in quite a few places, and it of course
> > makes a lot better sense if powerpcle-linux follows that.
>
> Right, I should have clarified I was talking about Linux ABIs
> specifically.

That was the link you deleted.

> > What patches did you need? I regularly build >30 cross compilers (on
> > both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
> > in the past those worked fine as well). I also cross-built
> > powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
> > from various x86).
>
> There was just an assumption that LE == powerpc64le in libgo, spotted by
> q66 (daniel@ on the CC). I just pushed the patch to [1].

Please send GCC patches to gcc-patches@ ?

FWIW, that patch alone is not very useful, we'd need to otherwise patch libgo to recognize a new GOARCH (as right now it's likely to just use 'ppc' which is wrong).

That said, we'll get back to you with any patches we have. One I can already think of - we will need to update the dynamic linker name so that it uses ld-musl-powerpcle.so instead of powerpc (musl needs to be updated the same way by adding the subarch variable for the 'le' prefix).

> > Almost no project that used 32-bit PowerPC in LE mode has sent patches
> > to the upstreams.
>
> Right, but I have heard concerns from at least one person familiar with
> the ppc kernel about breaking existing users of this arch-endianness
> combo, if any. It seems likely that none of those use upstream, though ^^;

So we don't care, because we *cannot* care.

Well, that's the reason this thread was opened in the first place - to call out to any potential users, and synchronize with upstreams on a single way forward that all upstreams can agree on, since this effort requires changes in various parts of the stack. We don't want to hog changes locally or otherwise do any changes that would be in conflict with upstream projects, as that would mean needlessly diverging, which only means trouble later on.

> > A huge factor in having good GCC support for powerpcle-linux (or
> > anything else) is someone needs to regularly test it, and share test
> > results with us (via gcc-testresults@). Hint hint hint :slight_smile:
> >
> > That way we know it is in good shape, know when we are regressing it,
> > know there is interest in it.
>
> Once I have more of a bootstrapped userland than a barely-functional
> cross chroot, I'll get back to you on that :slight_smile:

Cool! Looking forward to it.

Thanks,

Either way, thanks for the hints so far.

Segher

Daniel (q66)

As a general comment on the glibc side of things, if this is considered
like a new port, and it probably is, the same principles that apply to new
ports apply here.

There's a general discussion at
<https://sourceware.org/glibc/wiki/NewPorts>, although much of that is
only applicable when adding new CPU architecture support. More specific
points include that new 32-bit ports should default to 64-bit time and
file offsets from the start, with no support for 32-bit time or offsets
(meaning that if you want to use this with some kind of library call
translation, the library call translation will need to deal with
corresponding type size conversions). And a new port should not be added
that uses the IBM long double format. You can use IEEE binary128 long
double, possibly with an ABI similar to that used on powerpc64le, or can
use long double = double, but should not support IBM long double, and
preferably should only have one long double format rather than using the
glibc support for building with different options resulting in functions
for different long double formats being called.

Thanks, these are great points, and the same applies for musl I think.
We always have 64-bit off_t anyway, but new ports should have 64-bit
time_t to begin with rather than defining _REDIR_TIME64.

It's a little bit complicated by the fact that powerpcle would be a
"subarch" for powerpc, and we don't yet have any like that where the
subarchs' time64 statuses differ, but it doesn't look like that should
be hard to do. The arch-specific alltypes.h.in already has access to
endianness knowledge. src/ldso/powerpc/dlsym_time64.S would also need
added preprocessor conditionals.

Exemption from this would be open to discussion if there are existing
non-upstream users of powerpcle musl that otherwise complies with ABI
policy except for time64, but I'm not aware of any that aren't
experimental.

Rich

> Hey all, a couple of us over in #talos-workstation on freenode have been
> working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> designated for this (unless you count the patchset from a decade ago [1]), so
> it's pretty much uncharted territory as far as Linux is concerned. We want to
> sync up with libc and the relevant kernel folks to establish the best path
> forward.

As a general comment on the glibc side of things, if this is considered
like a new port, and it probably is, the same principles that apply to new
ports apply here.

There's a general discussion at
<https://sourceware.org/glibc/wiki/NewPorts>, although much of that is
only applicable when adding new CPU architecture support. More specific
points include that new 32-bit ports should default to 64-bit time and
file offsets from the start, with no support for 32-bit time or offsets
(meaning that if you want to use this with some kind of library call
translation, the library call translation will need to deal with
corresponding type size conversions). And a new port should not be added
that uses the IBM long double format. You can use IEEE binary128 long
double, possibly with an ABI similar to that used on powerpc64le, or can
use long double = double, but should not support IBM long double, and
preferably should only have one long double format rather than using the
glibc support for building with different options resulting in functions
for different long double formats being called.

Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.

128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).

While we're at long double, I'd actually be interested in transitioning the existing big endian ports in Void (64-bit and 32-bit, neither has VSX baseline requirement in my case) to using 64-bit long double, abandoning the IBM format altogether (little endian will transition to 128-bit IEEE long double once it's ready on your side, as that assumes POWER8 baseline which includes VSX).

What would be the best way for me to proceed with that? I actually experimented with this, using the old glibc compat symbols from pre-ibm128 times, and I mostly had it working, except I haven't managed to find a way to switch the default symbols to 64-bit ones, which is problematic as linking everything against nldbl_nonshared is fragile and potentially quirky (breaks dlsym, function pointer equality across libraries, etc).

There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?

--
Joseph S. Myers
joseph@codesourcery.com

Daniel

Are you sure this would be a new port? Glibc already works in this
combination, as it seems to me it'd be best if it was just a variant of
the existing 32-bit PowerPC port, sharing most conventions besides
endianness with the BE port.

The supported glibc ABIs are listed at
<https://sourceware.org/glibc/wiki/ABIList>. This would be a new ABI,
which should have a new ABI-and-architecture-specific dynamic linker name
(all new ports are expected to have a unique dynamic linker name for each
ABI, to support systems using multiarch directory arrangements), new
symbol versions and avoid legacy features such as 32-bit time or offsets
or IBM long double.

128-bit IEEE long double would not work, since that relies on VSX being
present (gcc will explicitly complain if it's not). I'd be all for using

The minimum supported architecture for powerpc64le (POWER8) has VSX. My
understanding was that the suggestion was for 32-bit userspace to run
under powerpc64le kernels running on POWER8 or later, meaning that such a
32-bit LE port, and any ABI designed for such a port, can assume VSX is
available. Or does VSX not work, at the hardware level, for 32-bit
POWER8? (In which case you could pick another ABI for binary128 argument
passing and return.)

There is also one more thing while we're at this. The 64-bit big endian
Void port uses the ELFv2 ABI, even on glibc. This is not officially
supported on glibc as far as I can tell, but it does work out of box,
without any patching (things in general match little endian then, i.e.
ld64.so.2 etc, but they're big endian). Is there any chance of making
that support official?

If you want to support ELFv2 for 64-bit big endian in glibc, again that
should have a unique dynamic linker name, new symbol versions, only
binary128 long double, etc. - avoid all the legacy aspects of the existing
ELFv1 port rather than selectively saying that "ELFv1" itself is the only
legacy aspect and keeping the others (when it's the others that are
actually more problematic in glibc).

In fact, my understanding is that the ABI for passing binary128 values in
vector registers is perfectly implementable for processors with just VMX
(AltiVec) and not VSX. So if you do want to support binary128 for a new
ABI for either 32-bit LE or 32-bit or 64-bit BE, you don't need to require
VSX for that ABI, you just need to change any GCC requirement for VSX for
binary128 to allow it with VMX when building for your new ABI.

> Are you sure this would be a new port? Glibc already works in this
> combination, as it seems to me it'd be best if it was just a variant of
> the existing 32-bit PowerPC port, sharing most conventions besides
> endianness with the BE port.

The supported glibc ABIs are listed at
<https://sourceware.org/glibc/wiki/ABIList>. This would be a new ABI,
which should have a new ABI-and-architecture-specific dynamic linker name
(all new ports are expected to have a unique dynamic linker name for each
ABI, to support systems using multiarch directory arrangements), new
symbol versions and avoid legacy features such as 32-bit time or offsets
or IBM long double.

> 128-bit IEEE long double would not work, since that relies on VSX being
> present (gcc will explicitly complain if it's not). I'd be all for using

The minimum supported architecture for powerpc64le (POWER8) has VSX. My
understanding was that the suggestion was for 32-bit userspace to run
under powerpc64le kernels running on POWER8 or later, meaning that such a
32-bit LE port, and any ABI designed for such a port, can assume VSX is
available. Or does VSX not work, at the hardware level, for 32-bit
POWER8? (In which case you could pick another ABI for binary128 argument
passing and return.)

POWER8 may have VSX (well, actually POWER7 and newer has VSX and can run LE, but glibc does not support this, musl potentially does), but the overall assumption here is that the resulting binaries should eventually not be limited to being just userspace under ppc64le, but should be runnable on a native kernel as well, which should not be limited to any particular baseline other than just PowerPC.

While it should in theory be possible to do IEEE ldbl128 using a different ABI, I don't really see any benefit in this - for one, the baseline hardware doesn't support on any level, it would mean further complicating the ABI, and it would require explicit support in the compiler, which currently doesn't exist. Using 64-bit long doubles sounds like a much better way out to me.

> There is also one more thing while we're at this. The 64-bit big endian
> Void port uses the ELFv2 ABI, even on glibc. This is not officially
> supported on glibc as far as I can tell, but it does work out of box,
> without any patching (things in general match little endian then, i.e.
> ld64.so.2 etc, but they're big endian). Is there any chance of making
> that support official?

If you want to support ELFv2 for 64-bit big endian in glibc, again that
should have a unique dynamic linker name, new symbol versions, only
binary128 long double, etc. - avoid all the legacy aspects of the existing
ELFv1 port rather than selectively saying that "ELFv1" itself is the only
legacy aspect and keeping the others (when it's the others that are
actually more problematic in glibc).

Again, the BE port cannot use binary128 long double, at least not with the same ABI as on POWER8, since it runs on all 64-bit PowerPC systems starting with 970 (G5, and potentially even POWER4 if built without AltiVec). Unique dynamic linker names are complicated, since as it is, glibc uses ld64.so.1 for ELFv1, and ld64.so.2 for ELFv2. (on 32-bit PowerPC, it's ld.so.1, and uses the SVR4 ABI which is not related to either the AIX/ELFv1 nor the ELFv2 ABIs) If we were to introduce new ports, what would those use? ld64.so.3 for BE/v2? ld.so.2 for LE/32-bit? I can see the reason for a new dynamic linker name though (multi-arch setups).

However, the effective difference between the ports would be rather minimal, if any, as far as I can see. As I already said, we have a whole glibc/ELFv2/BE system, with nearly all of the existing Linux userland covered by the distro, and there haven't been any issues whatsoever.

--
Joseph S. Myers
joseph@codesourcery.com

Daniel

> The minimum supported architecture for powerpc64le (POWER8) has VSX. My
> understanding was that the suggestion was for 32-bit userspace to run
> under powerpc64le kernels running on POWER8 or later, meaning that such a
> 32-bit LE port, and any ABI designed for such a port, can assume VSX is
> available. Or does VSX not work, at the hardware level, for 32-bit
> POWER8? (In which case you could pick another ABI for binary128 argument
> passing and return.)

In fact, my understanding is that the ABI for passing binary128 values in
vector registers is perfectly implementable for processors with just VMX
(AltiVec) and not VSX. So if you do want to support binary128 for a new
ABI for either 32-bit LE or 32-bit or 64-bit BE, you don't need to require
VSX for that ABI, you just need to change any GCC requirement for VSX for
binary128 to allow it with VMX when building for your new ABI.

Which still doesn't help us even if true, since we plan to support hardware that doesn't have any kind of vector functionality in the first place (PowerPC G3/G4, and for the ELFv2 64-bit BE port, the minimum for binary packages is 970/G5 which does have AltiVec, but it is also supported to build your userland from source without this, for e.g. POWER5 machines, or e5500 SoCs)

> > There was just an assumption that LE == powerpc64le in libgo, spotted by
> > q66 (daniel@ on the CC). I just pushed the patch to [1].
>
> Please send GCC patches to gcc-patches@ ?

FWIW, that patch alone is not very useful, we'd need to otherwise patch libgo to recognize a new GOARCH (as right now it's likely to just use 'ppc' which is wrong).

Gotcha.

That said, we'll get back to you with any patches we have. One I can already think of - we will need to update the dynamic linker name so that it uses ld-musl-powerpcle.so instead of powerpc (musl needs to be updated the same way by adding the subarch variable for the 'le' prefix).

Thanks! That would be good progress.

> > > Almost no project that used 32-bit PowerPC in LE mode has sent patches
> > > to the upstreams.
> >
> > Right, but I have heard concerns from at least one person familiar with
> > the ppc kernel about breaking existing users of this arch-endianness
> > combo, if any. It seems likely that none of those use upstream, though ^^;
>
> So we don't care, because we *cannot* care.

Well, that's the reason this thread was opened in the first place - to call out to any potential users, and synchronize with upstreams on a single way forward that all upstreams can agree on, since this effort requires changes in various parts of the stack. We don't want to hog changes locally or otherwise do any changes that would be in conflict with upstream projects, as that would mean needlessly diverging, which only means trouble later on.

Much appreciated!

I don't actually foresee any huge problems -- just lots of hard work :wink:

Segher

Hi Joseph,

> Hey all, a couple of us over in #talos-workstation on freenode have been
> working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> designated for this (unless you count the patchset from a decade ago [1]), so
> it's pretty much uncharted territory as far as Linux is concerned. We want to
> sync up with libc and the relevant kernel folks to establish the best path
> forward.

As a general comment on the glibc side of things, if this is considered
like a new port, and it probably is, the same principles that apply to new
ports apply here.

There's a general discussion at
<https://sourceware.org/glibc/wiki/NewPorts>, although much of that is
only applicable when adding new CPU architecture support. More specific
points include that new 32-bit ports should default to 64-bit time and
file offsets from the start, with no support for 32-bit time or offsets
(meaning that if you want to use this with some kind of library call
translation, the library call translation will need to deal with
corresponding type size conversions).

Either that, or use the same as BE 32-bit PowerPC Linux, I'd say (it
won't make things worse, and if it is easier?) But preferably the
newer, better, thing of course :slight_smile:

And a new port should not be added
that uses the IBM long double format. You can use IEEE binary128 long
double, possibly with an ABI similar to that used on powerpc64le, or can
use long double = double, but should not support IBM long double, and
preferably should only have one long double format rather than using the
glibc support for building with different options resulting in functions
for different long double formats being called.

You cannot use IEEE QP float ("binary128") here, but more on that in a
later post.

(I so very much agree about the problems having more than one long
double format -- on the other hand, you'll just share it with BE, and
with the existing powerpcle-linux (sup)port).

Segher

Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.

That's right. Except it isn't an "official" existing port, never has
been "officially" supported.

128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).

The current IEEE QP float support requires VSX for its emulation, yes
(possibly even Power8?) As Mike reminded me today, it also requires
__int128 support, which rules out anything 32-bit currently. Without
that restriction, we could just make QP float passed in GPRs (use the
ABIs for any struct passed that way), and that'll just work out with
all ABIs, older or not.

While we're at long double, I'd actually be interested in transitioning the existing big endian ports in Void (64-bit and 32-bit, neither has VSX baseline requirement in my case) to using 64-bit long double, abandoning the IBM format altogether (little endian will transition to 128-bit IEEE long double once it's ready on your side, as that assumes POWER8 baseline which includes VSX).

I recommend new ports that cannot jump to IEEE QP float directly to use
long double == double for the time being, avoiding the extra
complications that IBM double double would bring. But you'll still have
a transition to IEEE 128 if you ever want to go there.

But if you already use double-double, I don't know if the cost changing
away from that is worth it now.

What would be the best way for me to proceed with that? I actually experimented with this, using the old glibc compat symbols from pre-ibm128 times, and I mostly had it working, except I haven't managed to find a way to switch the default symbols to 64-bit ones, which is problematic as linking everything against nldbl_nonshared is fragile and potentially quirky (breaks dlsym, function pointer equality across libraries, etc).

Yup. "Rebuild the world" works :-/ I don't have any better advice,
nothing you cannot figure out yourself.

There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?

(I don't talk for glibc).

The first thing needed is for "us" to have faith in it. That starts
with seeing test results for the testsuites!

(Something similar goes for the GCC port -- there is no official support
for BE ELFv2, but of course it does work, and if we get test results we
may keep it that way, hint hint :slight_smile: )

Segher

Hi Joseph,

>
> > Hey all, a couple of us over in #talos-workstation on freenode have been
> > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > designated for this (unless you count the patchset from a decade ago [1]), so
> > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > sync up with libc and the relevant kernel folks to establish the best path
> > forward.
>
> As a general comment on the glibc side of things, if this is considered
> like a new port, and it probably is, the same principles that apply to new
> ports apply here.
>
> There's a general discussion at
> <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is
> only applicable when adding new CPU architecture support. More specific
> points include that new 32-bit ports should default to 64-bit time and
> file offsets from the start, with no support for 32-bit time or offsets
> (meaning that if you want to use this with some kind of library call
> translation, the library call translation will need to deal with
> corresponding type size conversions).

Either that, or use the same as BE 32-bit PowerPC Linux, I'd say (it
won't make things worse, and if it is easier?) But preferably the
newer, better, thing of course :slight_smile:

> And a new port should not be added
> that uses the IBM long double format. You can use IEEE binary128 long
> double, possibly with an ABI similar to that used on powerpc64le, or can
> use long double = double, but should not support IBM long double, and
> preferably should only have one long double format rather than using the
> glibc support for building with different options resulting in functions
> for different long double formats being called.

You cannot use IEEE QP float ("binary128") here, but more on that in a
later post.

(I so very much agree about the problems having more than one long
double format -- on the other hand, you'll just share it with BE, and
with the existing powerpcle-linux (sup)port).

Well, it'd be nice to use the opportunity to ditch the IBM long doubles altogether, since these get in the way for me in various places (for instance, GCC still can't constant-fold them, which breaks on constexpr in C++, as well as makes it impossible to e.g. enable the runtime/standard library for GDC in GCC as that heavily relies on the `real` type in D, which maps directly to `long double` and druntime/phobos heavily relies on constant folding of the `real` type being functional; there are also assorted libraries and applications in the common userland that don't like the IBM format for one reason or another)

That said, that's also problematic:

1) ppc64le is going to newly use IEEE754 binary128, so we're all good there - baseline mandates VSX, so at least passing them is fine, which is the important thing (actual binary128 instructions came with ISA 3.0 but that can be detected at runtime, at least in glibc) - ibm128 is then implemented via symvers/compat, which is fine.

2) ppc64 for now uses IBM 128-bit long double, so it's problematic. I don't care about ELFv1, as it's legacy and has tons of drawbacks on its own, and there's a whole legacy ecosystem relying on it; that said, a new ELFv2 port (let's say, with ld64.so.3 dynamic linker) could easily default to another long double format, without worrying about compat - I propose this format to be binary64, as it's already used by musl on BE/64 (allows projects such as `gcompat` to work) and thus is already implemented in all toolchains we care about and can easily be flipped on, and is fully compatible with all CPUs that can run ppc64 code, even without VMX/VSX

3) ppcle would be a new port (let's say ld.so.2), so it could default to a new format; binary64 would be good

4) that leaves ppc32/BE, which would be the only outlier - while I could probably implement compat in much the same way as ppc64le does with binary128 (bump symvers for math symbols, leave older symvers for existing binaries, change defaults), this would be divergent from the existing port; glibc/IBM probably won't want to switch this, and while I would definitely like to, maintaining a divergent downstream patchset seems non-ideal. There is some basis for this - glibc did use to use binary64 on ppc32 and ppc64 BE in the past, and the compatibility symbols are still there under the old symvers, but not quite sure what to do there.

Segher

Daniel