endian independence

Hi,

I'd like to use LLVM to compile and optimise code when I don't know
whether the target CPU is big- or little-endian. This would allow me
to create a single optimised LLVM bitcode binary of an application,
and then run it through a JIT compiler on systems of differening
endianness.

I realise that in general the LLVM IR depends on various
characteristics of the target; I'd just like to be able to remove this
dependency for the specific case of unknown target endianness.

Here's a sketch of how it would work:

1. Extend TargetData::isBigEndian() and LLVM bitcode's "target data
layout string" so that endianness is represented as either big, little
or unknown. (I see there's already support for something like this in
Module::getEndianness().)

2. For optimisations (like parts of SRA) that depend on knowing the
target endianness, restrict or disable them as necessary if the target
endianness is unknown. I think this will only affect a small handful
of optimisations.

3. In llvm-gcc, if the LLVM backend reports unknown endianness, make
sure that the conversion from GCC trees to LLVM IR doesn't depend on
endianness. This seems to be fairly straightforward, *except* for
access to bitfields, which is a bit convoluted.

4. In llvm-gcc, if the LLVM backend reports unknown endianness, make
sure that GCC's optimisations on trees don't depend on endianness.

5. Have the linker refuse to link a big-endian module with a
little-endian one, but allow linking a module of unknown endianness
with a module of any endianness at all. (I think this might work
already.)

I'm already working on this myself. Would you be interested in having
this work contributed back to LLVM?

Thanks,
Jay.

I would find this functionality useful if it made it back into trunk.

scott

Hi,

I'd like to use LLVM to compile and optimise code when I don't know
whether the target CPU is big- or little-endian. This would allow me
to create a single optimised LLVM bitcode binary of an application,
and then run it through a JIT compiler on systems of differening
endianness.

Ok.

I realise that in general the LLVM IR depends on various
characteristics of the target; I'd just like to be able to remove this
dependency for the specific case of unknown target endianness.

Sure. In practice, it should be possible to produce target-independent LLVM IR if you have a target-independent input language. The trick is making it so that the optimizers preserve this property. Endianness is only one piece of this puzzle.

3. In llvm-gcc, if the LLVM backend reports unknown endianness, make
sure that the conversion from GCC trees to LLVM IR doesn't depend on
endianness. This seems to be fairly straightforward, *except* for
access to bitfields, which is a bit convoluted.

This will never work for llvm-gcc. To much target-specific stuff is already folded before the llvm backend is even involved.

I'm already working on this myself. Would you be interested in having
this work contributed back to LLVM?

If this were to better support target independent languages, it would be very useful. If you're just trying to *reduce* the endianness assumptions that leak through, I don't think it's a good approach. There is just no way to solve this problem with C. By the time the preprocessor has run, your C code has already had #ifdef __LITTLE_ENDIAN__ etc evaluated, for example.

How do you propose to handle things like:

struct foo {
#ifdef __LITTLE_ENDIAN__
   int x, y;
#else
   int y, x;
#endif
};

-Chris

I'm already working on this myself. Would you be interested in having
this work contributed back to LLVM?

If this were to better support target independent languages, it would
be very useful. If you're just trying to *reduce* the endianness
assumptions that leak through, I don't think it's a good approach.
There is just no way to solve this problem with C.

Yes, I can see that the llvm part of this is more straightforward and
less controversial than the llvm-gcc part. Maybe I should submit the
llvm part (since it applies to all source languages) and keep the
llvm-gcc part as a local hack.

How do you propose to handle things like:

struct foo {
#ifdef __LITTLE_ENDIAN__
  int x, y;
#else
  int y, x;
#endif
};

I can't make all C programs work regardless of target endianness. This
one will only work on little-endian:

  int x = 1;
  assert(*(char *)&x == 1);

You've just highlighted another restriction that I'll have to impose:
you shouldn't expect to be able to detect target endianness at compile
time.

All I want is that, if you write your source code so that it doesn't
make assumptions about endianness, then the compiler and its
optimisations won't introduce any new assumptions about endianness.

Thanks,
Jay.

Yes, I can see that the llvm part of this is more straightforward and
less controversial than the llvm-gcc part. Maybe I should submit the
llvm part (since it applies to all source languages) and keep the
llvm-gcc part as a local hack.

Here (attached) is a patch for the llvm parts of this. It doesn't
introduce any new failures in "make check". Some points:

1. I don't understand why Module has its own DataLayout string, and
its own code to parse it (in getEndianness and getPointerSize).
Couldn't it have an instance of TargetData instead? Or is it just that
Module wants a concept of unknown endianness/pointer size, which
TargetData didn't support?

2. I'm assuming that for most code in lib/Target/, lib/CodeGen and
lib/ExecutionEngine you have a real CPU target with known endianness,
so I haven't changed any of that code to check for unknown endianness.

3. The Endianness enumeration should probably live somewhere other
than Module. But I don't know where.

Any comments?

Thanks,
Jay.

endianness (12 KB)

Ok, if you want to address this in LLVM, the place to start is to make the optimizers completely targetdata-independent. The best way to do this (IMO) is to change passes to use "getAnalysisToUpdate" instead of "getAnalysis/AddRequired" on TargetData. Then, change opt to only add targetdata to the passmgr if a target data string exists in the module.

This would make the optimizers transparently take advantage of TD when available, but gracefully handle the cases when it isn't.

-Chris

Ok, if you want to address this in LLVM, the place to start is to make
the optimizers completely targetdata-independent. The best way to do
this (IMO) is to change passes to use "getAnalysisToUpdate" instead of
"getAnalysis/AddRequired" on TargetData. Then, change opt to only add
targetdata to the passmgr if a target data string exists in the module.

This would make the optimizers transparently take advantage of TD when
available, but gracefully handle the cases when it isn't.

This sounds a bit all-or-nothing. In my case, I know everything about
target data *except* for the endianness, and I don't think I'd want to
disable any significant optimisations that depend on other aspects of
the target data. But maybe I'm an unusual case!

Thanks,
Jay.

Hi,

Chris Lattner wrote: