Status of stack walking in LLVM on Win64?

Hi all,

I’m developing a novel language that has historically used LLVM for code generation, behind a custom self-hosted front-end. The runtime for this language offers a garbage collector.

In the past I’ve been on 32-bit Windows where precise garbage collection is relatively straightforward (as long as FPO is disabled). Walking the stack is a simple bit of pointer chasing and some fixups. I’ve had a working GC on Win32 for years and actually posted here in the past about my experience with .gcroot and other related miscellany involved in building a real GC with LLVM.

Recently I’ve decided to begin moving the language to 64-bit native, and this has posed a few interesting challenges. The most pressing issue for me now is that stack walking is very different on Win64. It is not immediately clear to me if LLVM (3.8 or ToT) is in a position to enable reliable stack walks.

Please note I’m using purely AoT compilation here with a custom linker/binary image emitter. I can sort of hack things up in JITted code using a manual process to populate data for RtlAddFunctionTable but I’m ultimately not interested in JIT code generation.

I’ve managed to intercept data destined for the .pdata and .xdata COFF sections, but it seems to be garbage. All functions claim to begin at offset 0x0 and the UNWIND_INFO offsets are all 0-based instead of pointing into any relevant part of the image.

Given a day or so I could provide a small example that generates this useless data, but before I go to that extent - am I just doing something dumb/forgetting a step/etc. when it comes to generating stack walk metadata on Win64? Does LLVM even emit correct data yet when doing AoT compilation?

Thanks,

  • Mike

Hi all,

I'm developing a novel language that has historically used LLVM for code
generation, behind a custom self-hosted front-end. The runtime for this
language offers a garbage collector.

In the past I've been on 32-bit Windows where precise garbage collection
is relatively straightforward (as long as FPO is disabled). Walking the
stack is a simple bit of pointer chasing and some fixups. I've had a
working GC on Win32 for years and actually posted here in the past about my
experience with .gcroot and other related miscellany involved in building a
real GC with LLVM.

Recently I've decided to begin moving the language to 64-bit native, and
this has posed a few interesting challenges. The most pressing issue for me
now is that stack walking is very different on Win64. It is not immediately
clear to me if LLVM (3.8 or ToT) is in a position to enable reliable stack
walks.

Please note I'm using purely AoT compilation here with a custom
linker/binary image emitter. I can sort of hack things up in JITted code
using a manual process to populate data for RtlAddFunctionTable but I'm
ultimately not interested in JIT code generation.

I've managed to intercept data destined for the .pdata and .xdata COFF
sections, but it seems to be garbage. All functions claim to begin at
offset 0x0 and the UNWIND_INFO offsets are all 0-based instead of pointing
into any relevant part of the image.

Given a day or so I could provide a small example that generates this
useless data, but before I go to that extent - am I just doing something
dumb/forgetting a step/etc. when it comes to generating stack walk metadata
on Win64? Does LLVM even emit correct data yet when doing AoT compilation?

I can confirm that LLVM emits correct data when used in an AoT
configuration for x64, exception handling would be totally broken without
it.

For JITs it would appear that there is a patch needed for some kind of
relocations.

https://llvm.org/bugs/show_bug.cgi?id=24233

Is the patch really needed? What does it do? I'm not an expert here so asking.

For JITs it would appear that there is a patch needed for some kind of
relocations.

https://llvm.org/bugs/show_bug.cgi?id=24233

Is the patch really needed? What does it do? I'm not an expert here so
asking.

I'm not really interested in the JIT case as I said originally, so I can't
answer that question.

> I can confirm that LLVM emits correct data when used in an AoT
configuration
> for x64, exception handling would be totally broken without it.
>

Two points of clarification:

- Are you talking about Win64 or just x64 in general (i.e. *nix/MacOS)?
Again given the presence of bugs going back to 2015 (including one linked
in this thread) and other scant data from the list, I really can't tell
what the expected state of this functionality is on Win64.

- Are you referring to data generated by LLVM that is embedded in COFF
object files and then placed in the binary image by the linker? This data
is at a minimum relocated by link.exe on Windows as near as I can tell. I
do not want a dependency on link.exe. I can handle doing my own relocations
prior to emitting the final image, but I want to know if there's a turnkey
implementation of this already or if I have to roll my own here.

Thanks,

- Mike

For JITs it would appear that there is a patch needed for some kind of
relocations.

https://llvm.org/bugs/show_bug.cgi?id=24233

Is the patch really needed? What does it do? I'm not an expert here so
asking.

I'm not really interested in the JIT case as I said originally, so I can't
answer that question.

> I can confirm that LLVM emits correct data when used in an AoT
configuration
> for x64, exception handling would be totally broken without it.
>

Two points of clarification:

- Are you talking about Win64 or just x64 in general (i.e. *nix/MacOS)?

I don't think many on the Unix side of things would refer to x86_64 as
x64. Win64 is ambiguous because there have been other, non x64,
implementations of Windows (IA64, ARM64). I am referring to the x64
implementation of Win64 :slight_smile:

Again given the presence of bugs going back to 2015 (including one linked
in this thread)

The issue linked to, PR24233, has nothing to do with the static compiler
workflow.

and other scant data from the list, I really can't tell what the expected
state of this functionality is on Win64.

- Are you referring to data generated by LLVM that is embedded in COFF
object files and then placed in the binary image by the linker?

LLVM's MC layer will give you an object file with code and associated
pdata/xdata.

This data is at a minimum relocated by link.exe on Windows as near as I
can tell. I do not want a dependency on link.exe. I can handle doing my own
relocations prior to emitting the final image, but I want to know if
there's a turnkey implementation of this already or if I have to roll my
own here.

The code in RuntimeDyldCOFFX86_64 is responsible for making a pre-linked
object file loadable.