RFC: Support x86 interrupt and exception handlers

The interrupt and exception handlers are called by x86 processors. X86
hardware puts information on stack and calls the handler. The
requirements are

1. Both interrupt and exception handlers must use the 'IRET' instruction,
instead of the 'RET' instruction, to return from the handlers.
2. All registers are callee-saved in interrupt and exception handlers.
3. The difference between interrupt and exception handlers is the
exception handler must pop 'ERROR_CODE' off the stack before the 'IRET'
instruction.

The design goals of interrupt and exception handlers for x86 processors
are:

1. No new calling convention in compiler.
2. Support both 32-bit and 64-bit modes.
3. Flexible for compilers to optimize.
4. Easy to use by programmers.

To implement interrupt and exception handlers for x86 processors, a
compiler should support:

1. void * __builtin_ia32_interrupt_data (void)

This function returns a pointer to interrupt or exception data pushed
onto the stack by processor.

The __builtin_frame_address builtin isn't suitable for interrupt and
exception handlers since it returns the stack frame address on the
callee side and compiler may generate a new stack frame for stack
alignment.

2. 'interrupt' attribute

Use this attribute to indicate that the specified void function without
arguments is an interrupt handler. The compiler generates function entry
and exit sequences suitable for use in an interrupt handler when this
attribute is present. The 'IRET' instruction, instead of the
'RET' instruction, is used to return from interrupt handlers. All
registers, except for the EFLAGS register which is restored by the
'IRET' instruction, are preserved by the compiler. The red zone
isn't supported in an interrupt handler; that is an interrupt
handler can't access stack beyond the current stack pointer.

You can use the builtin '__builtin_ia32_interrupt_data' function to access
data pushed onto the stack by processor:

void
f () __attribute__ ((interrupt))
{
  void *p = __builtin_ia32_interrupt_data ();
  ...
}

3. 'exception' attribute

Use 'exception' instead of 'interrupt' for handlers intended to be
used for 'exception' (i.e. those that must pop 'ERROR_CODE' off the
stack before the 'IRET' instruction):

void
f () __attribute__ ((exception))
{
  void *p = __builtin_ia32_interrupt_data ();
  ...
}

Any comments, suggestions?

Thanks.

Who is the intended target audience? I see a very low potential for
compiler optimisations and quite a few things that can go wrong. If I
look at the NetBSD code for interrupt handlers, there is quite a bit
complexity a generic compiler won't handle. For real interrupts for
example, this is the code to identify the interrupt source and
acknowledge the PIC. There is also the question of %fs/%gs handling in
the kernel. All in all, I don't see why this is a good idea for x86.

Joerg

-gcc-dev, not because I’m exclusionary, I’m just not sure if cross-posting to it is OK…

For what it’s worth, there’s precedent for this in clang. We already support attribute((interrupt)) on ARM and MSP430. Neither of them introduce a new calling convention at the Clang level. I think MSP430 adds one at the LLVM level, but I don’t think it matters.

For what it's worth, there's precedent for this in clang. We already support
__attribute__((interrupt)) on ARM and MSP430. Neither of them introduce a
new calling convention at the Clang level. I think MSP430 adds one at the
LLVM level, but I don't think it matters.

No. MSP430 interrupts are not using anything different compared to the
ordinary CC.
It only causes additional codegen sugar, so the interrupt handler
could be easily pinned to a particular interrupt vector #

The interrupt and exception handlers are called by x86 processors. X86
hardware puts information on stack and calls the handler. The
requirements are

1. Both interrupt and exception handlers must use the 'IRET' instruction,
instead of the 'RET' instruction, to return from the handlers.
2. All registers are callee-saved in interrupt and exception handlers.
3. The difference between interrupt and exception handlers is the
exception handler must pop 'ERROR_CODE' off the stack before the 'IRET'
instruction.

The design goals of interrupt and exception handlers for x86 processors
are:

1. No new calling convention in compiler.
2. Support both 32-bit and 64-bit modes.
3. Flexible for compilers to optimize.
4. Easy to use by programmers.

To implement interrupt and exception handlers for x86 processors, a
compiler should support:

1. void * __builtin_ia32_interrupt_data (void)

I got a feedback on the name of this builtin function. Since
it also works for 64-bit, we should avoid ia32 in its name.
We'd like to change it to

void * __builtin_interrupt_data (void)

Any comments?

H.J. Lu <hjl.tools@gmail.com> writes:

> The interrupt and exception handlers are called by x86 processors. X86
> hardware puts information on stack and calls the handler. The
> requirements are
>
> 1. Both interrupt and exception handlers must use the 'IRET' instruction,
> instead of the 'RET' instruction, to return from the handlers.
> 2. All registers are callee-saved in interrupt and exception handlers.
> 3. The difference between interrupt and exception handlers is the
> exception handler must pop 'ERROR_CODE' off the stack before the 'IRET'
> instruction.
>
> The design goals of interrupt and exception handlers for x86 processors
> are:
>
> 1. No new calling convention in compiler.
> 2. Support both 32-bit and 64-bit modes.
> 3. Flexible for compilers to optimize.
> 4. Easy to use by programmers.
>
> To implement interrupt and exception handlers for x86 processors, a
> compiler should support:
>
> 1. void * __builtin_ia32_interrupt_data (void)

I got a feedback on the name of this builtin function. Since
it also works for 64-bit, we should avoid ia32 in its name.
We'd like to change it to

void * __builtin_interrupt_data (void)

Any comments?

For what it's worth, this seems like a good plan to me. I don't know x86
but how many variations of interrupt and exception handling mechanisms
are there? If there are lots then you may want to make it clear which
subset of them you intend to support. I just added a few more variations
of interrupt handlers to MIPS and it got complicated quite quickly.

I think I remember someone asking about interrupt handler support for
x86 some time ago and the answer then was that there were too many
variants to make it useful.

Thanks,
Matthew

H.J. Lu <hjl.tools@gmail.com> writes:

> The interrupt and exception handlers are called by x86 processors. X86
> hardware puts information on stack and calls the handler. The
> requirements are
>
> 1. Both interrupt and exception handlers must use the 'IRET' instruction,
> instead of the 'RET' instruction, to return from the handlers.
> 2. All registers are callee-saved in interrupt and exception handlers.
> 3. The difference between interrupt and exception handlers is the
> exception handler must pop 'ERROR_CODE' off the stack before the 'IRET'
> instruction.
>
> The design goals of interrupt and exception handlers for x86 processors
> are:
>
> 1. No new calling convention in compiler.
> 2. Support both 32-bit and 64-bit modes.
> 3. Flexible for compilers to optimize.
> 4. Easy to use by programmers.
>
> To implement interrupt and exception handlers for x86 processors, a
> compiler should support:
>
> 1. void * __builtin_ia32_interrupt_data (void)

I got a feedback on the name of this builtin function. Since
it also works for 64-bit, we should avoid ia32 in its name.
We'd like to change it to

void * __builtin_interrupt_data (void)

Any comments?

For what it's worth, this seems like a good plan to me. I don't know x86
but how many variations of interrupt and exception handling mechanisms
are there? If there are lots then you may want to make it clear which
subset of them you intend to support. I just added a few more variations
of interrupt handlers to MIPS and it got complicated quite quickly.

I think I remember someone asking about interrupt handler support for
x86 some time ago and the answer then was that there were too many
variants to make it useful.

In my proposal, there are only 2 handlers: interrupt and exception.
__builtin_interrupt_data is provided to programmers to implement
different variants of those handlers.

H.J. Lu <hjl.tools@gmail.com> writes:

The thing that is still missing here is a use case. Interrupt handlers in existing operating systems all contain some assembly code because they need to do complicated things, like mask interrupts long enough to get a consistent state, store things in OS-specific context switch structures, and so on. I can’t imagine any OS using this feature in the compiler.

Existing compilers provide similar functionality for embedded targets, where you are not running an OS, but these often rely on some assembly code shipped with the compiler to allow the C versions to work (do you intend to contribute this code to compiler-rt for x86?). Are enough (any?) people doing bare-metal x86 embedded work for this to be a useful feature?

David

We only support writing interrupt handler in C. Hookup/install an
interrupt handler is up to the programmer.

Yes, this feature is intended for IA MCU, which runs bare-metal/embedded
OS. I don't know how compiler-rt will be used for this purpose. Do you
have an example?

See my earlier question on the same topic. For amd64, the entry code of
an interrupt handler is quite non-trivial and non-portable. The logic
for getting %gs correctly working is quite specific. Very short
interrupt handlers that can benefit from not spilling all registers is
by far the exception from the code I have seen.

Joerg

Here is the updated spec.

H.J.

As I said before, I don't see the optimisation potential and this is far
too limited for many real world users like most OS kernels. As such, I
would like to see a clear demonstration of it improving anything before
any complexity and maintainance cost associated with this hits the tree.

Joerg

This updated spec adds

   unsigned int __builtin_exception_error (void)
   unsigned long long int __builtin_exception_error (void)

This function returns the exception error code pushed onto the stack by
processor. Its return value is 64 bits in 64-bit mode and 32 bits in
32-bit mode. This function can only be used in exception handler.

It also changes the definition of

void * __builtin_interrupt_data (void)

so that it returns a pointer to the data layout pushed onto stack
by processor for both interrupt and exception handlers.

To implement interrupt and exception handlers for x86 processors, a
compiler should support:

1. void * __builtin_ia32_interrupt_data (void)

I got a feedback on the name of this builtin function. Since
it also works for 64-bit, we should avoid ia32 in its name.
We'd like to change it to

void * __builtin_interrupt_data (void)

Here is the updated spec.

This updated spec adds

    unsigned int __builtin_exception_error (void)
    unsigned long long int __builtin_exception_error (void)

This function returns the exception error code pushed onto the stack by
processor. Its return value is 64 bits in 64-bit mode and 32 bits in
32-bit mode. This function can only be used in exception handler.

Exception handlers can, in general, call regular functions which, in turn, might want to access the error code. Given that operating system kernels are always entered via an interrupt, trap, or system call, there should always be an error code available (on x86, non-error-code interrupts can just make up an error code).

It also changes the definition of

void * __builtin_interrupt_data (void)

so that it returns a pointer to the data layout pushed onto stack
by processor for both interrupt and exception handlers.

You might want to have a look at Secure Virtual Architecture (SVA). One of the things we discovered is that commodity operating systems access the most recently used interrupt data (which SVA calls an "interrupt context"). Over the years, we figured out that it's better to provide intrinsics (i.e., builtins) that implicitly access the top-most interrupt context. We also found that we could limit the operations performed on interrupt contexts so that we could safely implement signal handlers and exception recovery without letting the operating system kernel have pointers to the interrupt context which would need to be checked. In short, despite common belief, the OS does not need to do whatever it wants with interrupted program state.

I recommend you take a look at Appendix A of my dissertation (https://www.ideals.illinois.edu/handle/2142/50547). It describes the SVA-OS instructions used to abstract away the hardware details. You'll also notice that the design is pretty processor transparent (MMU notwithstanding), so designing your builtins based on SVA may make them more portable if you decide to use another processor later on. Chapter 2 describes some of the rationale behind the design, though it's for the first version of SVA (Appendix A is the final instruction set after 4 papers).

If the implementation is useful, SVA is publicly available at https://github.com/jtcriswell/SVA.

Finally, to echo Joerg's concerns, it's not clear that having exception/interrupt handlers declared as a special type is really helpful. It's not immediately obvious that you get a benefit from doing that vs. doing what most system software does (having assembly code that saves processor state and calls a C function). I think you should do some experiments to demonstrate the benefit that one can get with your method to see if it is worth adding complexity to the compiler.

Regards,

John Criswell

To implement interrupt and exception handlers for x86 processors, a
compiler should support:

1. void * __builtin_ia32_interrupt_data (void)

I got a feedback on the name of this builtin function. Since
it also works for 64-bit, we should avoid ia32 in its name.
We'd like to change it to

void * __builtin_interrupt_data (void)

Here is the updated spec.

This updated spec adds

    unsigned int __builtin_exception_error (void)
    unsigned long long int __builtin_exception_error (void)

This function returns the exception error code pushed onto the stack by
processor. Its return value is 64 bits in 64-bit mode and 32 bits in
32-bit mode. This function can only be used in exception handler.

Exception handlers can, in general, call regular functions which, in turn,
might want to access the error code. Given that operating system kernels
are always entered via an interrupt, trap, or system call, there should
always be an error code available (on x86, non-error-code interrupts can
just make up an error code).

It also changes the definition of

void * __builtin_interrupt_data (void)

so that it returns a pointer to the data layout pushed onto stack
by processor for both interrupt and exception handlers.

You might want to have a look at Secure Virtual Architecture (SVA). One of

I believe my x86 interrupt attribute is unrelated to SVA.

If the implementation is useful, SVA is publicly available at
https://github.com/jtcriswell/SVA.

Finally, to echo Joerg's concerns, it's not clear that having
exception/interrupt handlers declared as a special type is really helpful.
It's not immediately obvious that you get a benefit from doing that vs.
doing what most system software does (having assembly code that saves
processor state and calls a C function). I think you should do some
experiments to demonstrate the benefit that one can get with your method to
see if it is worth adding complexity to the compiler.

The main purpose of x86 interrupt attribute is to allow programmers
to write x86 interrupt/exception handlers in C WITHOUT assembly
stubs to avoid extra branch from assembly stubs to C functions. I
want to keep the number of new intrinsics to minimum without sacrificing
handler performance. I leave faking error code in interrupt handler to
the programmer.

Let's take just this statement. Are you sure that the call/ret pair for
such a stub has *any* significant impact on the interrupt latency?
I'm quite doubtful of that. That's completely ignoring the fact that raw
interrupt or exception latency is generally not an issue.

Joerg

To implement interrupt and exception handlers for x86 processors, a
compiler should support:

1. void * __builtin_ia32_interrupt_data (void)

I got a feedback on the name of this builtin function. Since
it also works for 64-bit, we should avoid ia32 in its name.
We'd like to change it to

void * __builtin_interrupt_data (void)

Here is the updated spec.

This updated spec adds

     unsigned int __builtin_exception_error (void)
     unsigned long long int __builtin_exception_error (void)

This function returns the exception error code pushed onto the stack by
processor. Its return value is 64 bits in 64-bit mode and 32 bits in
32-bit mode. This function can only be used in exception handler.

Exception handlers can, in general, call regular functions which, in turn,
might want to access the error code. Given that operating system kernels
are always entered via an interrupt, trap, or system call, there should
always be an error code available (on x86, non-error-code interrupts can
just make up an error code).

It also changes the definition of

void * __builtin_interrupt_data (void)

so that it returns a pointer to the data layout pushed onto stack
by processor for both interrupt and exception handlers.

You might want to have a look at Secure Virtual Architecture (SVA). One of

I believe my x86 interrupt attribute is unrelated to SVA.

Actually, I really think that it is. Part of the SVA work extended the LLVM IR to support an operating system kernel. Your design for interrupt handlers and accessing interrupted program state looks very similar to my first draft of those extensions and has the exact same limitations (plus at least one limitation that my design did not have). It's pretty clear to me that you're redesigning a subset of the SVA-OS extensions from scratch; I find that unfortunate because you are literally reinventing the wheel.

If the implementation is useful, SVA is publicly available at
https://github.com/jtcriswell/SVA.

Finally, to echo Joerg's concerns, it's not clear that having
exception/interrupt handlers declared as a special type is really helpful.
It's not immediately obvious that you get a benefit from doing that vs.
doing what most system software does (having assembly code that saves
processor state and calls a C function). I think you should do some
experiments to demonstrate the benefit that one can get with your method to
see if it is worth adding complexity to the compiler.

The main purpose of x86 interrupt attribute is to allow programmers
to write x86 interrupt/exception handlers in C WITHOUT assembly
stubs to avoid extra branch from assembly stubs to C functions. I
want to keep the number of new intrinsics to minimum without sacrificing
handler performance. I leave faking error code in interrupt handler to
the programmer.

If you want to do that, there is another approach that should work just as well and will require only localized changes to the compiler.

Interrupt handlers are typically registered to some interrupt vector number using a registration function. In FreeBSD, it's setidt(), and in Linux, I think it's set_gate(). You can write a compiler transform that looks for these registration functions, determines the function that is registered as an interrupt handler, and generate the more efficient code for that interrupt handler function as you describe.

This solution avoids language extensions to the C/C++ front-end (which requires getting approval from the Clang developers) yet should get you the performance that you want (provided that it does improve performance, for which I'm a little skeptical but open to convincing via performance measurements). You can probably write this transform as a single LLVM MachineFunctionPass that your patched version of Clang runs during code generation.

In any event, that's my two cents.

Regards,

John Criswell

We may discuss different things here. My proposal covers
writing interrupt/exception handlers in C. It doesn't deal with
interrupt/exception handler registration at all.

There are couple issues of writing x86 nterrupt/exception handlers in C:

1. Both interrupt and exception handlers must use the 'IRET' instruction,
instead of the 'RET' instruction, to return from the handlers.
2. All registers are callee-saved in interrupt and exception handlers.
3. The difference between interrupt and exception handlers is the
exception handler must pop 'ERROR_CODE' off the stack before the 'IRET'
instruction.

You need a way to tell compiler about them.