[GSoC] "Microsoft Direct3D shader bytecode backend" proposal

[Charles_Davis1]

Here's the other of my proposals for this year's Google Summer of Code.
(The first is on cfe-dev.) It was suggested to me by Dan Kegel (from the
Wine project, they really want this).

Title: Microsoft Direct3D shader bytecode backend

Abstract:

There is a distinct lack of open-source frameworks for compiling HLSL,
the shader language used by Direct3D, into bytecode that D3D can
understand. Currently, the only such framework is Ryan Gordon's
MojoShader, whose HLSL compiler component is still under heavy
development. By utilizing LLVM, it may be possible to generate
high-performance shader code from HLSL, just as Apple is known to do for
GLSL. The first step is a backend to generate D3D bytecode from LLVM IR.

Content:

1. Proposal

Currently, there are very few open-source frameworks for compiling
High-Level Shader Language (HLSL) into shader bytecode that can be
understood by Direct3D, a popular interface to 3D hardware in games.
This is because Microsoft provides its own interface in its D3DX (and
underlying D3DCompiler) DLLs. Most games therefore end up using D3DX to
compile their Direct3D shaders.

Microsoft seems to have paid no attention to optimization of the
resulting shader code, though. With LLVM, we can do better.

By using LLVM's optimizers, programs can potentially generate
high-performance shader code that works well on the platform on which
they are running. In addition, an open-source implementation would allow
the compiler to be embedded in many different applications--not the
least of which is Wine, which is in desperate need of a working shader
compiler.

The first step to this HLSL compiler is an LLVM backend for generating
D3D shader bytecode from LLVM IR. Therefore, I intend to implement such
a backend. Because implementing a full backend is a daunting task, I
intend to implement just enough to get simple examples working with
Wine's Direct3D implementation.

2. Interest

Over the past few years, I've become interested in graphics programming.
I'm also interested in learning how to design backends for LLVM.

3. Usefulness to LLVM

This should increase adoption of LLVM, particularly by games and other
3D graphical applications that use Direct3D.

4. Prior knowledge in compilers and LLVM in particular

I have been working with LLVM since the fall of 2009. I have a little
experience working with the x86 backend, but no experience writing my
own backend. I have, however, read the documents on building a new
backend, and I am very familiar with the TableGen syntax.

5. Academic, industry, etc. experience

I am a student at the Colorado School of Mines studying in Computer
Science. I am a Senior by year and credits, and I am set to graduate in
December of 2011. In addition, I have contributed to several open-source
projects, including Wine and LLVM.

6. Contact information

- E-mail: cdavis@mymail.mines.edu
- IRC nick: cdavis5x

Chip

[jholewinski]

Here’s the other of my proposals for this year’s Google Summer of Code.
(The first is on cfe-dev.) It was suggested to me by Dan Kegel (from the
Wine project, they really want this).

Title: Microsoft Direct3D shader bytecode backend

Abstract:

There is a distinct lack of open-source frameworks for compiling HLSL,
the shader language used by Direct3D, into bytecode that D3D can
understand. Currently, the only such framework is Ryan Gordon’s
MojoShader, whose HLSL compiler component is still under heavy
development. By utilizing LLVM, it may be possible to generate
high-performance shader code from HLSL, just as Apple is known to do for
GLSL. The first step is a backend to generate D3D bytecode from LLVM IR.

Content:

1. Proposal

Currently, there are very few open-source frameworks for compiling
High-Level Shader Language (HLSL) into shader bytecode that can be
understood by Direct3D, a popular interface to 3D hardware in games.
This is because Microsoft provides its own interface in its D3DX (and
underlying D3DCompiler) DLLs. Most games therefore end up using D3DX to
compile their Direct3D shaders.

Microsoft seems to have paid no attention to optimization of the
resulting shader code, though. With LLVM, we can do better.

Do you have any sources for this? In my experience, fxc is able to do some clever tricks such as replacing if-then-else conditions with predicated instructions and swizzles.

By using LLVM’s optimizers, programs can potentially generate
high-performance shader code that works well on the platform on which
they are running. In addition, an open-source implementation would allow
the compiler to be embedded in many different applications–not the
least of which is Wine, which is in desperate need of a working shader
compiler.

I’m a bit confused how Wine would take advantage of a Direct3D bytecode compiler. Would they not want to re-compile Direct3D bytecode (most often shipped with games in binary form instead of HLSL source) to something an OpenGL implementation on *nix could handle?

The first step to this HLSL compiler is an LLVM backend for generating
D3D shader bytecode from LLVM IR. Therefore, I intend to implement such
a backend. Because implementing a full backend is a daunting task, I
intend to implement just enough to get simple examples working with
Wine’s Direct3D implementation.

Could you be a bit more specific on your goals? A few quick questions that come to mind are:

1. What shader models will you aim to support?
2. What types of shared will you aim to support? e.g. vertex, pixel, geometry, hull
3. How do you propose to handle vertex buffer semantics? e.g. POSITION0, TEXCOORD0, NORMAL, etc.

Perhaps a simple example would be nice, showing a very simple LLVM IR input and the (proposed) bytecode output.

[Charles_Davis1]

    Here's the other of my proposals for this year's Google Summer of Code.
    (The first is on cfe-dev.) It was suggested to me by Dan Kegel (from the
    Wine project, they really want this).

    Title: Microsoft Direct3D shader bytecode backend

    Abstract:

    There is a distinct lack of open-source frameworks for compiling HLSL,
    the shader language used by Direct3D, into bytecode that D3D can
    understand. Currently, the only such framework is Ryan Gordon's
    MojoShader, whose HLSL compiler component is still under heavy
    development. By utilizing LLVM, it may be possible to generate
    high-performance shader code from HLSL, just as Apple is known to do for
    GLSL. The first step is a backend to generate D3D bytecode from LLVM IR.

    Content:

    1. Proposal

    Currently, there are very few open-source frameworks for compiling
    High-Level Shader Language (HLSL) into shader bytecode that can be
    understood by Direct3D, a popular interface to 3D hardware in games.
    This is because Microsoft provides its own interface in its D3DX (and
    underlying D3DCompiler) DLLs. Most games therefore end up using D3DX to
    compile their Direct3D shaders.

    Microsoft seems to have paid no attention to optimization of the
    resulting shader code, though. With LLVM, we can do better.

Do you have any sources for this? In my experience, fxc is able to do
some clever tricks such as replacing if-then-else conditions with
predicated instructions and swizzles.

I heard rumors that drivers tend not to like highly optimized bytecode
input. But that seems to count against optimization. I could take that
part out.

    By using LLVM's optimizers, programs can potentially generate
    high-performance shader code that works well on the platform on which
    they are running. In addition, an open-source implementation would allow
    the compiler to be embedded in many different applications--not the
    least of which is Wine, which is in desperate need of a working shader
    compiler.

I'm a bit confused how Wine would take advantage of a Direct3D bytecode
compiler. Would they not want to re-compile Direct3D bytecode (most
often shipped with games in binary form instead of HLSL source) to
something an OpenGL implementation on *nix could handle?

They already do that. What they want right now (among other things) is a
D3DCompiler_*.dll implementation (see http://wiki.winehq.org/HLSLCompiler ).

    The first step to this HLSL compiler is an LLVM backend for generating
    D3D shader bytecode from LLVM IR. Therefore, I intend to implement such
    a backend. Because implementing a full backend is a daunting task, I
    intend to implement just enough to get simple examples working with
    Wine's Direct3D implementation.

Could you be a bit more specific on your goals? A few quick questions
that come to mind are:

1. What shader models will you aim to support?

SM1-3. SM4 was a huge break from SM3 (so I've gathered from reading Wine
source; all the opcodes seem to be different), so I'll do that later.

2. What types of shared will you aim to support? e.g. vertex, pixel,
geometry, hull

Since I'm only doing up to SM3, vertex and pixel shaders only.

3. How do you propose to handle vertex buffer semantics? e.g.
POSITION0, TEXCOORD0, NORMAL, etc.

I can think of several ways:

- Make the frontend declare special symbols, and handle these symbols
specially in the backend
- Decorate the struct members with metadata

I'm leaning towards the latter one.

Perhaps a simple example would be nice, showing a very simple LLVM IR
input and the (proposed) bytecode output.

How about this (this is from
HLSL Introduction)?

struct a2v {
  float4 position : POSITION;
};

struct v2p {
  float4 position : POSITION;
};

void main(in a2v IN, out v2p OUT, uniform float4x4 ModelViewMatrix) {
  OUT.position = mul(IN.position, ModelViewMatrix);
}

This would generate something like this (assuming I took the metadata
route):

%struct.a2v = { <4 x float> !0 }
%struct.v2p = { <4 x float> !0 }

!0 = metadata !{ <something that indicates this is a position attribute> }

define void @main(%struct.a2v* %IN, %struct.v2p* %OUT, [4 x <4 x float>]
%ModelViewMatrix) {
  %OUT.Position = getelementptr %struct.v2p* %OUT, i32 0, i32 0
  %IN.Position = getelementptr %struct.a2v* %IN, i32 0, i32 0
  %0 = load <4 x float>* %IN.Position
  # Note the intrinsic for matrix multiplies
  %1 = call <4 x float> @llvm.d3d.mul.float4(<4 x float>%0, [4 x <4 x

] %ModelViewMatrix)

  store <4 x float> %1, <4 x float>* %OUT.Position
}

and should generate assembly to the effect of:

vs_1_1
dcl_position v0
m4x4 oPos, v0, c0
mov oD0, c4

Thanks for your input.

Chip

[James_Grosbach]

   Here's the other of my proposals for this year's Google Summer of Code.
   (The first is on cfe-dev.) It was suggested to me by Dan Kegel (from the
   Wine project, they really want this).

   Title: Microsoft Direct3D shader bytecode backend

   Abstract:

   There is a distinct lack of open-source frameworks for compiling HLSL,
   the shader language used by Direct3D, into bytecode that D3D can
   understand. Currently, the only such framework is Ryan Gordon's
   MojoShader, whose HLSL compiler component is still under heavy
   development. By utilizing LLVM, it may be possible to generate
   high-performance shader code from HLSL, just as Apple is known to do for
   GLSL. The first step is a backend to generate D3D bytecode from LLVM IR.

   Content:

   1. Proposal

   Currently, there are very few open-source frameworks for compiling
   High-Level Shader Language (HLSL) into shader bytecode that can be
   understood by Direct3D, a popular interface to 3D hardware in games.
   This is because Microsoft provides its own interface in its D3DX (and
   underlying D3DCompiler) DLLs. Most games therefore end up using D3DX to
   compile their Direct3D shaders.

   Microsoft seems to have paid no attention to optimization of the
   resulting shader code, though. With LLVM, we can do better.

Do you have any sources for this? In my experience, fxc is able to do
some clever tricks such as replacing if-then-else conditions with
predicated instructions and swizzles.

I heard rumors that drivers tend not to like highly optimized bytecode
input. But that seems to count against optimization. I could take that
part out.

   By using LLVM's optimizers, programs can potentially generate
   high-performance shader code that works well on the platform on which
   they are running. In addition, an open-source implementation would allow
   the compiler to be embedded in many different applications--not the
   least of which is Wine, which is in desperate need of a working shader
   compiler.

I'm a bit confused how Wine would take advantage of a Direct3D bytecode
compiler. Would they not want to re-compile Direct3D bytecode (most
often shipped with games in binary form instead of HLSL source) to
something an OpenGL implementation on *nix could handle?

They already do that. What they want right now (among other things) is a
D3DCompiler_*.dll implementation (see http://wiki.winehq.org/HLSLCompiler ).

   The first step to this HLSL compiler is an LLVM backend for generating
   D3D shader bytecode from LLVM IR. Therefore, I intend to implement such
   a backend. Because implementing a full backend is a daunting task, I
   intend to implement just enough to get simple examples working with
   Wine's Direct3D implementation.

Could you be a bit more specific on your goals? A few quick questions
that come to mind are:

1. What shader models will you aim to support?

SM1-3. SM4 was a huge break from SM3 (so I've gathered from reading Wine
source; all the opcodes seem to be different), so I'll do that later.

2. What types of shared will you aim to support? e.g. vertex, pixel,
geometry, hull

Since I'm only doing up to SM3, vertex and pixel shaders only.

3. How do you propose to handle vertex buffer semantics? e.g.
POSITION0, TEXCOORD0, NORMAL, etc.

I can think of several ways:

- Make the frontend declare special symbols, and handle these symbols
specially in the backend
- Decorate the struct members with metadata

I'm leaning towards the latter one.

Does this mean that the metadata would be required for correctness? If so, metadata is not a good choice, as being able to strip it and still get correct output is an LLVM constrain. If it would just result in suboptimal, but still correct, results, then it's a fine choice. See, for example, TBAA.

-Jim

[Charles_Davis1]

- Decorate the struct members with metadata

Does this mean that the metadata would be required for correctness?

'Fraid so.

If so, metadata is not a good choice, as being able to strip it and still get correct output is an LLVM constrain. If it would just result in suboptimal, but still correct, results, then it's a fine choice. See, for example, TBAA.

So much for that idea. Looks like unless anyone has a better idea, I'll
have to go with GLSL-style global inputs and outputs.

Chip

[Dan_Kegel]

Wine already does re-compile Direct3D bytecode to glsl (or arb
shaders), and it works.
The problem is, some games actually ship ascii hlsl and expect to
be able to compile it at startup time (or even later).
See e.g. WineHQ Bugzilla – Bug List
and WineHQ Bugzilla – Bug List
- Dan