SPIRV binary to MLIR hello world example

MLIR
1

Hello everyone,

I am trying to convert OpenGL shader (from the SPIRV tutorials) to MLIR through SPIRV. The flow is as follows:
.frag to .spv to .mlir
To achieve this I used:
gslangValidator binary to convert .frag to .spv
Then
mlir-translate to deserialize .spv to .mlir.

I kept getting errors about unhandled decorations in the deserializer. I edited the mlir spirv deserializer to handle (do nothing) Location and Perspective decorations but then I faced a Core Dump error related to processing other instructions.

I tried a different example (.frag file) but I got an OpCode 25 not handled error.

I donā€™t have strong experience in LLVM/MLIR/SPIRV but I followed the documentation and tutorials I could find about SPIRV Dialect, nonetheless, I am still stuck trying to get a hello world example to work.

Can someone point me to an end-to-end example of converting .frag to .spv and then to MLIR? Or whether this is something doable in the first place. I appreciate any material that can help.

Thanks

2

Hi @hazem, the SPIR-V dialect at the moment only supports common instructions for the compute pipeline for OpenGL/Vulkan. Fragment shader belongs to the graphics pipeline. While itā€™s our aim to leave the door open for future graphics instructions and use cases, at the moment they are not implemented yet. So itā€™s unlikely to work out of the box. But you are free to add the instructions needed and the procedure should be reasonably straightforward, explained in 'spv' Dialect - MLIR. Iā€™m curious about your use case there; could you tell more what you are trying to achieve if possible? :slight_smile:

3

Hello @antiagainst, thanks for the prompt reply and clarification. I am trying to learn to what degree is SPIR-V dialect mature in terms of supporting Graphics related operations and data types, and whether there is a room for contribution (research or development) there.

4

Hey @hazem, thanks for the interest! Yes, there are certainly plenty of room where we can improve and if youā€™d like to help, that would be awesome! As said, the work thus far has been focusing on bring up the compute pipeline. We have a fairly good coverage of types and ops and are able to represent and run non-trivial compute shaders via Vulkan applications. Graphics pipeline share lots of common types and math ops with the compute pipeline, so whatā€™s there can be reused directly. As for graphics pipeline specific types and ops, they are almost nonexistent, except ImageType. But Iā€™d assume defining them would be a straightforward task; one can find existing code and use as examples. Also weā€™ve a few helper scripts that should simplify the procedure. Iā€™d recommend you to checkout the dialect doc for details. Let me know what you think. If youā€™d like to know more details, Iā€™d happy to expand on things and find examples.

5

Thank you! I am currently playing with some Vulkan examples, trying to identify what is there and what is missing, and at the same time going through the tutorials and documentation to learn more. Still I noticed that there isnā€™t a concrete full Vulkan compute example. That would be a great addition to the documentation and learning material especially as a start point. I will keep you updated how things go if I decided to contribute and work on some missing data types and operations related to the graphics pipeline in particular.

6

Cool, thanks! Looking forward to know your findings! :slight_smile:

7

Hello @antiagainst, I am currently trying to get a basic compute shader to work as I described earlier in my messages but I get an error about signdedness. I noticed that in the status page unsigned integer is still not supported. And even though it is listed under Vulkan compute API, I assumed that it is not implemented in the SPIRV dialect for any API yet.

However, the documentation page mentions that signed and unsigned integers are supported in the current SPIRV dialect. My goal is to determine whether uint is supported or not, then based on this decide whether to work on adding it to the current SPIR-V dialect or not.

This serves the purpose of learning how to add new data types, and actually adding a basic type that is used frequently especially in indexing for graphics APIs.

Thanks,
Hazem

8

Hey @hazem, sorry for the confusion. The tracking issue you mentioned was used back when MLIR was still a separate repo. MLIR is now in LLVM monorepo and that status page is outdated. (Iā€™ve updated the issue to make it clear.) The SPIR-V dialect supports integer signedness since [mlir][spirv] Support integer signedness Ā· llvm/llvm-project@9600b55 Ā· GitHub. Iā€™m curious what kind of error you see regarding signedness. Could you provide more details?

If you are particularly interested in how types are handled and would like to help on some missing ones, I think they are OpTypeMatrix, OpTypeSampler, OpTypeSampledImage. OpTypeMatrix is a bit complicated compared to the others given you can have decorations. The other two should be simpler but yes they are mainly used for graphics.

9

Thank you for the clarification. The error I get is:

llvm-project/mlir/lib/IR/Attributes.cpp:328: int64_t mlir::IntegerAttr::getInt() const: Assertion `(getImpl()->getType().isIndex() || getImpl()->getType().isSignlessInteger()) && ā€œmust be signless integerā€ā€™ failed.

Specifically this error happens in this line:
outBuffer.data[storePos].v = vec4(sin(alpha) * radius, cos(alpha) * radius, 0.0, 1.0);

it was a part of a shader that I found online and was experimenting with it. Just basic vectors, addition and multiplication operations.

The problem seems to be related to spv.constant generation when accessing struct elements. It complains about the signdedness of the index.

I am not sure if this is related but I also get an error when trying to convert the mlir output to llvmir about:
error: ā€˜spv.moduleā€™ op unsupported module-level operation
spv.module Logical GLSL450 requires #spv.vce<v1.0, [Shader], > {

But I believe this is a separate issue in the translation to LLVMIR and not directly related to the first issue.

10

As a followup, I started working on the OpTypeSampler, but I get an error related to the ImageType (unhandled opcode 25), also the decorator NonReadable was missing which is used with the ImageType as well. Do you have a basic example shader that uses ImageType and works?

Thanks

11

Interesting! At a first glance the used operators should all be supported. Curious, did you compile this with glslang and try to deserialize the SPIR-V into MLIR? If so, could you provide the full GLSL here so I can take a look?

In general the serialization/deserialization at the moment works more smoothly with the patterns generated from ML workloads. So donā€™t be too surprised to see some sharp corners. We certainly need to harden it for patterns directly generated from other compilers. And thank you for helping on this front!

Yep at the moment there is no conversion to go from SPIR-V to LLVM. Thatā€™s actually the GSOC project for this summer! :slight_smile:

Awesome!

Oops, sorry for the sharp corner again. We havenā€™t actually wired up the serialization/deserialization for image types. So you are seeing unhandled opcode 25 (which is for OpTypeImage). It would be nice if you can help to wire things up there!

Here is the list of opcodes for types:

So you can see OpTypeImage is missing there. We have a handy script to update opcodes (and other instructions, etc.). I believe if you do ./utils/spirv/define_opcodes.sh OpTypeImage in the source root directory it will define the opcode there. Then the real work would be to teach (de)serialization to understand image types. You can probably use the logic for other types as examples of how to do that. For example, for serialization:

For deserialization:

And then there should be round-trip tests to cover them, for example for struct types we have: llvm-project/struct.mlir at 29b955f97cc6a4b96aba0ccdc033884c51ef466f Ā· llvm/llvm-project Ā· GitHub

Would you mind to wire up the (de)serialization support for image types? Thanks!

12

You can find examples here llvm-project/types.mlir at 29b955f97cc6a4b96aba0ccdc033884c51ef466f Ā· llvm/llvm-project Ā· GitHub for how image types are represented in the IR.

13

Thank you for the detailed reply and numerous resources. I will look into them and keep you updated if I make any progress. I attached the shader that I used and generated the errors I mentioned before:

#version 430

layout(location = 1) uniform writeonly image1D myImage;

layout(location = 0) uniform float radius;
struct AttribData
{
	vec4 v;
	vec4 c;
};
layout(std430, binding = 0) buffer destBuffer
{
	AttribData data[];
} outBuffer;

layout (local_size_x = 64, local_size_y = 1, local_size_z = 1) in;
void main()
{
	int storePos = 1;
	int gSize = 4;
	float alpha = 2.0 * 3.14159265359 * (float(storePos) / float(gSize));

	//outBuffer.data[storePos].v = vec4(sin(alpha) * radius, cos(alpha) * radius, 0.0, 1.0);
	//outBuffer.data[storePos].c = vec4(float(storePos) / float(gSize), 0.0, 1.0, 1.0);
}
14

Hello @antiagainst, I have another question regarding sampler data type (OpTypeSampler), after adding the opcode and implementing the deserialization/serialization of the ImageType, I get a new error related to SampledImageType (unhandled opcode 27) so I added the opcode and deserialization/serialization steps for that as well and the shader I was using with a Sampler type declared was able to be deserialized.

However it puzzles me as to why the sampler type is being converted into a SampledImage type; from the SPIRV documentation it says OpSampledImage needs to consume both a Sampler and Image type in order to create a SampledImage type, but there is no implementation for the Sampler type yet. Am I doing something wrong or do we not need the Sampler type?

15

Hey @hazem, sorry for the late reply!

Have you dumped the raw SPIR-V using spirv-dis to see whatā€™s inside? I guess itā€™s the frontend that generates OpTypeSampledImage from the source code? The deserialization in the SPIR-V dialect just takes whatā€™s in the raw SPIR-V and maps the opcode to some op.

16

This is correct. Thanks!

17

Another question, when defining any SPIR-V data type and adding it to the Dialect there is a syntax defined here ('spv' Dialect - MLIR)
for each data type, for example arrays have the following syntax:

array-type ::= `!spv.array` `<` integer-literal `x` element-type
               (`,` `stride` `=` integer-literal)? `>`

I know this is required for the printing/parsing methods, but is there a reference/guide for this scheme?, or should I follow these examples as close as possible while adding/removing items as needed depending on the data type I am adding?

The same question for defining new Operations as well (e.g., OpVectorShuffle).

18

For type parsing I think the documentation might be a bit sparse. Following examples of other type parsing/print methods would be the way to proceed.

For new op-definition there are scripts that pull in the spec from the SPIR-V spec pages and generates the ODS specification of the op. See this for more details : 'spv' Dialect - MLIR

For most ops the serilaization/deserialization into SPIR-V binary is also auto-generated. See here and here for more details.

19

Thank you!

I will follow the examples then for parsing and printing as much as possible. I got one more question, I am currently also working on the matrix data type. According to the specification, a matrix defined as a member of a struct must have: MatrixStride, and RowMajor/ColMajor decorations beside the Offset ofcourse.

I was able to add basic support for Matrix (I can deserialize a shader with a matrix to MLIR textual IR), but adding support for a matrix within a Structure requires some understanding of the current status of Struct Datatype, more specifically how it handles Decorations.

My question is: if I want to handle MatrixStride decoration, should it be added as a Layoutinfo similar to the offset? or do I need to make some changes in the processStructType method to parse the matrix decorations within a Struct.

[Edit] I can see your comment under StructType class about LayoutInfo needed to be changed to accommodate other member types, I think this is where I should begin the changes to support MatrixStride and ArrayStride and other key-value based decorations, is that correct?

20

Thatā€™s awesome! Thanks @hazem!

It would be nice if you can send out a patch if possible. No need to have everything fully implemented before sending out patches; actually smaller incremental implementations is preferrable and it allows faster review cycles.

Yes exactly. The LayoutInfo is just a ā€œplaceholderā€ for now and it only accounts for offsets. It should be turned into a struct to consider other decorations. This part can be a bit tricky given it requires to modify the type storage class. There are some docs here and you can look at the code for StructTypeStorage in SPIRVTypes.cpp to see how it is implemented. We will need to redefine the IR for !spv.struct a bit too to accommodate the change. We can discuss further when youā€™d like to dive into details. Let us know how you think.