[RFC] Requantization in quant dialect

MLIR
1

Hi All,

We would like to propose adding explicit support for requantization in the quant dialect.

Motivation:

Currently, MLIR provides quant.qcast and quant.dcast for casting between storage and expressed types, and quant.uniform for defining quantized types. However, there is no built-in operation to perform requantization i.e., converting from one quantized representation to another with different scale and zero-point parameters (e.g., from i32 to i8 with new quantization parameters).

This is a common requirement in many quantized inference pipelines, especially when:

  1. Intermediate tensors are accumulated in higher precision (e.g., i32) and need to be requantized to lower precision(e.g., i8) for further processing or storage.
  2. Models require layer-wise or channel-wise quantization adjustments.
  3. Backends need to lower quantized ops efficiently with hardware-specific constraints.

Proposal

This section proposes the introduction of a new operation, named quant.reqcast, with the following characteristics:

Inputs: a tensor of quantized or integer type, scale and zero-point for input and output tensors
Attributes: optional rounding mode, clamp behavior, and overflow handling.
Semantics: apply scale and zero-point transformation from the input to the target quantized type.

Syntax

operation ::= `quant.reqcast` $input attr-dict `:` type($input) `to` type($result)

Operands

  • %input: A tensor of integer type (e.g., i32, i16, i8) that is already quantized using some scale and zero point.

Attributes

  • input_scale / input_zero_point: Describe the quantization parameters of the input tensor.
  • output_scale / output_zero_point: Describe the target quantization parameters.
  • rounding_mode (optional): e.g., "TONEAREST", "UPWARD", "TOWARD_ZERO".
  • clamp (optional): Whether to clamp the output to the valid range of the output type (e.g., [-128, 127] for i8).

Result

  • A tensor of the same shape but with a different quantized type (e.g., i8 instead of i32), using the new quantization parameters.

Example

%requantized = "quant.reqcast"(%input_tensor, %output_tensor, %rq_input_scale, %rq_input_zp, %rq_output_scale, %rq_output_zp) : 
      (tensor<1x16x8x64xi32>, tensor<1x16x8x64xi8>, 0.05, 120, 0.06, 128) -> tensor<1x16x8x64xi8>

This operation can be lowered down as shown in the example below:

!qalias = !quant.uniform<i32:f32, %rq_input_scale:%rq_input_zp>
!qalias1 = !quant.uniform<i8:f32, %rq_output_scale:%rq_output_zp>
func.func @requantization(%arg0: tensor<1x16x8x64xi32>) -> tensor<1x16x8x64xi8> {
   %0 = quant.scast %arg0 : tensor<1x16x8x64xi32> to tensor<1x16x8x64x!qalias>
   %1 = quant.dcast %0 : tensor<1x16x8x64x!qalias> to tensor<1x16x8x64xf32>
   %2 = quant.qcast %1 : tensor<1x16x8x64xf32> to tensor<1x16x8x64x!qalias1>
   %3 = quant.scast %2 : tensor<1x16x8x64x!qalias1> to tensor<1x16x8x64xi8>
   return %3 : tensor<1x16x8x64xi8> 
}

Benefits

  1. Enables clean and explicit modeling of quantization pipelines.
  2. Avoids overloading qcast/dcast with semantics they weren’t designed for.
  3. Facilitates backend lowering and optimization passes that rely on quantization boundaries.

We would love to hear thoughts from the community—especially around naming, semantics, and whether this fits best in the quant dialect or should be modeled elsewhere. If this looks okay, I will raise a follow up PR for this.

Thanks!

LLVM Weekly - #596, June 2nd 2025