> ## Documentation Index
> Fetch the complete documentation index at: https://lmsysorg-cheng-refactor-decoder-stage-api.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Fused Kernels

> The fused CUDA/Triton kernels SGLang Diffusion ships, what each one replaces, and which are on by default.

Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise and selection chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues, and MoE routing — each of which can require several kernel launches and HBM round trips in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion).

This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is `--quality`, described below.

## Numerical contracts and quality tiers

Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. The `quality` switch distinguishes unconditional bit-exact replacements from non-bit-exact eager-chain fusions:

**Bit-exact — mounted unconditionally.** The kernel reproduces every rounding boundary of the eager chain, so `torch.equal` holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch's `vectorized_layer_norm_kernel` down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer's CuTe-DSL `RMSNormKernel` fragment order and `shfl.bfly` fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch.

**Not bit-exact — request-gated.** These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for `quality="extra-high"` and `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. The default `quality="lossless"` runs the unmodified reference chain.

**Model/checkpoint-native.** Generic close-contract kernels, sparse operators, and FP8/NVFP4 producers can be part of a model implementation or a separately selected deployment path. They are documented in the inventory, but `quality` does not select or undo those choices.

**Selection-equivalent routing — enabled unconditionally.** LingBot Video's fused group-limited top-k returns the same selected expert-id set as its guarded CUDA `torch.topk(..., sorted=False)` reference chain. The order of those ids is not part of either path's contract. Because the selected experts are unchanged, this path does not depend on the request `quality` tier.

<Note>
  A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR, which is what motivated the bit-exact rewrite of that path.
</Note>

The quality levels are cumulative:

| `quality` | Included optimization set |
| - | - |
| `lossless` | The selected deployment's reference path plus every unconditional bit-exact replacement |
| `extra-high` | Everything in `lossless`, plus request-gated DiT and VAE kernel fusions; this level does not itself enable sparse, caching, or another approximate path |
| `high` | Everything in `extra-high`, plus any model-owned high-only optimization, such as an audited Cache-DiT policy or lower-precision VAE decode |

If a model has no eligible request-gated fusion, `extra-high` can execute the same path as `lossless`. Likewise, `high` adds only the model-specific high-only paths that the active pipeline implements.

<Note>
  `quality` is not a master precision switch. A quantized checkpoint, an explicitly selected approximate attention backend, or an independently enabled cache remains active at every quality tier.
</Note>

## Enabling the request-gated set

```bash theme={null}
sglang generate --model-path MODEL_PATH --prompt "..." --quality extra-high
```

The server default stays `lossless`; the OpenAI-compatible endpoints carry it per request. Images:

```bash theme={null}
curl -X POST http://${HOST}:${PORT}/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'
```

Video, same field:

```bash theme={null}
curl -X POST http://${HOST}:${PORT}/v1/videos \
  -H 'Content-Type: application/json' \
  -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'
```

<Warning>
  The `quality` field in a **video response** body is unrelated. It is Sora-compatible response metadata and is always reported as `"standard"`; it does not reflect the sampling quality the request ran with.
</Warning>

`quality` participates in the dynamic-batch signature, so mixed-quality traffic is batched separately and the transition happens safely at a batch boundary. Mounting is all-or-nothing: if any marked site on a transformer fails its static guards, no site on that transformer is fused.

<Warning>
  Do not combine request-gated DiT fusions with `--enable-breakable-cuda-graph`.
  BCG warmup captures the lossless module branches before an `extra-high` or
  `high` request mounts its DiT fusions, so replay would bypass the requested
  kernels. SGLang rejects this combination for models with eligible DiT quality
  sites. Models whose request-gated path changes only VAE decode remain allowed
  because BCG captures the DiT only.
</Warning>

These fusion families mount under both `quality="extra-high"` and
`quality="high"`:

| Fusion | What it folds |
| - | - |
| Linear + tanh-GELU | Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip |
| Wan NVFP4 linear + GELU | Bias-add and GELU fused into Wan's NVFP4 FFN projection output |
| Qwen-Image added-QKV | Added Q/K/V projections fused into joint-buffer production |
| LayerNorm + modulate | `layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate |
| LTX-2 RMSNorm + modulate | `rms_norm(x) * (1 + scale) + shift` in one launch |
| Gate RMSNorm (BF16-native) | `RMSNorm + tanh + mul + add` in one pass |
| HunyuanVideo strided QK RMSNorm | Per-head QK RMSNorm over the packed QKV layout |
| LingBot Video fused RMSNorm | Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels |
| LingBot Video per-token gated residual + RMSNorm modulate | Folds `residual + gate * update` (per-token `[B, S, 1]` gate) and the `rmsnorm(x) * (1 + scale) + shift` adaLN chain (strided `[B, S, 6D]` chunk views) into single kernels |
| SANA-Video BF16-input linear attention | Keeps the first linear-attention GEMM's inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32 |
| FLUX-family VAE fast paths | Channels-last decode, GroupNorm(+SiLU), upsample, and attention replacements for FLUX.2 and AutoencoderKL-based FLUX.1, Z-Image, and SD3 pipelines |
| Wan VAE RMSNorm + SiLU | Replaces the channel-first RMSNorm/SiLU chain while keeping the decode in `channels_last_3d` |

## Kernel inventory

45 operators are registered in the kernel registry across 51 implementations (some operators carry several backends). Backends are named by provenance, not device: `KDA` identifies Kernel Design Agents implementations, `JIT` compiles under nvcc *and* hipcc, `TRITON` identifies Triton sources, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, and `AOT` comes from the `sgl_kernel` wheel. Per-operator capability metadata determines which devices can load each implementation.

### Normalization

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `rmsnorm_scale_shift` | Triton | bit-exact | RMSNorm + `* (1 + scale) + shift` (4 kernels) |
| `scale_residual_norm_scale_shift` | KDA / Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) or backend-specific close contract | the above plus the preceding `residual + gate * update` |
| `scale_residual_norm_scale_shift_nvfp4` | JIT CUDA | matches the selected NVFP4 producer contract | Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization |
| `layernorm_modulate` | Triton | bit-exact | affine-free LayerNorm + adaLN modulate |
| `try_fused_fp32_layernorm_bf16` | KDA / Triton | bit-exact against the verified native dispatch | LingBot World causal FP32 LayerNorm on Hopper with FP32 affine or adaLN post-ops, including BF16 input/output conversions; other devices retain native LayerNorm |
| `qk_head_layernorm` | Triton | bit-exact | per-head LayerNorm on q/k |
| `qk_rmsnorm_native` | Triton | bit-exact | Z-Image per-head QK RMSNorm |
| `norm_scale_shift` | KDA / CuTe-DSL / FlyDSL | backend-specific close contract | LN-or-RMS + scale/shift, many broadcast modes |
| `rmsnorm_scale`, `rmsnorm_tanh_residual` | Triton | bf16-native statistics | `RMSNorm(x) * scale`, `x + tanh(gate) * RMSNorm(y)` |
| `apply_group_norm_silu` | Triton | close | `GroupNorm + SiLU`, NCHW-contiguous |
| `group_norm_silu_4d`, `group_norm_silu_rows` | Triton | close | channels-last GroupNorm(+SiLU); what lets a VAE decoder run channels\_last end to end with no `nchwToNhwc` transposes |
| `wan_rmsnorm_silu` | Triton | close | Wan VAE `channels_last_3d` RMSNorm + SiLU |
| `wan_norm_silu_post` | JIT CUDA | bit-exact, first-sight verified | Wan VAE FP32 divide, scale, affine, and SiLU after the unchanged native L2 reduction; dense NCDHW and `channels_last_3d` |

### adaLN modulation and gating

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `modulate_scale_shift` | JIT CUDA | bit-exact | `x * (1 + scale) + shift` |
| `residual_gate_add` | KDA (JIT CUDA) | bit-exact | `residual + gate * update` |
| `try_fused_scaled_residual_bf16` | Triton | bit-exact against the verified native dispatch | LingBot World causal BF16 residual with an FP32 per-frame gate; skips the unused normalized output before camera conditioning |
| `timestep_embedding` | JIT CUDA | close | sinusoidal timestep embedding |
| `temb_table_slices` | Triton | bit-exact | see note below |
| `ltx2_ada_values` | Triton | bit-exact | LTX-2 nine-way adaLN value split, slices come out contiguous |

LingBot World camera conditioning also uses `modulate_scale_shift` with one
affine row per token. The causal normalization and residual paths verify each
new signature outside CUDA graph capture and fall back to their native chains
on mismatch or unsupported layouts. These paths preserve `lossless` arithmetic;
kernel replay support does not enable BCG for a model configuration that disables it.

<Tip>
  `temb_table_slices` is worth knowing about. The eager `(scale_shift_table + temb.float()).chunk(6, dim=2)` materializes roughly 8 GB of fp32 at 704p/121f *and* hands six **strided** slices downstream, whose `.contiguous()` calls then copy each one again. The fused kernel produces the six slices in one pass, each naturally contiguous, so the downstream copies become no-ops.
</Tip>

### RoPE and QK-norm

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True` | separate QK-norm kernel + RoPE |
| `flux2_qkv_epilogue` | KDA (JIT CUDA) | bit-exact against its selected BF16 reference chain | FLUX.2 QK RMSNorm + RoPE + joint text/image QKV packing |
| `flux2_strided_qknorm_rope` | Triton | bit-exact against native tiled RMSNorm followed by FlashInfer interleaved RoPE; verified per signature | FLUX.2 Klein single-stream packed Q/K copies + QK RMSNorm + RoPE on Hopper, BF16, 24 heads of dimension 128 |
| `qwen_qkv_epilogue` | JIT CUDA | bit-exact against its selected BF16 reference chain | Qwen-Image QK RMSNorm + RoPE + joint QKV writes on SM100+ |
| `rope_rotate_half` | Triton | bit-exact | `chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection |
| `interleaved_rope_fp64` | JIT CUDA | bit-exact | paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels |
| `helios_qk_rope` | JIT CUDA | bit-exact | paired in-place Helios Q/K RoPE with transposed frequency layout |
| `ltx2_qknorm_split_rope` | KDA (JIT CUDA) | close (validated on B200) | LTX-2 QK-norm + split RoPE |
| `ltx25_decoder_rope` | JIT CUDA | bit-exact | paired LTX-2.5 decoder 3D RoPE from cached compact axis tables |
| `hunyuan_qkv_rope_pack` | Triton | bit-exact | QKV pack and RoPE in one pass |

The Klein strided path preserves the BF16 rounding between RMSNorm and RoPE,
including signed zeros. It reads packed Q/K views without changing the adjacent
V or MLP values. Unsupported layouts, devices, or RMSNorm dispatches use the
original helper. A new signature is verified outside CUDA Graph capture; an
unverified capture falls back. `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0` also disables
this path. This kernel does not enable model-level BCG support for FLUX.2.

### Activation

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `silu_mul` | Triton | bit-exact | `F.silu(a) * b` for split-projection SwiGLU, where the concatenated `silu_and_mul` kernels do not apply without an extra full-width `cat` |
| `bias_silu`, `bias_glu` | Triton | bit-exact | Sana GLUMB conv bias + SiLU / GLU post-processing |
| `linear_gelu_tanh` | AOT (cublasLt) | not bit-exact, request-gated | bias-add and tanh-GELU folded into the GEMM epilogue |

### Attention

| Operator | Backend | Notes |
| - | - | - |
| `sparse_linear_attn_fwd` | Triton | block-map, compression, and forward for sparse linear attention |
| `bigdn` | Triton | Sana-WM bidirectional gated delta-net; the chunkwise form splits phase A along the KV and Z streams so two blocks stay resident per SM, and stores `(I - P)` so phase B's MMA folds the identity-add in |

### MoE routing

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `group_limited_topk` | Triton | selected expert-id set matches the guarded CUDA reference; output order is unspecified | LingBot Video's per-group top-2 reduction, group top-k, mask construction, masked expert scores, and final expert top-k |

### Data movement

Every kernel here only moves values (plus zero fill, plus at most one same-order add), so each is bitwise identical to the aten chain it replaces.

| Operator | Backend | Replaces |
| - | - | - |
| `usp_merge_heads` | JIT CUDA | USP all-to-all output head merge (`permute` + `contiguous`) |
| `pack_qkv_destination_major` | Triton | Ulysses destination-major QKV pack |
| `varlen_pack_qkv`, `varlen_scatter_to_padded` | Triton | varlen gather/scatter around the masked attention path |
| `varlen_pack_segmented_qkv` | Triton | varlen gather from a virtual prefix/main Q/K/V sequence |
| `joint_qkv_cat` | Triton | Joy Image Edit's three image/text Q/K/V concatenations, including strided packed-V inputs |
| `causal_conv3d_cat_pad` | KDA (JIT CUDA) / Triton | causal Conv3d `cat` + `pad` |
| `cat_pad_channels_last_3d` | Triton | Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass |
| `dup_up3d_add` | Triton | `repeat_interleave + permute().contiguous() + add` |

Joy Image Edit uses the joint-copy path for eligible CUDA FP16/BF16 image
tensors of at least 32 MiB. It preserves image-first token order, copies
values without arithmetic, and verifies each new shape/stride signature
against the native concatenations before enabling it. Small inputs,
unsupported layouts, gradient-bearing inputs, and unverified signatures
during graph capture use the native path. This does not enable model-level
BCG support.

For large Hopper BF16 image Q/K with 32 heads of width 128, Joy also uses
the existing out-of-place QK-Norm + RoPE kernel to read the packed projection
directly. This removes two input copies while retaining the original CUDA
arithmetic and contiguous outputs. Each new signature is checked bitwise;
inputs remain intact if the operation fails. Other shapes and platforms,
compilation, unverified capture, and `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0`
retain the original helper.

### Quantized layout producers

These kernels preserve the quantized checkpoint path's selected reference operation. They are not a claim that FP8 or NVFP4 is equivalent to an unquantized BF16 checkpoint.

| Operator | Backend | Replaces |
| - | - | - |
| `flux2_layernorm_modulate_fp8_quant` | KDA (Triton) | FLUX.2 LayerNorm plus adaLN modulation directly into static FP8 output |
| `flux2_token_cat_fp8` | KDA (Triton) | FLUX.2 single-block attention/MLP concatenation plus static FP8 quantization |
| `flux2_token_cat_nvfp4` | JIT CUDA | FLUX.2 single-block attention/MLP concatenation plus NVFP4 quantization |

## Coverage by model

Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal.

| Model | Fused paths |
| - | - |
| FLUX.1 | LN+modulate, modulate, residual-gate add, linear+GELU |
| FLUX.2 | LN+modulate, fused LN+modulate-to-FP8, packed SwiGLU, gated residual/norm, residual-gate add, QK RMSNorm+RoPE+joint QKV packing, FP8/NVFP4 token-cat producers |
| Qwen-Image | linear+GELU, select-0/1 LN modulation, added-QKV fusion, QK RMSNorm+RoPE+joint QKV writes, residual norm/modulate+NVFP4 producer |
| GLM-Image | LN+modulate, per-head qk LN, residual-gate add, linear+GELU |
| ERNIE-Image | RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add |
| Joy Image Edit | Bitwise image/text QKV concatenation; strided image QK-Norm + RoPE on Hopper |
| Z-Image | BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm |
| Ideogram 4 | gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add |
| LTX-2 | QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU |
| LTX-2.5 decoder | paired 3D RoPE with shared axis-table cache |
| HunyuanVideo / Helios | QKV+RoPE pack, strided QK RMSNorm, linear+GELU; Helios also has paired in-place Q/K RoPE |
| LingBot Video MoE | Default-on group-limited top-k expert selection; fused RMSNorm, per-token gated residual, and fused RMSNorm+modulate at `quality=extra-high` or `quality=high` |
| Sana | LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add |
| SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; GLUMB bias+SiLU / bias+GLU in eager and BCG; LN+modulate and residual-gate add during BCG; BF16-input linear attention at `quality=extra-high` or `quality=high` |
| Sana-WM | bidirectional gated delta-net, fused QK inverse-RMS |
| Wan / LongLive 2 VAE | temb table slices (Wan); VAE cat+pad and DupUp3D add; lossless FP32 normalization post-ops; full `channels_last_3d` RMSNorm+SiLU at `quality=extra-high` or `quality=high` |
| Cosmos3 | QK-norm + RoPE + KV packing; rounded T2I fusion also supports Hopper TP2/SP1 Super-Text2Image with the 5120-wide SwiGLU architecture |
| Krea2 / MiniMax-H3 | QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation) |
| FLUX.2 VAE / HunyuanVAE / latent upsampler | GroupNorm + SiLU (channels-last two-pass for FLUX.2) |

## Inspecting what is registered

Every kernel is described by a `KernelSpec` in the process-wide registry, so the inventory is queryable without importing any backend:

```python theme={null}
from sglang.kernels.registry import registry

diffusion_ops = [op for op in registry.ops() if op.startswith("diffusion.")]
for spec in registry.get("diffusion.scale_residual_norm_scale_shift"):
    print(spec.backend, spec.target, spec.capabilities)
```

Registration is metadata only — it imports neither torch nor a backend and triggers no JIT build. To pick a specific implementation of an operator that has several:

```python theme={null}
from sglang.kernels import select_kernel, KernelBackend

fn = select_kernel(
    "diffusion.scale_residual_norm_scale_shift", backend=KernelBackend.CUTE_DSL
).load()
```

## Importing the kernels

Runtime code imports from the package, never from a submodule:

```python theme={null}
from sglang.kernels.ops.diffusion import fused_rmsnorm_scale_shift_bitexact
```

Resolution is lazy: the backends have disjoint, heavy dependencies (Triton, CUTLASS/CuTe-DSL, and FlyDSL on ROCm), so an eager re-export would make every one of them an import-time requirement on every platform. Each public kernel is a predicate-plus-kernel pair — call `can_use_<op>(...)` first and fall back to the reference chain when it returns `False`; the kernel raises on an unsupported input rather than silently returning `None`.

The package `README.md` carries a selection matrix for the cases where several kernels look interchangeable and are not. The normalization domain alone holds more than a dozen implementations that differ by numerical contract, activation layout, and backend rather than by speed.

## References

* [Performance Optimization](/docs/sglang-diffusion/performance-optimization)
* [Attention Backends](/docs/sglang-diffusion/attention_backends)
* [Quantization](/docs/sglang-diffusion/quantization)
* [Profiling](/docs/sglang-diffusion/profiling)
* [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion) — source and selection matrix
* [RFC #29630](https://github.com/sgl-project/sglang/issues/29630) — the unified `sglang.kernels` namespace


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.