> ## Documentation Index > Fetch the complete documentation index at: https://lmsysorg-cheng-refactor-decoder-stage-api.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Attention Backends > Select and configure attention backends for SGLang diffusion pipelines. This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them. ## Overview Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`. Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`). `--attention-backend` is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use `--component-attention-backends` when an auxiliary component must use a specific backend; incompatible overrides, including an override for a component that constructs no SGLang attention layer, fail unless a sparse backend is being replaced for cross-attention. When using the diffusers backend, `--attention-backend` is passed through to diffusers' `set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`). * **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel. * **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. * **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA. * **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA. * **MPS**: always uses PyTorch SDPA. * **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA. ## Backend options For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.

CLI value	Enum value	Notes
`fa` / `fa3` / `fa4`	`FA`	FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.post\_init`).
`torch_sdpa`	`TORCH_SDPA`	PyTorch `scaled\_dot\_product\_attention`.
`sliding_tile_attn`	`SLIDING_TILE_ATTN`	Sliding Tile Attention (STA). Requires `st\_attn`. Configure via `--attention-backend-config`.
`sage_attn`	`SAGE_ATTN`	Requires `sageattention`. On Hopper (SM90), PyPI `sageattention==2.2.0` is unsupported because it lacks the upstream SM90 binding fix. Install `pip install --force-reinstall git+[https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4](https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4) --no-build-isolation`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120; see upstream `setup.py`.
`sage_attn_3`	`SAGE_ATTN_3`	Requires SageAttention3 installed per upstream instructions.
`sparge_attn`	`SPARGE_ATTN`	Training-free sparse SageAttention2. CUDA SM80/86/87/89/90, FP16/BF16, head dim 64/128, and square self-attention with sequence length at least 128. Other attention shapes use dense SDPA. Install `pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation`; configure retained block ratio with `topk`.
`sol_attn`	`SOL_ATTN`	Requires the upstream `sol-attn` package. Install with `pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends)`. BF16, head dim 128. Configure via `--attention-backend-config`.
`video_sparse_attn`	`VIDEO_SPARSE_ATTN`	Requires `vsa`. Configure `sparsity` via `--attention-backend-config`.
`subblock_sparse_attn`	`SUBBLOCK_SPARSE_ATTN`	Training-free SubBlock sparsity for MiniMax-H3 on CUDA SM90, SM100, and SM120. The default compute mode is BF16; `compute\_mode=sage\_fp8` is an approximate SM90-only path that requires `SpargeAttn`. See the MiniMax-H3 recipe.
`video_sparse_attn_h3`	`VIDEO_SPARSE_ATTN_H3`	Video Sparse Attention for MiniMax-H3 / FastH3 (VSA-H3). In-tree Triton block-sparse kernel (SM90 / SM100 / SM103); no external package. Configure via `--attention-backend-config`.
`hybrid_window_attn_h3`	`HYBRID_WINDOW_ATTN_H3`	VDN-H3 hybrid attention: chunk-aligned window softmax (exact, gated) plus the Video Delta linear branch over the window complement. The window runs as a union of FlashAttention varlen calls (FA4 on SM100 / SM103 / SM120, FA3 on SM90; on SM80 / SM86 / SM89 the same FA3 build runs its Sm80 mainloop at FA2-class speed). Configure via `--attention-backend-config`.
`vmoba_attn`	`VMOBA_ATTN`	Requires `kernel.attn.vmoba\_attn.vmoba`. Configure via `--attention-backend-config`.
`aiter`	`AITER`	Requires `aiter`.
`aiter\_sage`	`AITER\_SAGE`	Requires `aiter`.
`sla\_attn`	`SLA\_ATTN`	Sparse Linear Attention. Requires `SpargeAttn`. Install with `pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation`.
`sage\_sla\_attn`	`SAGE\_SLA\_ATTN`	SageAttention + Sparse Linear Attention. Requires `SpargeAttn` (same install as SLA).
`sparse_video_gen_2_attn`	`SPARSE_VIDEO_GEN_2_ATTN`	Requires `svg`. See installation instructions at [https://github.com/svg-project/Sparse-VideoGen](https://github.com/svg-project/Sparse-VideoGen).
`laser\_attn`	`LASER\_ATTN`	Requires `attentions` which can be installed with `sgl\_kernel\_npu`; available only for NPU.
`block\_sparse\_attn`	`BLOCK\_SPARSE\_ATTN`	Requires `attentions` which can be installed with `sgl\_kernel\_npu`; available only for NPU.
`rain\_fusion\_attn`	`RAIN\_FUSION\_ATTN`	Requires `attentions` which can be installed with `sgl\_kernel\_npu`; available only for NPU.
`fp8\_fa\_sm120`	`FP8\_FA\_SM120`	FP8 (E4M3) dense attention for SM120 GPUs (GeForce RTX 50, RTX PRO Blackwell), written in CuTe-DSL. Q/K/V are quantized per head on the fly; the output stays BF16. Non-causal, batch 1, head dim 128, BF16 inputs; other calls use `torch\_cudnn\_sdpa`. Opt-in only: outputs differ from BF16 attention by about 5% relative RMS per call. Each new sequence length compiles once (about 10 s).

## Selection priority The selection order in `runtime/layers/attention/selector.py` is: 1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)` 2. Component override from `--component-attention-backends` while that component is being constructed 3. CLI `--attention-backend` (`ServerArgs.attention_backend`) 4. Layer or component default, when declared 5. Auto selection (platform capability, dtype, and installed packages) An explicit global backend mismatch fails for DiT self-attention. Auxiliary components may fall back to their declared default or another compatible backend. Sparse backends selected for self-attention similarly fall back to a compatible dense backend for cross-attention. Explicit component overrides are otherwise strict. Model and layer backend sets guide automatic selection; they are not positive allowlists for explicit requests. A backend selected with `--attention-backend` or `--component-attention-backends` can therefore be used without first adding it to every model's set. It still fails closed when the platform cannot resolve it or when it lacks a required operation such as dense cross-attention, packed variable-length attention, or ring-attention LSE. Model paths that require one backend for correctness declare it as required; those layers keep that backend even when the surrounding component is overridden. For an automatic, measurement-based choice, pass `--enable-attention-backend-autotune true`. On each layer's first sufficiently large input, the native runtime times compatible candidates and switches only when the output remains within its numerical guard and the measured gain exceeds the noise margin. The tuner is disabled by default and currently validated on SM90 and SM12x. It does not replace a backend selected explicitly through `--attention-backend` or `--component-attention-backends`, or a backend required by the model for correctness. ## Configuration Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts: * A path to a JSON or YAML configuration file. * A JSON string (e.g., `'{"sparsity": 0.5}'`). * Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`). ### Supported Configuration Parameters **Sliding Tile Attention (`sliding_tile_attn`)**

Parameter	Type	Description	Default
`mask_strategy_file_path`	`str`	Required. Path to the mask strategy JSON file.	-
`sta_mode`	`str`	Mode of STA.	`STA_inference`
`skip_time_steps`	`int`	Number of steps to use full attention before switching to sparse attention.	`15`

**Video Sparse Attention (`video_sparse_attn`)**

Parameter	Type	Description	Default
`sparsity`	`float`	Validation sparsity (0.0 - 1.0).	`0.0`

Parameter	Type	Description	Default
`VSA_sparsity`	`float`	Fraction of video tiles excluded from the top-k selection (0.0 - 1.0). `0.9` is the FastH3 trained policy.	`0.9`
`vsa_mode`	`str`	`exempt`: non-video keys (text/audio prefix tiles) are always selected. `compete`: they compete with video tiles in the top-k.	`exempt`
`vsa_dense_first_n_steps`	`int`	Use dense attention for the first N denoising steps.	`0`
`vsa_dense_layers`	`list[int]`	Layer indices kept dense, e.g. `[0, 1]`.	`[]`
`vsa_tile_size`	`int`	Kernel tile size. Only `64` (the trained (4, 4, 4) geometry) is accepted.	`64`

VSA-H3 constraints: * Only the DiT runs sparse; the token refiner, text encoder, and VAEs keep their dense defaults. An explicit `--component-attention-backends text_encoder=fa` is rejected because the H3 text encoder has SDPA-only layers. * Uses the checkpoint's trained `to_gate_compress` compression branch. Base MiniMax-H3 weights load zero gates and run pure sparse. * Ulysses sequence parallelism is supported; `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution are rejected. **Hybrid window attention for VDN-H3 (`hybrid_window_attn_h3`)** [VDN-H3](/cookbook/diffusion/MiniMax/MiniMax-H3#7-vdn-h3-hybrid-attention-8-step-distill) replaces every DiT block's dense self-attention with two branches. The softmax branch is an exact softmax over a chunk-aligned frame window (frame `t` belongs to chunk `t // 5` and attends to chunks `c - 1 .. c + 1`; frames 0 and F-1 are dense as rows and columns; every text / audio pair stays dense) scaled by a per-head sigmoid gate. The linear branch (a frame-wise Video Delta rule recurrence, forward and reverse over frames) covers exactly the window's complement and is driven by the attention module. The mask is request-static, so the metadata is built once per request. The checkpoint's `transformer/config.json` carries the window geometry; the backend reads it.

Parameter	Type	Description	Default
`vdn_h3_dense_smoke`	`bool`	Allows a dense transformer backend (`fa`) on the VDN-H3 weights for the base-H3 + LoRA equivalence smoke. The gates and the linear branch are skipped: a study, not a sample.	`false`
`vdn_max_gather_rows`	`int`	Upper bound on the gathered K/V rows per window FlashAttention call; consecutive chunk groups fill one call up to it. Splitting changes no query's kept key set, only the pass count and the gather's peak memory.	`200000`

Hybrid window attention constraints: * VDN-H3 requires this backend for the transformer; a dense backend on these weights would silently skip the linear branch and the gates, so it is rejected unless `vdn_h3_dense_smoke` is set. The token refiner, text encoder, and VAEs keep dense attention. * Base MiniMax-H3 and FastH3 checkpoints have no linear branch and are rejected by this backend. * Ulysses sequence parallelism is supported (QK-norm + RoPE run after the all-to-all on the head shard); `--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph execution are rejected. **V-MoBA (`vmoba_attn`)**

Parameter	Type	Description	Default
`temporal_chunk_size`	`int`	Chunk size for temporal dimension.	-
`temporal_topk`	`int`	Top-K tokens to select in temporal dimension.	-
`spatial_chunk_size`	`list[int]`	Chunk size for spatial dimension (H, W).	-
`spatial_topk`	`int`	Top-K tokens to select in spatial dimension.	-
`st_chunk_size`	`list[int]`	Chunk size for spatiotemporal dimension (T, H, W).	-
`st_topk`	`int`	Top-K tokens to select in spatiotemporal dimension.	-
`moba_select_mode`	`str`	Selection mode (e.g., `threshold`).	`threshold`
`moba_threshold`	`float`	Threshold value for selection.	`0.25`
`moba_threshold_type`	`str`	Type of thresholding (e.g., `query_head`).	`query_head`
`first_full_step`	`int`	Number of initial steps to use full attention.	`12`
`first_full_layer`	`int`	Number of initial layers to use full attention.	`0`
`temporal_layer`	`int`	Number of temporal layers.	`1`
`spatial_layer`	`int`	Number of spatial layers.	`1`
`st_layer`	`int`	Number of spatiotemporal layers.	`1`

**Block Sparse Attention (`block_sparse_attn`)**

Parameter	Type	Description	Default
`skip_first_steps`	`int`	Number of steps to use laser attention before switching to sparse attention.	`10`
`sparsity`	`float`	The sparsity coefficient must be in the range (0, 1).	`0.2`

**Sol-Attn (`sol_attn`)**

Parameter	Type	Description	Default
`tau`	`float`	Routing threshold scale. Higher values select fewer exact KV blocks.	`1.0`
`thresh_type`	`str`	Threshold mode: `diag` or `exact`.	`diag`
`sink_tokens`	`int`	Exact KV sink length for prefix tokens such as text/audio rows.	`0`
`sink_start`	`int`	Start index of the exact KV sink range.	`0`
`dense_steps`	`int`	Use dense attention for the first N denoising steps.	`10`
`dense_layers`	`str`	Layer indices kept dense, e.g. `0,1` or `0-2`.	`0,1`
`dense_backend`	`str`	Backend used for the dense prefix: `fa` (default) or `sage_attn`. `sage_attn` is approximate.	`fa`
`kv_splits`	`int \| str`	KV split factor passed to the Sol-Attn kernel. Use `auto` on long sequences.	`auto`

## Platform support matrix

Backend	CUDA	ROCm	XPU	MUSA	MPS	NPU	Notes
`fa`	Yes	Yes	✅	✅	❌	✅	CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch\_sdpa`. No extra installations are required for NPU
`torch_sdpa`	Yes	Yes	Yes	Yes	✅	✅	Most compatible option across platforms.
`sliding_tile_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `st\_attn`. Configure via `--attention-backend-config`.
`sage_attn`	Yes	No	No	Yes	❌	❌	Optional dependency on CUDA and MUSA. On Hopper, also falls back to FlashAttention when the installed package lacks the SM90 binding fix.
`sage_attn_3`	Yes	No	No	No	❌	❌	CUDA-only (optional dependency).
`sparge_attn`	Yes	No	No	No	❌	❌	CUDA SM80/86/87/89/90 only. Requires `SpargeAttn`; head dim 64/128 and square self-attention with sequence length at least 128.
`sol_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `sol-attn`. Install with `pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends)`. Configure via `--attention-backend-config`.
`video_sparse_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`.
`video_sparse_attn_h3`	Yes	No	No	No	❌	❌	CUDA-only (SM90 / SM100 / SM103). In-tree Triton kernel, no external dependency. Configure via `--attention-backend-config`.
`sla\_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `SpargeAttn`.
`sage\_sla\_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `SpargeAttn`.
`vmoba\_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `kernel.attn.vmoba\_attn.vmoba`. Configure via `--attention-backend-config`.
`aiter`	❌	✅	No	No	❌	❌	Requires `aiter`.
`aiter\_sage`	❌	✅	No	No	❌	❌	Requires `aiter`.
`sparse_video_gen_2_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `svg`.
`laser\_attn`	❌	❌	❌	❌	❌	✅	NPU-only. Requires `attentions` from `sgl\_kernel\_npu`. Uses SDPA if seqlen less than 2048.
`block\_sparse\_attn`	❌	❌	❌	❌	❌	✅	NPU-only. Requires `attentions` from `sgl\_kernel\_npu`. Configuration via `--attention-backend-config`.
`rain\_fusion\_attn`	❌	❌	❌	❌	❌	✅	NPU-only. Requires `attentions` from `sgl\_kernel\_npu` Configuration via `--attention-backend-config`.
`fp8\_fa\_sm120`	Yes	❌	❌	❌	❌	❌	CUDA SM120 only; falls back to `torch\_cudnn\_sdpa` on other devices and for unsupported calls.

## Usage ### Select a backend via CLI ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa ``` ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend torch_sdpa ``` ### Override one component Use component overrides when a specific module needs different attention semantics from the main transformer: ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Use this override when the fallback must be pinned: unlike the global backend, an incompatible component override raises an error instead of selecting another backend. The one role-based exception is a sparse self-attention backend, which uses a compatible dense backend for cross-attention layers in the same component. The component must construct SGLang-selectable attention or explicitly defer selection until first use; components with fixed attention reject the override. Per-component overrides apply only to native pipelines. The Diffusers backend accepts the global `--attention-backend` passthrough instead. ### Per-request override (denoise loop) A single server can serve exact and approximate attention side by side: requests may switch the DiT denoise attention backend via the `attention_backend_override` sampling param. Valid values are the exact/drop-in dense kernels — `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`. The field participates in the dynamic-batch signature, so requests with different backends never share a batch. ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend-override sage_attn ``` ```python theme={null} client.images.generate( model="", prompt="...", extra_body={"attention_backend_override": "sage_attn"}, ) ``` Incompatible server settings **reject the request** (with a server log) instead of silently falling back: breakable CUDA graphs and `torch.compile` bake the attention kernel into a captured/traced graph; sparse server backends (`sliding_tile_attn`, `video_sparse_attn`, ...) cannot be mixed with per-request dense switching; under ring parallelism the target must be ring-capable. Note `sage_attn` / `sage_attn_3` are lossy (quantized attention) — validate quality on your workload. ### Request-scoped Skip Softmax Skip Softmax (BLASST) keeps the QK matmul, but skips the exponential, softmax-state update, V load, and PV matmul for attention tiles whose estimated softmax mass is below a threshold. It is an explicit lossy optimization backed by the FlashInfer kernels shipped with SGLang's pinned dependencies. See the [BLASST paper](https://arxiv.org/abs/2512.12087) and NVIDIA's [video-generation study](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog28_Accelerating_Video_Generation_with_GEMM_Quantization_Attention_Quantization_and_Skip_Softmax_Attention_in_TensorRT-LLM.md) for the algorithm and measured quality/performance trade-offs. Pass `skip_softmax_params` in one image or video request. The runtime routes compatible self-attention layers through the FA/FlashInfer path for that batch; cross-attention keeps its normal backend, and the next request restores the server default. ```python theme={null} client.videos.create( model="", prompt="...", extra_body={ "skip_softmax_params": { "threshold_scale_factor": 500.0, "start_step": 14, } }, ) ``` * `threshold_scale_factor` is required and must be positive. The kernel uses `threshold_scale_factor / context_length`; larger values skip more work and usually increase quality loss. * `start_step` is the zero-based denoising step at which sparse execution starts. It defaults to `0`; keeping early high-noise steps dense is generally safer. There is intentionally no default threshold. Calibrate the threshold and start step against output-quality metrics for each model, resolution, step count, and deployment backend; values tuned for one workload are not portable. Current support is FP16/BF16, head dimension 128 or 256, and unmasked self-attention on Hopper SM90 and Blackwell SM100/SM103/SM107. Ulysses sequence parallelism is supported because the kernel runs after its all-to-all. Ring Attention, attention masks, `torch.compile`, and breakable CUDA graphs reject the request instead of silently running dense attention. Models that merely offer an FA backend do not automatically qualify: the runtime still checks the GPU, dtype, head dimension, attention role, and execution mode. ### Using SpargeAttention Install the optional CUDA extension, then select the backend explicitly: ```bash theme={null} pip install ninja pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation sglang generate \ --model-path \ --prompt "..." \ --attention-backend sparge_attn \ --attention-backend-config topk=0.5 ``` `sparge_attn` applies to compatible self-attention layers. Cross-attention uses the existing dense fallback; short self-attention and asymmetric Q/KV also use dense SDPA. LTX-2, LTX-2.3, and LTX-2.5 have compatible video (128) and audio (64) head dimensions, so their sufficiently long, unmasked self-attention uses SpargeAttention while prompt and audio-video cross-attention remain dense. Ulysses sequence parallelism is supported, but ring attention is not because the upstream kernel does not expose the softmax LSE needed for ring merging. K/V-gather SP therefore follows the normal sparse-backend rule and uses Ulysses instead. ### Sage then Sol hybrid `sol_attn` keeps the first `dense_steps` steps dense. Set `dense_backend=sage_attn` to run that prefix on SageAttention and the tail on Sol sparse attention. Keep the text encoder on `torch_sdpa`: ```bash theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --attention-backend sol_attn \ --attention-backend-config dense_backend=sage_attn,dense_steps=10 \ --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` Both `sage_attn` and `sol_attn` are approximate. The default DiT backend remains `fa`. ### Using Sliding Tile Attention (STA) ```bash theme={null} # Pass the mask strategy file path via config sglang generate \ --model-path \ --prompt "..." \ --attention-backend sliding_tile_attn \ --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json" ``` ### Notes for ROCm / MPS * ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment. * MPS: the platform implementation always uses `torch_sdpa`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.