Attention Backends

​

Overview

• CUDA: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel.
• ROCm: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
• Intel XPU: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA.
• MUSA: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA.
• MPS: always uses PyTorch SDPA.
• NPU: for ring attention uses FA otherwise uses PyTorch SDPA.

​

Backend options

​

Selection priority

1. global_force_attn_backend(...) / global_force_attn_backend_context_manager(...)
2. Component override from --component-attention-backends while that component is being constructed
3. CLI --attention-backend (ServerArgs.attention_backend)
4. Layer or component default, when declared
5. Auto selection (platform capability, dtype, and installed packages)

​

Configuration

• A path to a JSON or YAML configuration file.
• A JSON string (e.g., '{"sparsity": 0.5}').
• Key-value pairs (e.g., "sparsity=0.5,enable_x=true").

​

Supported Configuration Parameters

• Only the DiT runs sparse; the token refiner, text encoder, and VAEs keep their dense defaults. An explicit --component-attention-backends text_encoder=fa is rejected because the H3 text encoder has SDPA-only layers.
• Uses the checkpoint’s trained to_gate_compress compression branch. Base MiniMax-H3 weights load zero gates and run pure sparse.
• Ulysses sequence parallelism is supported; --ring-degree greater than 1, torch.compile, and breakable CUDA graph execution are rejected.

• VDN-H3 requires this backend for the transformer; a dense backend on these weights would silently skip the linear branch and the gates, so it is rejected unless vdn_h3_dense_smoke is set. The token refiner, text encoder, and VAEs keep dense attention.
• Base MiniMax-H3 and FastH3 checkpoints have no linear branch and are rejected by this backend.
• Ulysses sequence parallelism is supported (QK-norm + RoPE run after the all-to-all on the head shard); --ring-degree greater than 1, torch.compile, and breakable CUDA graph execution are rejected.

​

Platform support matrix

​

Usage

​

Select a backend via CLI

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend fa

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend torch_sdpa

​

Override one component

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend fa \
  --component-attention-backends text_encoder=torch_sdpa

​

Per-request override (denoise loop)

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend-override sage_attn

client.images.generate(
    model="<MODEL_PATH_OR_ID>",
    prompt="...",
    extra_body={"attention_backend_override": "sage_attn"},
)

​

Request-scoped Skip Softmax

client.videos.create(
    model="<MODEL_PATH_OR_ID>",
    prompt="...",
    extra_body={
        "skip_softmax_params": {
            "threshold_scale_factor": 500.0,
            "start_step": 14,
        }
    },
)

• threshold_scale_factor is required and must be positive. The kernel uses threshold_scale_factor / context_length; larger values skip more work and usually increase quality loss.
• start_step is the zero-based denoising step at which sparse execution starts. It defaults to 0; keeping early high-noise steps dense is generally safer.

​

Using SpargeAttention

pip install ninja
pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend sparge_attn \
  --attention-backend-config topk=0.5

​

Sage then Sol hybrid

sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --attention-backend sol_attn \
  --attention-backend-config dense_backend=sage_attn,dense_steps=10 \
  --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output

​

Using Sliding Tile Attention (STA)

# Pass the mask strategy file path via config
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend sliding_tile_attn \
  --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"

​

Notes for ROCm / MPS

• ROCm: use --attention-backend torch_sdpa or fa depending on what is available in your environment.
• MPS: the platform implementation always uses torch_sdpa.