| Lever | +Effect | +
|---|---|
--mamba-radix-cache-strategy extra_buffer_lazy |
+ 4 state slots per request instead of 5 | +
--mamba-ssm-dtype bfloat16 |
+ ~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton | +
--kv-cache-dtype fp8_e4m3 |
+ halves KV bytes per token; under PD both roles must match at connect | +
--mem-fraction-static 0.90–0.92 |
+ cheapest first win when the boot log shows a large idle avail mem |
+
SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1 |
+ frees one more slot per request (experimental, under validation) | +
| Platform | +Topology | +Notes | +
|---|---|---|
| B300 1×8 | +TP8 (+DCP8) | +accuracy-first defaults on Low-Latency and Balanced | +
| GB300 2×4 | +TP8/DCP8 | +MNNVL transport and cuMem auto-detected | +
| B200 2×8 | +PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | +Unified serves all three operating points; Long-Context is a Prefill-only strategy |
+
| GB200 4×4 | +TP16/DCP16 | +MNNVL auto-detected | +
| H200 2×8 (4×8 on Unified High-Throughput) | +TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with extra_buffer_lazy |
+ same block on every node; export the cross-node NIC (GLOO_SOCKET_IFNAME / NCCL_SOCKET_IFNAME, SGLANG_HOST_IP); keep NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1 |
+
| H100 4×8 | +TP32/EP32, Marlin + FlashMLA | +SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) | +
| MI350X/MI355X 1×8 | +TP8/DCP8 ROCm/AITER (Unified Balanced) | +AITER A8W4 FlyDSL MoE, AITER prefill/decode attention with sharded target MLA KV, graph bs up to 256, fp8 kvcache; DSPARK supported. Activation-quant and fused-KDA-decode knobs: AMD ROCm/AITER environment | +
| Ascend A3 Series 4×8 (32 cards / 64 dies) | +TP64/DP4 + DeepEP | +Unified plus an asymmetric 2P4D PD split — 2-node prefill (TP16/PP2/DP2, NOSPEC, no CUDA graph) + 4-node decode (TP64/DP4, DSPARK); pin GLOO/HCCL_SOCKET_IFNAME on every node |
+
| Ascend 950PR/DT Series 4×8 | +TP32/dp1 + DeepEP | +Unified only; DSPARK baked in; shared experts / dense MLP shard over attention-TP (--shared-experts-tp-size 4); radix cache off; pin GLOO/HCCL_SOCKET_IFNAME on every node |
+
| Env var | +Default | +Effect | +
|---|---|---|
AITER_SITUV2_A8W4=1 |
+ unset | +SiTU v2 MoE with A8W4 activation quantization, on AITER's GU-interleaved preshuffled weight layout. The performance default the cell ships. | +
AITER_SITUV2_A4W4=1 |
+ unset | +A4W4 instead, on the generic separated shuffle layout. Numerically correct but slower than A8W4 (530.8 vs 537.3 tok/s median output on 8×MI35x). Setting both gives A8W4 precedence — SGLang follows AITER and keeps the GU-interleaved layout. | +
SGLANG_ROCM_K3_KDA_FUSED_BACKEND=aiter |
+ unset | +Opt in to the fused ROCm KDA decode boundary: the f_b projection is deferred into the gfx950 FlyDSL kernel so decode fuses f_b + convolution + recurrent state update + gated RMSNorm. Any other value (or unset) keeps the unfused KDA path. |
+
SGLANG_K3_FLYDSL_SOURCE |
+ auto |
+ Which FlyDSL implementation backs the fused decode: auto prefers SGLang's vendored kernels and falls back to the AITER module, sglang / aiter pin one. Leave it alone unless you are bisecting the two. |
+
| Pool layout | +A3 Series (2P4D) | +
|---|---|
| Prefill pool | +2 nodes, TP16 × PP2 × DP2, NOSPEC, no CUDA graph | +
| Decode pool | +4 nodes, TP64 × DP4, DSPARK, decode graph at bs 16 | +
| Total | +6 nodes | +
| Picker selection | +Processor-to-scheduler feature path | +
|---|---|
| Auto · single-node Unified CUDA | +CUDA IPC | +
| Auto · Unified GB200/GB300 | +CUDA VMM when IMEX is available; CPU otherwise | +
| Auto · PD or other topologies | +CPU | +
| CPU | +CPU, with no GPU feature pool | +
| Feature | +K3 behavior | +
|---|---|
| PD | +Supported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in PD disaggregation. | +
| EPD | +Supported on the public kimi-k3 branch. Use an --encoder-only vision role and a --language-only prefill role; add the normal decode role for full EPD. See the EPD guide. |
+
| MM encoder DP | +Built in. K3 shards complete images across TP ranks, so leave --mm-enable-dp-encoder unset in unified, PD-prefill, and encoder-only roles. |
+
| MM feature transport | +Processor-to-scheduler features only. EPD encoder output and PD KV/KDA transfer use their own backends. | +
| ViT BCG | +Compatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below. | +
| Preset | +What it trades | +Pick it for | +
|---|---|---|
Peak Throughput — dp = k, attention-TP 8 |
+ State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. --kv-cache-dtype fp8_e4m3 is load-bearing — bf16 KV does not fit 128 requests per replica. |
+ Maximum sustained TPS — the default large-scale shape. | +
Peak Capacity (+DCP8) — dp = k + --dcp-size 8 |
+ Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL. | +Context ≥ ~16K, or per-replica concurrency past 128. | +
| GPUs | +B200/B300 nodes | +GB200/GB300 nodes | +--tp-size / --ep-size |
+ --attn-dp-size |
+
|---|---|---|---|---|
| 16 | +2×8 | +4×4 | +16 | +2 | +
| 32 | +4×8 | +8×4 | +32 | +4 | +
| 64 | +8×8 | +16×4 | +64 | +8 | +