From 1e036ed714dd642e790be0a6b20774f1a75fe207 Mon Sep 17 00:00:00 2001 From: Yu-Yun Chang Date: Wed, 28 Oct 2026 09:58:12 +0000 Subject: [PATCH] docs: convert the Kimi-K3 cookbook page's pipe tables to JSX tables - Convert all 8 markdown pipe tables to the JSX style of the DeepSeek-V4 page - Markdown pipe tables are forbidden by the cookbook-review-pr checklist (section 9b) - Cell text and inline markup are unchanged in all 116 cells; nothing outside the tables changes --- .../autoregressive/Moonshotai/Kimi-K3.mdx | 339 +++++++++++++++--- 1 file changed, 288 insertions(+), 51 deletions(-) diff --git a/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx b/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx index 1d457d5c99ff..4fa8b135e949 100644 --- a/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx +++ b/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx @@ -170,13 +170,40 @@ not been re-measured on any cell — re-measure before you rely on one. Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload: -| Lever | Effect | -|---|---| -| `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 | -| `--mamba-ssm-dtype bfloat16` | ~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton | -| `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect | -| `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` | -| `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) | +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
LeverEffect
--mamba-radix-cache-strategy extra_buffer_lazy4 state slots per request instead of 5
--mamba-ssm-dtype bfloat16~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton
--kv-cache-dtype fp8_e4m3halves KV bytes per token; under PD both roles must match at connect
--mem-fraction-static 0.90–0.92cheapest first win when the boot log shows a large idle avail mem
SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1frees one more slot per request (experimental, under validation)
Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise). @@ -190,17 +217,67 @@ Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request **Per-platform notes:** -| Platform | Topology | Notes | -|---|---|---| -| B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced | -| GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected | -| B200 2×8 | PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | Unified serves all three operating points; `Long-Context` is a `Prefill`-only strategy | -| GB200 4×4 | TP16/DCP16 | MNNVL auto-detected | -| H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` | -| H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) | -| MI350X/MI355X 1×8 | TP8/DCP8 ROCm/AITER (Unified Balanced) | AITER A8W4 FlyDSL MoE, AITER prefill/decode attention with sharded target MLA KV, graph bs up to 256, fp8 kvcache; DSPARK supported. Activation-quant and fused-KDA-decode knobs: [AMD ROCm/AITER environment](#amd-env) | -| Ascend A3 Series 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | `Unified` plus an asymmetric 2P4D PD split — 2-node prefill (TP16/PP2/DP2, NOSPEC, no CUDA graph) + 4-node decode (TP64/DP4, DSPARK); pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | -| Ascend 950PR/DT Series 4×8 | TP32/dp1 + DeepEP | `Unified` only; DSPARK baked in; shared experts / dense MLP shard over attention-TP (`--shared-experts-tp-size 4`); radix cache off; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
PlatformTopologyNotes
B300 1×8TP8 (+DCP8)accuracy-first defaults on Low-Latency and Balanced
GB300 2×4TP8/DCP8MNNVL transport and cuMem auto-detected
B200 2×8PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16Unified serves all three operating points; Long-Context is a Prefill-only strategy
GB200 4×4TP16/DCP16MNNVL auto-detected
H200 2×8 (4×8 on Unified High-Throughput)TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with extra_buffer_lazysame block on every node; export the cross-node NIC (GLOO_SOCKET_IFNAME / NCCL_SOCKET_IFNAME, SGLANG_HOST_IP); keep NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1
H100 4×8TP32/EP32, Marlin + FlashMLASM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB)
MI350X/MI355X 1×8TP8/DCP8 ROCm/AITER (Unified Balanced)AITER A8W4 FlyDSL MoE, AITER prefill/decode attention with sharded target MLA KV, graph bs up to 256, fp8 kvcache; DSPARK supported. Activation-quant and fused-KDA-decode knobs: AMD ROCm/AITER environment
Ascend A3 Series 4×8 (32 cards / 64 dies)TP64/DP4 + DeepEPUnified plus an asymmetric 2P4D PD split — 2-node prefill (TP16/PP2/DP2, NOSPEC, no CUDA graph) + 4-node decode (TP64/DP4, DSPARK); pin GLOO/HCCL_SOCKET_IFNAME on every node
Ascend 950PR/DT Series 4×8TP32/dp1 + DeepEPUnified only; DSPARK baked in; shared experts / dense MLP shard over attention-TP (--shared-experts-tp-size 4); radix cache off; pin GLOO/HCCL_SOCKET_IFNAME on every node
**Blackwell DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles: @@ -216,12 +293,42 @@ Outside the two verified B300 1×8 `Unified` cells, no cell has a serving round **AMD ROCm/AITER environment (MI350X / MI355X).** The MI35x cell emits `SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1` — the first three turn on the AITER ROCm path, its K3-specific fused kernels, and the FlyDSL MoE kernels; the rest of this table is what you can change on top. Everything here is gfx950/ROCm-only and inert elsewhere. These knobs first ship in the `20260903` daily ROCm image pinned above — on an older image they are simply unread — and the two SiTU rows also need an AITER revision at or past [ROCm/aiter#4534](https://github.com/ROCm/aiter/pull/4534) (FlyDSL 0.3.0). -| Env var | Default | Effect | -|---|---|---| -| `AITER_SITUV2_A8W4=1` | unset | SiTU v2 MoE with A8W4 activation quantization, on AITER's GU-interleaved preshuffled weight layout. The performance default the cell ships. | -| `AITER_SITUV2_A4W4=1` | unset | A4W4 instead, on the generic separated shuffle layout. Numerically correct but slower than A8W4 (530.8 vs 537.3 tok/s median output on 8×MI35x). Setting **both** gives A8W4 precedence — SGLang follows AITER and keeps the GU-interleaved layout. | -| `SGLANG_ROCM_K3_KDA_FUSED_BACKEND=aiter` | unset | Opt in to the fused ROCm KDA decode boundary: the `f_b` projection is deferred into the gfx950 FlyDSL kernel so decode fuses `f_b` + convolution + recurrent state update + gated RMSNorm. Any other value (or unset) keeps the unfused KDA path. | -| `SGLANG_K3_FLYDSL_SOURCE` | `auto` | Which FlyDSL implementation backs the fused decode: `auto` prefers SGLang's vendored kernels and falls back to the AITER module, `sglang` / `aiter` pin one. Leave it alone unless you are bisecting the two. | + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Env varDefaultEffect
AITER_SITUV2_A8W4=1unsetSiTU v2 MoE with A8W4 activation quantization, on AITER's GU-interleaved preshuffled weight layout. The performance default the cell ships.
AITER_SITUV2_A4W4=1unsetA4W4 instead, on the generic separated shuffle layout. Numerically correct but slower than A8W4 (530.8 vs 537.3 tok/s median output on 8×MI35x). Setting both gives A8W4 precedence — SGLang follows AITER and keeps the GU-interleaved layout.
SGLANG_ROCM_K3_KDA_FUSED_BACKEND=aiterunsetOpt in to the fused ROCm KDA decode boundary: the f_b projection is deferred into the gfx950 FlyDSL kernel so decode fuses f_b + convolution + recurrent state update + gated RMSNorm. Any other value (or unset) keeps the unfused KDA path.
SGLANG_K3_FLYDSL_SOURCEautoWhich FlyDSL implementation backs the fused decode: auto prefers SGLang's vendored kernels and falls back to the AITER module, sglang / aiter pin one. Leave it alone unless you are bisecting the two.
The fused KDA backend is fail-closed at two levels: it arms during model init only when the flag is exactly `aiter` **and** the gfx950 FlyDSL kernels are importable, and each decode step re-validates shapes, dtypes, strides, state indices, and output buffers before dispatch — anything unexpected falls back to the stock KDA implementation rather than erroring. Batch size 2 automatically picks a separately validated kernel schedule; every other batch keeps the original build options. Measured on a 69-layer graph the fused boundary is 9.20 → 8.38 µs/layer (−8.9%), with GSM8K 1319 at 0.950. @@ -323,11 +430,32 @@ PD splits prefill and decode into separate server groups; because K3 is hybrid, The A3 Series disaggregates over the **Ascend memory fabric** (`--disaggregation-transfer-backend ascend`) in an asymmetric **2P4D** pool layout: -| Pool layout | A3 Series (2P4D) | -|---|---| -| Prefill pool | 2 nodes, TP16 × PP2 × DP2, NOSPEC, no CUDA graph | -| Decode pool | 4 nodes, TP64 × DP4, DSPARK, decode graph at bs 16 | -| Total | 6 nodes | + + + + + + + + + + + + + + + + + + + + + + + + + +
Pool layoutA3 Series (2P4D)
Prefill pool2 nodes, TP16 × PP2 × DP2, NOSPEC, no CUDA graph
Decode pool4 nodes, TP64 × DP4, DSPARK, decode graph at bs 16
Total6 nodes
- Every rank of **both** pools registers its transfer buffers against a store served from the **prefill head node** (`ASCEND_MF_STORE_URL`) — set the prefill head's IP on the decode nodes too, not just the prefill pool. - The A3 prefill role is **pipelined** (PP2) and runs NOSPEC — the Spec Decode overlay ships DSPARK on the flat recipes here; newer releases admit it behind a PD prefill ([sgl-project/sglang#40045](https://github.com/sgl-project/sglang/pull/40045)), but no measured A3 command uses it yet. @@ -369,12 +497,36 @@ processor rejects video and audio input. Use **VLM Transport** in the command picker. `Auto` is a topology-aware starting point, not a claim that one configuration is fastest for every workload. -| Picker selection | Processor-to-scheduler feature path | -|---|---| -| Auto · single-node Unified CUDA | CUDA IPC | -| Auto · Unified GB200/GB300 | CUDA VMM when IMEX is available; CPU otherwise | -| Auto · PD or other topologies | CPU | -| CPU | CPU, with no GPU feature pool | + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Picker selectionProcessor-to-scheduler feature path
Auto · single-node Unified CUDACUDA IPC
Auto · Unified GB200/GB300CUDA VMM when IMEX is available; CPU otherwise
Auto · PD or other topologiesCPU
CPUCPU, with no GPU feature pool
CUDA IPC and CUDA VMM reserve up to `SGLANG_MM_FEATURE_CACHE_MB` (1 GiB by default) on the base GPU and fall back to CPU per tensor when full. This setting @@ -383,13 +535,40 @@ does not control EPD encoder output or PD KV/KDA transfer. K3 already defaults t #### VLM compatibility -| Feature | K3 behavior | -|---|---| -| PD | Supported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in [PD disaggregation](#pd-disaggregation). | -| EPD | Supported on the public `kimi-k3` branch. Use an `--encoder-only` vision role and a `--language-only` prefill role; add the normal decode role for full EPD. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation). | -| MM encoder DP | Built in. K3 shards complete images across TP ranks, so leave `--mm-enable-dp-encoder` unset in unified, PD-prefill, and encoder-only roles. | -| MM feature transport | Processor-to-scheduler features only. EPD encoder output and PD KV/KDA transfer use their own backends. | -| ViT BCG | Compatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below. | + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FeatureK3 behavior
PDSupported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in PD disaggregation.
EPDSupported on the public kimi-k3 branch. Use an --encoder-only vision role and a --language-only prefill role; add the normal decode role for full EPD. See the EPD guide.
MM encoder DPBuilt in. K3 shards complete images across TP ranks, so leave --mm-enable-dp-encoder unset in unified, PD-prefill, and encoder-only roles.
MM feature transportProcessor-to-scheduler features only. EPD encoder output and PD KV/KDA transfer use their own backends.
ViT BCGCompatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below.
#### Should ViT BCG be enabled? @@ -430,10 +609,32 @@ sglang serve \ Two presets come out of this, at `N = 8k` GPUs: -| Preset | What it trades | Pick it for | -|---|---|---| -| **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. | -| **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL. | Context ≥ ~16K, or per-replica concurrency past 128. | + + + + + + + + + + + + + + + + + + + + + + + + + +
PresetWhat it tradesPick it for
Peak Throughput — dp = k, attention-TP 8State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. --kv-cache-dtype fp8_e4m3 is load-bearing — bf16 KV does not fit 128 requests per replica.Maximum sustained TPS — the default large-scale shape.
Peak Capacity (+DCP8) — dp = k + --dcp-size 8Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL.Context ≥ ~16K, or per-replica concurrency past 128.
- **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5. - The fully data-parallel extreme (`--attn-dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's ~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape. @@ -459,11 +660,47 @@ sglang serve \ Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation: -| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--attn-dp-size` | -|---|---|---|---|---| -| 16 | 2×8 | 4×4 | 16 | 2 | -| 32 | 4×8 | 8×4 | 32 | 4 | -| 64 | 8×8 | 16×4 | 64 | 8 | + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
GPUsB200/B300 nodesGB200/GB300 nodes--tp-size / --ep-size--attn-dp-size
162×84×4162
324×88×4324
648×816×4648
For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator).