Qwen3.8-Flash-Next-NVFP4
«- back to models- - for coding and everyday use
- - ~99.8% intelligence retained vs BF16
- - 133 GB pack, 52% of 360 GB BF16
- - ~7.8x speedup (23.6 → 184 tok/s @ c1)
Performance (Measured on 1× RTX 6000 Pro Max-Q)
Agentic coding throughput
Replay of 10 SWE-bench Pro / DeepSWE mini-swe-agent conversations × 6 turns each. Full chat history + tool observations fed back each turn; prefix cache on; hot32; context 16k.
| Concurrency | out tok/s | accept length | TTFT ms | ITL ms |
|---|---|---|---|---|
| 1 | 184 | 3.13 | 192 | 4.9 |
| 4 | 334 | 3.13 | 354 | 9.9 |
Cold prefill and long-context decode
Same stack with --disable-radix-cache and --flush-cache between lengths so every request is cold. sglang.benchmark.serving random, 8 prompts, concurrency 1.
| prompt | decode tok/s | prefill tok/s | TTFT mean / p50 / p90 / p99 (ms) | ITL mean / p50 / p90 / p99 (ms) | accept |
|---|---|---|---|---|---|
| 1024 | 152 | ~9.4k | 109 / 98 / 139 / 154 | 5.8 / 3.7 / 14.4 / 14.9 | 2.51 |
| 4096 | 133 | ~19k | 214 / 203 / 319 / 320 | 6.1 / 3.8 / 14.4 / 15.0 | 2.49 |
| 16384 | 89 | ~21k | 769 / 763 / 1230 / 1474 | 5.9 / 3.8 / 14.4 / 15.0 | 2.53 |
| 32768 | 69 | ~26k | 1257 / 1141 / 2123 / 2458 | 5.8 / 3.8 / 14.3 / 15.1 | 2.57 |
Decode slows as context grows; ITL stays flat; TTFT scales with prompt length.
Optimisation
- - NVFP4 W4A4 routed experts (E2M1, group 16); attention and shared experts stay BF16
- - MTP drafter with FP8 block-scaled experts + hot-vocab map (spec/hot_self_32k.pt)
- - NEXTN speculative decoding (3 steps, 4 draft tokens)
- - FlashInfer CUTLASS FP4 GEMM kernels (flashinfer_cutlass)
- - PLE n-gram embedding offloaded to pinned host memory
- - FP8 KV cache + full CUDA graphs on decode