Qwen3.8-Flash-Next-Ternary
«- back to models- - for fast retrieval and data analysis
- - ~91% intelligence retained vs BF16
- - 145 GB, 9% of 360 GB BF16
- - ~4.8x speedup
Performance (Measured on 1× RTX 6000 Pro Max-Q)
Agentic coding throughput
Replay of 10 SWE-bench Pro / DeepSWE mini-swe-agent conversations × 6 turns each. Full chat history + tool observations fed back each turn; prefix cache on; hot32; NEXTN 3/4.
| Concurrency | out tok/s | accept length | TTFT ms | ITL ms |
|---|---|---|---|---|
| 1 | 168 | 2.99 | 338 | 5.2 |
| 4 | 265 | 3.03 | 451 | 13.3 |
Cold prefill and long-context decode
Same stack with --disable-radix-cache and --flush-cache between lengths. sglang.benchmark.serving random, 8 prompts, concurrency 1.
| prompt | decode tok/s | prefill tok/s | TTFT mean / p50 / p90 / p99 (ms) | ITL mean / p50 / p90 / p99 (ms) | accept |
|---|---|---|---|---|---|
| 1024 | 135 | ~4.2k | 247 / 179 / 448 / 482 | 5.7 / 4.1 / 8.3 / 16.6 | 2.88 |
| 4096 | 114 | ~9.2k | 444 / 402 / 701 / 820 | 5.7 / 4.2 / 8.4 / 16.8 | 2.98 |
| 16384 | 61 | ~11k | 1490 / 1524 / 2322 / 2521 | 5.9 / 4.2 / 8.4 / 16.9 | 2.82 |
| 32768 | 43 | ~13k | 2517 / 2267 / 4240 / 4861 | 5.7 / 4.2 / 8.4 / 17.0 | 3.00 |
Decode slows as context grows; ITL stays flat; TTFT scales with prompt length.
Optimisation
- - Hadamard-rotated ternary GPTQ expert overlay (group 128, 48 layers, gate_up_down)
- - Attention, router, shared experts, and MTP stay in the base BF16 checkpoint
- - NEXTN speculative decoding from base MTP (3 steps, 4 draft tokens) + optional hot-vocab map
- - Goethe ternary_runtime kernels for ternary MoE decode
- - FP8 KV cache + full CUDA graphs on decode