Qwen3.8-27B-NVFP4-NInfer
«- back to models- - for personal agents
- - within ~5% of FP16
- - 25 GB, 44% of 56 GB BF16
- - ~3.7x speedup (69.5 → 255 tok/s @ c1)
Performance (Measured on 1× GeForce RTX 5090)
Agentic coding throughput
Replay of recorded mini-swe-agent turns, mean context ~28k, thinking on, prefix cache on, DFlash2 K=7 + lm-head-draft patches.
| Concurrency | out tok/s | accept length | TTFT ms | ITL ms |
|---|---|---|---|---|
| 1 | 255 | 4.40 | 1438 | 4.3 |
| 4 | 686 | 4.35 | 2421 | 5.4 |
Cold prefill and long-context decode
long profile, --no-prefix-reuse, 8 prompts, out=256, concurrency 1, greedy. Prefill tok/s ≈ prompt_len / mean_TTFT.
| prompt | decode tok/s | prefill tok/s | TTFT mean / p50 / p90 / p99 (ms) | ITL mean / p50 / p90 / p99 (ms) |
|---|---|---|---|---|
| 1024 | 211 | ~16k | 65 / 56 / 105 / 116 | 17.1 / 17.1 / 17.2 / 17.3 |
| 4096 | 147 | ~16k | 254 / 248 / 394 / 413 | 17.2 / 17.2 / 17.3 / 17.4 |
| 16384 | 81 | ~16k | 998 / 1010 / 1606 / 1777 | 17.4 / 17.4 / 17.7 / 17.8 |
| 32768 | 56 | ~18k | 1778 / 1586 / 3068 / 3639 | 17.6 / 17.6 / 18.0 / 18.2 |
Optimisation
- - NVFP4 dense 27B on NInfer (Goethe nvfp4-kernels); hybrid NVFP4 MLP + FP8 attention/head
- - DFlash2 speculative decoding (default, K=7) or MTP (batch profile, K=4); --lm-head-draft
- - FP8 KV cache, CUDA Graph decode, prefill chunk 1024, prefix reuse
- - MLP down-projection retile for ≤16-token drafts (~1.4% faster decode, bit-identical)
- - Draft shortlist patched to include <tool_call> tokens (~3.5% on agent traffic; ~5% combined vs unpatched)