Home › Articles › KV Cache Quantization Benchmarks for Long Context

KV Cache Quantization Benchmarks for Long Context

Tests on Qwen 3.6 27B show why TurboQuant is overrated but saved by TCQ, q5 deserves more attention, and symmetric q8 might be a waste of VRAM.

1. The Setup
2. PPL Hides the Tail, KLD Exposes It
3. Rotation Changed the Playing Field
4. What TCQ Adds
5. Asymmetric KV: Where the Bits Should Go
6. TurboQuant's Narrowed Niche
7. The q8 Fidelity Tier
8. The Preset Ladder
9. Weight Quant Changes the Cost of Cache Quant
10. What Else Was Tried
11. Benchmark Data
    1. Perplexity
    2. KL Divergence

Disclaimer: this is just results of a few basic KV cache benchmarks, not the whole truth. Longer context should make the differences more pronounced, but that still doesn't guarantee 1:1 with real usage. Maybe TurboQuant magic works better with actually getting tool calls correctly at extra long context in agentic coding, or maybe it's the other way around. I tried some other tests too, but in terms of meaningful results PPL and KLD are all I've got.

1. The Setup

This benchmark started with a narrow question: does TurboQuant-style KV cache compression have any defensible niche in BeeLlama's (my llama.cpp fork) recommendation set, or is q4/q5 strictly better where it fits?

Hardware: one RTX 3090 (24 GB VRAM), Ryzen 7 5700X3D, 32 GB RAM. Model: Qwen 3.6 27B. Q5_K_S weights plus a 64k bf16 KV cache fit on the card; 128k does not, the weights alone fill too much VRAM. IQ4_XS weights are smaller, so IQ4_XS plus a 128k bf16 KV cache fits. Each context length is capped at the maximum where bf16 KV still runs, because bf16 is the reference every quantized mode is measured against. IQ4_XS at 64k is also included to compare directly against Q5_K_S at the same 64k context, isolating the effect of weight quant on cache quality (§9).

The tok/s column in the benchmark data tables is prefill throughput (batch size 2048, ubatch 256), not generation speed. GPU power was lowered during the runs. These numbers are useful only for relative comparison between cache modes on the same hardware, the absolute values do not represent real-world generation performance.

The tests ran on BeeLlama.cpp v0.1.2, which keeps the normal llama.cpp flow but adds DFlash plus TurboQuant/TCQ KV-cache compression. The TurboQuant implementation being evaluated comes from TheTom's llama-cpp-turboquant, while the TCQ variants come from buun's llama.cpp fork and the accompanying TCQ paper and codebooks.

2. PPL Hides the Tail, KLD Exposes It

The PPL numbers look flat across the board. Even turbo modes barely move the average:

Full PPL results for all cache modes and configurations are in §11.1.

q5_0 uses 34.4% of bf16 KV and saves 65% memory; q4_0 uses 28.1%. Through q4_0, the entire PPL range is under 0.01. Even turbo3_tcq at 20.3% only adds about 0.02. turbo2 at 13.3% shows a visible hit, but even that is under 0.17 PPL absolute. If PPL were the whole story, the recommendation would be simple: compress aggressively and move on.

PPL averages over every token equally, so one position that destroys a JSON key or hallucinates a closing brace gets diluted by thousands of unremarkable tokens. The metric that picks up that tail damage is KL divergence against the bf16 baseline — specifically the 99.9% KLD, which measures the worst 0.1% of positions. The 99.9% precision column converts it to a percentage: 100 * exp(-(quantKLD - bf16KLD)).

The Q5_K_S 64k KL table shows what PPL hides. Mean precision stays above 99% for almost every mode, but the 99.9% column tells a different story:

Full KL tables for all three configurations are in §11.2.

The mean KLD column still looks reasonable: most modes stay below 0.01. But the 99.9% column diverges sharply. q5_0 at 34.4% of bf16 KV has a 99.9% KLD of 0.099, which is 42× its mean. q4_0 at 28.1% jumps to 0.130, which looks small as a number but is a 32% increase over q5_0 in the tail, and this will break your tool calls. Below q4_0, turbo modes fall off a cliff: turbo3_tcq at 20.3% of bf16 KV reaches 0.227 in the tail, and turbo2 at 13.3% hits 0.904, so roughly one full nat of divergence at the worst one-in-a-thousand positions.

The size column makes the comparison direct. q4_0 saves 71.9% of bf16 KV and keeps 89.8% of 99.9% precision. turbo3_tcq saves 79.7% and keeps 81.6%. turbo2 saves 86.7% and collapses to 41.5%. The PPL table suggested turbo3_tcq was fine; the tail says it is not, unless you genuinely need the memory and can accept the risk. And then turbo2 is only viable for tasks where exact structure does not matter.

3. Rotation Changed the Playing Field

The q4/q5/q8 modes in these results are not naive scalar quantization. llama.cpp applies a random rotation to KV cache vectors before quantizing them — the same basic trick TurboQuant uses. The rotation spreads outlier energy across coordinates, so the scalar quantizer faces a more uniform distribution and makes fewer catastrophic rounding decisions on extreme values. The difference is what happens after rotation: q4/q5 quantize each value independently to 4–5 bits with a scalar codebook, while TurboQuant quantizes to 2–3 bits with a scalar codebook and optionally adds a QJL residual for inner-product estimation. TCQ (§4) constrains index sequences through a trellis instead.

TurboQuant is also slower. turbo4 runs at ~705 tok/s (prefill) versus ~850 tok/s for q4_0 on the Q5_K_S config, a 17% throughput penalty. turbo3_tcq runs at ~794–844 tok/s across the three configs, slower than q4_0 in every case. The rotation and QJL stages add compute that scalar quantization avoids.

The rotation overlap means TurboQuant is not competing against a helpless baseline. It is competing against q4/q5 modes that already benefit from the same outlier smoothing. At 2–3 bits, the scalar codebook is starved enough that rotation alone cannot save you, and TurboQuant's extra structure matters. At 4 bits, turbo4 has no quality advantage over q4_0, saves almost no memory (25.8% vs 28.1% of bf16 KV), and runs slower. The value in TurboQuant is at the low bit rates where q4/q5 cannot reach at all.

4. What TCQ Adds

TCQ changes the low-bit side of the table. buun's paper ("Closing the Gap: Trellis-Coded Quantization for KV Cache at 2–3 Bits") describes TCQ as the first application of trellis-coded quantization to LLM KV cache compression. Instead of choosing each scalar index independently, TCQ constrains index sequences through a finite-state trellis, which gives a much larger effective codebook at the same bit rate. The encoder uses Viterbi to find a globally optimal assignment, while the decode path stays parallel: each value can be decoded in O(1), which is why the method can still fit GPU flash attention kernels.

The reported TCQ results claim 10–44% KL-divergence reduction over scalar quantization at 2–3 bits per value, with context-adaptive norm scaling and FWHT rotation plus random sign flips. That matches what this BeeLlama bench sees directionally: turbo3_tcq is consistently much better than plain turbo3, and turbo2_tcq is much better than plain turbo2, especially in the 99.9% tail.

There is no turbo4_tcq in this table, and that makes sense. TCQ is most valuable where independent scalar codes are starved for bits. At 2–3 bits, the larger effective codebook can close a visible gap. At 4 bits, the ordinary scalar codebook already has enough resolution, and the extra trellis decode overhead is not worth paying for marginal quality gains on top of a mode that is already uncompetitive with q4_0.

5. Asymmetric KV: Where the Bits Should Go

The asymmetric rows are the most useful part of the final report. They confirm the K-first intuition, but with a limit.

At 31.3% of bf16 KV, q5_0-q4_0 (K at q5_0, V at q4_0) has the same size as symmetric q4_1, yet it beats q4_1 across all three 99.9% precision tables: 91.4% vs 88.8% in Q5_K_S 64k, 96.1% vs 93.7% in IQ4_XS 64k, and 96.2% vs 95.0% in IQ4_XS 128k. Spending the same budget of bits on a stronger K and weaker V outperforms splitting them evenly.

q5_0-q4_0 is the clean trade at that size: same footprint as q4_1, but better tail behavior from spending bits asymmetrically toward K. One step up, q5_0-q4_1 costs 32.8% of bf16 KV, only 1.5 points more than q5_0-q4_0, and improves the 99.9% tail to 92.6%, 96.2%, and 96.5% across the three KL tables. That is not a huge jump, but it is cheap.

The direct comparison between q5_0-q4_1 and q5_1-q4_0 answers the next-bit question. Both are the same size. q5_0-q4_1 is better in Q5_K_S and IQ4_XS 128k, while IQ4_XS 64k is effectively tied in 99.9% precision. After K reaches q5_0, the next useful bit appears to go to V, not to q5_1 K.

The K-first ladder that falls out of these comparisons, for VRAM-constrained setups in particular:

q5_1 remains useful as an extra-conservative option, but it is not where the marginal value is.

6. TurboQuant's Narrowed Niche

The bench is harsh on turbo modes, but it does not invalidate them. I'd say it just clearly defines their niche, albeit one that is much more narrow than I expected.

turbo4 is the easiest mode to drop from recommendations. At IQ4_XS 128k, it saves only 192 MiB over q4_0, 2112 MiB vs 2304 MiB, while throughput drops from 710.51 tok/s to 519.76 tok/s and 99.9% precision falls from 94.3% to 93.1%. That is not a good exchange.

Plain turbo3 and turbo2 are also hard to recommend when TCQ variants are available. turbo3_tcq beats turbo3 clearly in the tail, and turbo2_tcq is much better than turbo2. Hardware support matters here: if a backend does not support the TCQ variants, users have to choose between paying more VRAM for q4/q5 or accepting the quality loss from the plain turbo modes. But that should be a documented fallback, not the main path.

The valuable turbo result is turbo3_tcq. It lands at 20.3% of bf16 KV, with 99.9% precision of 81.6%, 84.5%, and 86.1% across the three KL tables. That is not q4 quality, and it should not be sold as such. But it is a real precision/VRAM/speed compromise for users who need the context to fit.

turbo2_tcq is the last resort. It keeps broad PPL better than the tail suggests, but the 99.9% precision values are 54.4%, 58.6%, and 63.5%. That is a mode for rough summarization, long-context reading where exactness is not the job, or setups that otherwise cannot run. It is not a tool-call, JSON, code-edit, or math mode.

There is also a small middle rung: q4_0-turbo3_tcq. It uses 24.2% of bf16 KV and sits between q4_0 and symmetric turbo3_tcq in the 99.9% tail. It too is not a precision preset, but it's useful for users who need something smaller than q4 and cleaner than full turbo3_tcq.

7. The q8 Fidelity Tier

The added q8 asymmetric rows do not bring q8 back as the normal recommendation. They create a proper fidelity tier.

q8_0-q5_0 at 43.8% of bf16 KV keeps 99.9% precision at 93.7%, 98.2%, and 98.0% across the three KL tables. q8_0-q5_1 is a little better at 45.3%, but the gain is small: 94.2%, 98.3%, and 98.1%. Full q8_0/q8_0 sits at 53.1% and is mostly useful for validation and blame isolation.

q8_0-q4_1 is more interesting technically than practically. Its 99.9% precision is close to q5_1 in the table, which reinforces that K precision matters more than V. But it is bigger than q5_1, and its Q5_K_S speed is anomalous at 786.54 tok/s — likely an external issue rather than a real difference, since other asymmetric pairs at similar sizes and the IQ4_XS re-run both show normal throughput.

8. The Preset Ladder

The old recommendation set treated TurboQuant too broadly. The new one should separate ordinary precision modes, fidelity modes, and long-context survival modes.

The modes I would not present as first-class recommendations are q4_1/q4_1, q5_1/q4_1, q5_1/q4_0, q8_0/q4_0, q8_0/turbo3_tcq, q5_0/turbo3_tcq, q5_1/turbo3_tcq, q4_1/turbo3_tcq, q4_0/turbo2_tcq, plain turbo3, plain turbo2, and turbo4 unless its speed/implementation profile changes.

That is not a rejection of TurboQuant, but a better boundary. If q5 or q4 fits, use q5 or q4. If the job is strict and tool-heavy, prefer q5_0. If the job is memory-bound but still ordinary, use q5_0/q4_0 or q5_0/q4_1. If the job cannot fit without deeper KV compression, that is where turbo3_tcq earns its place. If even that does not fit, turbo2_tcq exists, but the user should know what they are giving up.

The benchmark does not show "Q beats TQ" or "TQ beats Q." It shows that the line between them has moved. Rotation made ordinary q4/q5 quite decent, TCQ makes very low-bit turbo modes much more viable than plain turbo2/turbo3, and the user wins beacuse there's a somewhat clear preset ladder with obvious trade-offs.

9. Weight Quant Changes the Cost of Cache Quant

The bench ran two weight quantizations: Q5_K_S and IQ4_XS. Same model, same context lengths where they overlap, same cache modes. The comparison is a controlled variable that most KV cache benchmarks do not isolate. It should be: weight precision changes how much cache quantization costs in the tail.

At 64k context, every symmetric cache mode lands 3–5% higher in 99.9% precision on IQ4_XS than on Q5_K_S:

IQ4_XS is consistently less damaged by the same cache quant. The gap exists across almost every mode and only collapses at turbo2, where both weight quants are already in the floor.

The raw KLD numbers show the same thing more sharply. Q5_K_S q8_0 has 99.9% KLD of 0.078709; IQ4_XS q8_0 has 0.017372. That is a 4.5× difference. At q4_0 the ratio drops to 1.7× (0.130419 vs 0.076663). The gap compresses as cache quant gets more aggressive, because the cache quantization noise starts to dominate over the weight quantization difference.

This looks backwards at first glance. Q5_K_S preserves more weight precision. The attention computation is more accurate. Shouldn't stricter weights make the model more robust to cache quantization, not less?

The answer is in what KL divergence measures here: the shift in output distribution when you change only the KV cache, holding everything else constant. Q5_K_S produces richer KV distributions. The attention scores carry more fine-grained detail because the weight matrices they multiply through are more precise. Those richer KV vectors have more tail-relevant information: structurally important tokens, sharp probability differences, outlier activations that carry real signal. When you quantize those KV values, you lose more because there is more to lose. The perturbation is larger relative to the information content of the uncompressed cache.

IQ4_XS already injects weight quantization noise into the attention path. The KV distributions it produces are smoother, carry less fine detail, and have shallower tails because the weight quantization already smoothed out some of the sharper features. Cache quantization on top of that does less incremental damage because there is less fragile detail left to damage.

This is not "loose weights + loose cache is better than strict weights + loose cache." The absolute output quality of Q5_K_S with bf16 KV is better than IQ4_XS with bf16 KV. But KL divergence is measured against each model's own bf16 baseline, not against an external ground truth. The metric captures incremental cost, not absolute quality. A Q5_K_S model at q4_0 cache has moved further from its own potential than an IQ4_XS model at q4_0 cache has moved from its (lower) potential.

The practical resolution: the same cache preset is tail-safer on a lower-weight-precision model than on a higher-weight-precision model. If you are running Q5_K_S, you should lean harder toward q5_0 over q4_0 than you would on IQ4_XS. The preset ladder in §8 does not account for this interaction, and it should: Q5_K_S users should treat balanced (q5_0/q4_0) as their floor, while IQ4_XS users can safely step down to memory (q4_0/q4_0) without incurring the same tail cost.

10. What Else Was Tried

Wikitext PPL and KL divergence are the only benchmarks that made it into this article, but several other tests were tried and abandoned because they could not distinguish between cache quantization modes.

Multiple-choice benchmarks (ARC-Challenge, ARC-Easy, HellaSwag, MMLU) all run at short context, well under 10K tokens. The KV cache is tiny at those lengths, and every cache mode scores within noise of every other mode. They measure whether the model knows things, not whether quantized cache degrades retrieval or reasoning. The spread across cache types was indistinguishable from run-to-run variance.

Perplexity-based HellaSwag and Winogrande have the same problem: they do not exercise the KV cache enough to show a difference. They confirm that the model still speaks English after cache quantization, which was never in doubt.

Synthetic passkey and needle-in-a-haystack retrieval were tried at 32K context. Every cache type, including turbo2, scored 100%. The model can regurgitate a hidden string just fine even with aggressively quantized cache, because retrieval of a single attended token is a different failure mode than slowly diverging output distributions over thousands of tokens. A 30-question passkey test does not have the statistical power to catch what KL divergence catches.

JSON schema generation (structured output constrained by a schema) also hit 100% for all cache types. Cache quantization does not break template compliance at the scale tested.

Native passkey via llama-passkey was excluded because it requires the model's native context length of 262144 tokens, which OOMs on 24 GB VRAM and has no --ctx-size override.

AIME (30 mathematics competition problems) is the only discarded benchmark that could show a difference. The problem is cost: on an RTX 3090, one pass through the 30 problems for a single cache configuration would take roughly 3–4 days of non-stop inference. I committed to it anyway and ran bf16, q8_0, and q4_0 before realizing that a single pass is nearly useless: 30 questions is wildly noisy for a binary measure of correctness, with q8_0 and q4_0 getting the same result. You need dozens of runs to average out the variance, and by the time that finishes, AGI will have already been invented and the whole topic of local inference optimization will be moot.

11. Benchmark Data

11.1 Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

11.2 KL Divergence

KL precision uses 100 * exp(-(quantKLD - bf16KLD)). The 99.9% precision column applies the same formula to 99.9% KLD. Symmetric and asymmetric cache configurations are combined in each table; asymmetric cache names are K-V.

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k