Home › Articles › KV Cache Quantization Benchmarks for Long Context

KV Cache Quantization Benchmarks for Long Context

Tests on Qwen 3.6 27B show why TurboQuant is overrated but saved by TCQ, q5 deserves more attention, and symmetric q8 might be a waste of VRAM.

1. The Setup
2. PPL Hides the Tail, KLD Exposes It
3. Rotation Changed the Playing Field
4. What TCQ Adds
5. Asymmetric KV: Where the Bits Should Go
6. TurboQuant's Narrowed Niche
7. The q8 Fidelity Tier
8. The Preset Ladder
9. Weight Quant Changes the Cost of Cache Quant
10. What Else Was Tried
11. Benchmark Data
    1. Perplexity
    2. KL Divergence
12. q6_0 Follow-Up Benchmarks
    1. Perplexity
    2. KL Divergence
13. q6_0 Follow-Up Analysis
    1. The q6 Slot Is q6 K + q5 V
    2. The q5 V Range Is Cheap, But Not Free
    3. turbo4 Still Loses the Tie-Breaker
    4. Updated Preset Ladder
14. bf16-K Follow-Up Benchmarks
15. bf16-K Follow-Up Analysis
    1. What The bf16-K Rows Test
    2. q8 V Does Not Earn Full K
    3. q6 V Is The High-End Ceiling
    4. q5 V Is The Practical Middle
    5. q4 V Is The Drop Full K Cannot Hide
    6. turbo V Stays Below The Normal Ladder
    7. What This Changes
16. bf16 vs f16 Follow-Up Benchmarks
    1. Perplexity
    2. KL Divergence
17. bf16 vs f16 Follow-Up Analysis
    1. Why The f32 Baseline Matters
    2. bf16 Beats f16 At The Same Size
    3. Precision Wins, Speed Is Mostly A Wash
    4. What This Changes

Disclaimer: this is just results of a few basic KV cache benchmarks, not the whole truth. Longer context should make the differences more pronounced, but that still doesn't guarantee 1:1 with real usage. Maybe TurboQuant magic works better with actually getting tool calls correctly at extra long context in agentic coding, or maybe it's the other way around. I tried some other tests too, but in terms of meaningful results PPL and KLD are all I've got.

1. The Setup

This benchmark started with a narrow question: does TurboQuant-style KV cache compression have any defensible niche in BeeLlama's (my llama.cpp fork) recommendation set, or is q4/q5 strictly better where it fits?

Hardware: one RTX 3090 (24 GB VRAM), Ryzen 7 5700X3D, 32 GB RAM. Model: Qwen 3.6 27B. Q5_K_S weights plus a 64k bf16 KV cache fit on the card; 128k does not, the weights alone fill too much VRAM. IQ4_XS weights are smaller, so IQ4_XS plus a 128k bf16 KV cache fits. Each context length is capped at the maximum where bf16 KV still runs, because bf16 is the reference every quantized mode is measured against. IQ4_XS at 64k is also included to compare directly against Q5_K_S at the same 64k context, isolating the effect of weight quant on cache quality (§9).

The tok/s column in the benchmark data tables is prefill throughput (batch size 2048, ubatch 256), not generation speed. GPU power was lowered during the runs. These numbers are useful only for relative comparison between cache modes on the same hardware, the absolute values do not represent real-world generation performance.

The tests ran on BeeLlama.cpp v0.1.2, which keeps the normal llama.cpp flow but adds DFlash plus TurboQuant/TCQ KV-cache compression. The TurboQuant implementation being evaluated comes from TheTom's llama-cpp-turboquant, while the TCQ variants come from buun's llama.cpp fork and the accompanying TCQ paper and codebooks.

2. PPL Hides the Tail, KLD Exposes It

The PPL numbers look flat across the board. Even turbo modes barely move the average:

Full PPL results for all cache modes and configurations are in §11.1.

q5_0 uses 34.4% of bf16 KV and saves 65% memory; q4_0 uses 28.1%. Through q4_0, the entire PPL range is under 0.01. Even turbo3_tcq at 20.3% only adds about 0.02. turbo2 at 13.3% shows a visible hit, but even that is under 0.17 PPL absolute. If PPL were the whole story, the recommendation would be simple: compress aggressively and move on.

PPL averages over every token equally, so one position that destroys a JSON key or hallucinates a closing brace gets diluted by thousands of unremarkable tokens. The metric that picks up that tail damage is KL divergence against the bf16 baseline — specifically the 99.9% KLD, which measures the worst 0.1% of positions. The 99.9% precision column converts it to a percentage: 100 * exp(-(quantKLD - bf16KLD)).

The Q5_K_S 64k KL table shows what PPL hides. Mean precision stays above 99% for almost every mode, but the 99.9% column tells a different story:

Full KL tables for all three configurations are in §11.2.

The mean KLD column still looks reasonable: most modes stay below 0.01. But the 99.9% column diverges sharply. q5_0 at 34.4% of bf16 KV has a 99.9% KLD of 0.099, which is 42× its mean. q4_0 at 28.1% jumps to 0.130, which looks small as a number but is a 32% increase over q5_0 in the tail, and this will break your tool calls. Below q4_0, turbo modes fall off a cliff: turbo3_tcq at 20.3% of bf16 KV reaches 0.227 in the tail, and turbo2 at 13.3% hits 0.904, so roughly one full nat of divergence at the worst one-in-a-thousand positions.

The size column makes the comparison direct. q4_0 saves 71.9% of bf16 KV and keeps 89.84% of 99.9% precision. turbo3_tcq saves 79.7% and keeps 81.56%. turbo2 saves 86.7% and collapses to 41.47%. The PPL table suggested turbo3_tcq was fine; the tail says it is not, unless you genuinely need the memory and can accept the risk. And then turbo2 is only viable for tasks where exact structure does not matter.

3. Rotation Changed the Playing Field

The q4/q5/q8 modes in these results are not naive scalar quantization. llama.cpp applies a random rotation to KV cache vectors before quantizing them — the same trick TurboQuant uses. The rotation spreads outlier energy across coordinates, so the scalar quantizer faces a more uniform distribution and makes fewer catastrophic rounding decisions on extreme values. The difference is what happens after rotation: q4/q5 quantize each value independently to 4–5 bits with a scalar codebook, while TurboQuant quantizes to 2–3 bits with a scalar codebook and optionally adds a QJL residual for inner-product estimation. TCQ (§4) constrains index sequences through a trellis instead.

TurboQuant is also slower. turbo4 runs at ~705 tok/s (prefill) versus ~850 tok/s for q4_0 on the Q5_K_S config, a 17% throughput penalty. turbo3_tcq runs at ~794–844 tok/s across the three configs, slower than q4_0 in every case. The rotation and QJL stages add compute that scalar quantization avoids.

The rotation overlap means TurboQuant is not competing against a helpless baseline. It is competing against q4/q5 modes that already benefit from the same outlier smoothing. At 2–3 bits, the scalar codebook is starved enough that rotation alone cannot save you, and TurboQuant's extra structure matters. At 4 bits, turbo4 has no quality advantage over q4_0, saves almost no memory (25.8% vs 28.1% of bf16 KV), and runs slower. The value in TurboQuant is at the low bit rates where q4/q5 cannot reach at all.

4. What TCQ Adds

TCQ changes the low-bit side of the table. buun's paper ("Closing the Gap: Trellis-Coded Quantization for KV Cache at 2–3 Bits") describes TCQ as the first application of trellis-coded quantization to LLM KV cache compression. Instead of choosing each scalar index independently, TCQ constrains index sequences through a finite-state trellis, which gives a much larger effective codebook at the same bit rate. The encoder uses Viterbi to find a globally optimal assignment, while the decode path stays parallel: each value can be decoded in O(1), which is why the method can still fit GPU flash attention kernels.

The reported TCQ results claim 10–44% KL-divergence reduction over scalar quantization at 2–3 bits per value, with context-adaptive norm scaling and FWHT rotation plus random sign flips. That matches what this BeeLlama bench sees directionally: turbo3_tcq is consistently much better than plain turbo3, and turbo2_tcq is much better than plain turbo2, especially in the 99.9% tail.

There is no turbo4_tcq in this table, and that makes sense. TCQ is most valuable where independent scalar codes are starved for bits. At 2–3 bits, the larger effective codebook can close a visible gap. At 4 bits, the ordinary scalar codebook already has enough resolution, and the extra trellis decode overhead is not worth paying for marginal quality gains on top of a mode that is already uncompetitive with q4_0.

5. Asymmetric KV: Where the Bits Should Go

The asymmetric rows are the most useful part of the final report. They confirm the K-first intuition, but with a limit.

At 31.3% of bf16 KV, q5_0-q4_0 has the same size as symmetric q4_1, yet it beats q4_1 across all three 99.9% precision tables: 91.39% vs 88.82% in Q5_K_S 64k, 96.06% vs 93.73% in IQ4_XS 64k, and 96.18% vs 95.04% in IQ4_XS 128k. Spending the same budget of bits on a stronger K and weaker V outperforms splitting them evenly.

q5_0-q4_0 is the clean trade at that size: same footprint as q4_1, but better tail behavior from spending bits asymmetrically toward K. One step up, q5_0-q4_1 costs 32.8% of bf16 KV, only 1.5 points more than q5_0-q4_0, and improves the 99.9% tail to 92.65%, 96.21%, and 96.50% across the three KL tables. Not a huge jump, but it is cheap.

The direct comparison between q5_0-q4_1 and q5_1-q4_0 answers the next-bit question. Both are the same size. q5_0-q4_1 is better in Q5_K_S and IQ4_XS 128k, while IQ4_XS 64k is effectively tied in 99.9% precision. After K reaches q5_0, the next useful bit appears to go to V, not to q5_1 K.

q5_1 remains useful as an extra-conservative option, but it is not where the marginal value is.

6. TurboQuant's Narrowed Niche

The bench is harsh on turbo modes, but it does not invalidate them. I'd say it just clearly defines their niche, albeit one that is much more narrow than I expected.

turbo4 is the easiest mode to drop from recommendations. At IQ4_XS 128k, it saves only 192 MiB over q4_0, 2112 MiB vs 2304 MiB, while throughput drops from 710.51 tok/s to 519.76 tok/s and 99.9% precision falls from 94.34% to 93.07%. That is not a good exchange.

Plain turbo3 and turbo2 are also hard to recommend when TCQ variants are available. turbo3_tcq beats turbo3 clearly in the tail, and turbo2_tcq is much better than turbo2. Hardware support matters here: if a backend does not support the TCQ variants, users have to choose between paying more VRAM for q4/q5 or accepting the quality loss from the plain turbo modes. But that should be a documented fallback, not the main path.

The valuable turbo result is turbo3_tcq. It lands at 20.3% of bf16 KV, with 99.9% precision of 81.56%, 84.51%, and 86.06% across the three KL tables. That is not q4 quality, and it should not be sold as such. But it is a real precision/VRAM/speed compromise for users who need the context to fit.

turbo2_tcq is the last resort. It keeps broad PPL better than the tail suggests, but the 99.9% precision values are 54.38%, 58.62%, and 63.53%. That is a mode for rough summarization, long-context reading where exactness is not the job, or setups that otherwise cannot run. It is not a tool-call, JSON, code-edit, or math mode.

There is also a small middle rung: q4_0-turbo3_tcq. It uses 24.2% of bf16 KV and sits between q4_0 and symmetric turbo3_tcq in the 99.9% tail. It too is not a precision preset, but it's useful for users who need something smaller than q4 and cleaner than full turbo3_tcq.

7. The q8 Fidelity Tier

Symmetric q8_0 at 53.1% of bf16 KV sits at the top of the size range, and its numbers show it. Mean precision is 99.80% on Q5_K_S, 99.95% on IQ4_XS, and PPL precision actually exceeds bf16 (100.05% on Q5_K_S, 100.02% on IQ4_XS): the quantization noise is small enough that median perplexity changes by less than run-to-run variance. The 99.9% tail tells a slightly different story: 94.61% on Q5_K_S, 98.69% on IQ4_XS 64k, 98.52% on 128k.

Those are the highest tail numbers in the bench, but they are not perfect, and the Q5_K_S gap from 100% is visible. Combined with VRAM costs, this makes full q8_0-q8_0 a validation and blame-isolation mode, not a practical default.

The asymmetric pairs with q8_0 K walk the V down one tier at a time, and each step has a cost that mirrors the V-tier structure the rest of the bench already found. On Q5_K_S 64k:

• Dropping V from q8_0 to q5_1: 94.61% to 94.21%, a 0.40-point loss for a 7.8% size reduction. At 45.3% of bf16 KV, q8_0-q5_1 still clears 94% tail precision.
• Dropping V from q5_1 to q5_0: 94.21% to 93.69%, 0.52 points for only 1.5% less size. Stepping inside the q5 V range costs roughly half a point per tier, the same "nearly free but not zero" pattern from §5.
• Dropping V from q5_0 to q4_1: 93.69% to 92.70%, 0.99 points. This is the q5-to-q4 V cliff: the full-point penalty that shows up everywhere the V crosses below q5.
• Dropping V from q4_1 to q4_0: 92.70% to 92.18%, 0.52 points. Same half-point step as the q5 internal tiers.
• Dropping V from q4_0 to turbo3_tcq: 92.18% to 88.15%, a 4.03-point collapse. Even q8_0 K cannot rescue turbo3_tcq V, it falls below symmetric q4_0 (89.84%) despite costing more memory.

IQ4_XS shifts the absolute numbers but preserves the step structure. V steps inside q5 cost 0.09–0.14 points, the q5-to-q4 cliff costs 0.94–1.20 points, and the q4_0-to-turbo3_tcq drop is 4.30 points on both 64k and 128k. The same V-tier boundaries that the symmetric and q5-asymmetric modes already showed also govern the q8-tier walkdown.

That walkdown makes the fidelity tier's practical boundaries clear. At q8_0 K, V values inside q5 cost less than half a point per tier and stay above 93.6% tail precision; crossing below q5 V costs a full point. The two useful presets that balance VRAM and precision well are q8_0-q5_1 and q8_0-q5_0.

q8_0-q5_1 at 45.3% of bf16 KV is the best precision-per-size trade at the high end. It gives up only 0.40 points of 99.9% tail precision versus symmetric q8_0 (94.21% vs 94.61%) while shrinking the cache by 7.8 percentage points. On IQ4_XS, the loss is just 0.41 points on 64k (98.28% vs 98.69%) and 0.39 on 128k. Mean precision stays above 99.78% on both weight quants. That is the fidelity preset: the smallest q8-tier config that stays above 94% tail precision, and the one that earns the bold row in §8.

q8_0-q5_0 at 43.8% saves another 1.5% of bf16 KV but costs 0.52 more tail points (93.69% vs 94.21%). It exists as a fallback if q8_0-q5_1 does not fit by a narrow margin, not as a peer recommendation. Where dropping V from q8_0 to q5_1 costs 0.40 points for a 7.8% size reduction, dropping from q5_1 to q5_0 costs 0.52 points for only 1.5%.

Below q5 V, the returns collapse. q8_0-q4_1 at 42.2% loses 0.99 points from q8_0-q5_0 for 1.6% less size. q8_0-q4_0 at 40.6% loses another 0.52 points. Neither beats symmetric q5_0 (34.4%, 92.70%) in the tail despite being 6–8% larger, because the q5-to-q4 V cliff erases the q8_0 K advantage. The comparison with symmetric q5_1 makes this concrete: q8_0-q4_1 is 4.7% larger yet nearly tied with q5_1 at 92.70% vs 92.77%.

The q8_0 K tier buys back nearly all of the q4_1 V penalty, confirming that K precision dominates the tail, but the net result is no better than a fully symmetric pair that costs less. The same pattern holds on IQ4_XS (q8_0-q4_1 at 96.99% is 0.47 points below q5_1 at 97.46% on 64k, 0.65 on 128k). q8_0-q4_1 is more interesting technically than practically (its Q5_K_S speed of 786.54 tok/s is anomalous; other asymmetric pairs at similar sizes and the IQ4_XS re-run both show normal throughput), as it clearly shows the limits of K-vs-V trade-offs.

8. The Preset Ladder

The old recommendation set treated TurboQuant too broadly. The new one should separate ordinary precision modes, fidelity modes, and long-context survival modes.

The modes I would not present as first-class recommendations are q4_1/q4_1, q5_1/q4_1, q5_1/q4_0, q8_0/q4_0, q8_0/turbo3_tcq, q5_0/turbo3_tcq, q5_1/turbo3_tcq, q4_1/turbo3_tcq, q4_0/turbo2_tcq, plain turbo3, plain turbo2, and turbo4 unless its speed/implementation profile changes.

That is not a rejection of TurboQuant, but a better boundary. If q5 or q4 fits, use q5 or q4. If the job is strict and tool-heavy, prefer q5_0. If the job is memory-bound but still ordinary, use q5_0/q4_0 or q5_0/q4_1. If the job cannot fit without deeper KV compression, that is where turbo3_tcq earns its place. If even that does not fit, turbo2_tcq exists, but the user should know what they are giving up.

The benchmark does not show "Q beats TQ" or "TQ beats Q", but that the line between them has moved. Rotation made ordinary q4/q5 quite decent, TCQ makes very low-bit turbo modes much more viable than plain turbo2/turbo3, and the user wins because there's a somewhat clear preset ladder with obvious trade-offs.

9. Weight Quant Changes the Cost of Cache Quant

The bench ran two weight quantizations: Q5_K_S and IQ4_XS. Same model, same context lengths where they overlap, same cache modes. The comparison is a controlled variable that most KV cache benchmarks do not isolate. It should be: weight precision changes how much cache quantization costs in the tail.

At 64k context, most symmetric cache modes land 3–5% higher in 99.9% precision on IQ4_XS than on Q5_K_S (turbo3_tcq is just under 3%, and turbo2 has already collapsed on both):

IQ4_XS is consistently less damaged by the same cache quant. The gap exists across almost every mode and only collapses at turbo2, where both weight quants are already in the floor.

The raw KLD numbers make the gap sharper. Q5_K_S q8_0 has a 99.9% KLD of 0.078709; IQ4_XS q8_0 has 0.017372. Same cache mode, 4.5× the tail damage on the higher-precision weights. At q4_0 the ratio drops to 1.7× (0.130419 vs 0.076663): the cache quantization noise starts to dominate over the weight quantization the deeper you compress.

The reason is in what those KV values carry. Q5_K_S produces richer KV distributions: structurally important tokens, sharp probability differences, outlier activations with real signal. IQ4_XS already injects weight noise into the attention path, so its KV distributions are smoother and carry less fine detail.

When you quantize those KV values, you lose more from Q5_K_S because there is more to lose. The absolute output quality of Q5_K_S with bf16 KV is still better, but KL divergence is measured against each model's own bf16 baseline, so a Q5_K_S model at q4_0 cache has moved further from its own potential than an IQ4_XS model at q4_0 has from its (lower) potential.

The practical resolution: the same cache preset is tail-safer on a lower-weight-precision model than on a higher-weight-precision model. If you are running Q5_K_S, you should lean harder toward q5_0 over q4_0 than you would on IQ4_XS.

10. What Else Was Tried

Wikitext PPL and KL divergence are the only benchmarks that made it into this article, but several other tests were tried and abandoned because they could not distinguish between cache quantization modes.

Multiple-choice benchmarks (ARC-Challenge, ARC-Easy, HellaSwag, MMLU) all run at short context, well under 10K tokens. The KV cache is tiny at those lengths, and every cache mode scores within noise of every other mode. They measure whether the model knows things, not whether quantized cache degrades retrieval or reasoning. The spread across cache types was indistinguishable from run-to-run variance.

Perplexity-based HellaSwag and Winogrande have the same problem: they do not exercise the KV cache enough to show a difference. They confirm that the model still speaks English after cache quantization, which was never in doubt.

Synthetic passkey and needle-in-a-haystack retrieval were tried at 32K context. Every cache type, including turbo2, scored 100%. The model can regurgitate a hidden string just fine even with aggressively quantized cache, because retrieval of a single attended token is a different failure mode than slowly diverging output distributions over thousands of tokens. A 30-question passkey test does not have the statistical power to catch what KL divergence catches.

JSON schema generation (structured output constrained by a schema) also hit 100% for all cache types. Cache quantization does not break template compliance at the scale tested.

Native passkey via llama-passkey was excluded because it requires the model's native context length of 262144 tokens, which OOMs on 24 GB VRAM and has no --ctx-size override.

AIME (30 mathematics competition problems) is the only discarded benchmark that could show a difference. The problem is cost: on an RTX 3090, one pass through the 30 problems for a single cache configuration would take roughly 3–4 days of non-stop inference. I committed to it anyway and ran bf16, q8_0, and q4_0 before realizing that a single pass is nearly useless: 30 questions is wildly noisy for a binary measure of correctness, with q8_0 and q4_0 getting the same result. You need dozens of runs to average out the variance, and by the time that finishes, AGI will have already been invented and the whole topic of local inference optimization will be moot.

11. Benchmark Data

11.1 Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

11.2 KL Divergence

KL precision uses 100 * exp(-(quantKLD - bf16KLD)). The 99.9% precision column applies the same formula to 99.9% KLD. Symmetric and asymmetric cache configurations are combined in each table; asymmetric cache names are K-V.

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

12. q6_0 Follow-Up Benchmarks

Performed using pre-release v0.3.0 of BeeLlama that added the q6_0 cache mode. New modes tested: symmetric q6_0, asymmetric q6_0 pairs with q5_1, q5_0, q4_1, q4_0, turbo4, and turbo3_tcq V; plus q8_0-q6_0, q8_0-turbo4, and q5_0-turbo4. The KL tables also include re-runs of existing q8_0, q5_0, q4_0, and turbo3_tcq as a control group to verify stability after a build change. All control results matched within noise.

12.1 Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

12.2 KL Divergence

Same format as §11.2. Sorted by KV cache size descending. Includes re-run control rows for staple symmetric pairs (q8_0, q5_0, q4_0, turbo3_tcq) alongside the new entries.

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

13. q6_0 Follow-Up Analysis

The q6_0 follow-up does not change the shape of the KV-cache ladder. It changes two rungs: q8_0 / q6_0 becomes the better high-end preset, and q6_0 / q5_0 gets added as an optional step between normal q5 and q8 K. Everything else in the q6 sweep is mostly useful as a guardrail: it shows where the extra K bits stop helping, where V gets too weak, and why turbo4 still stays out of the normal recommendations.

13.1 The q6 Slot Is q6 K + q5 V

Symmetric q6_0 is a useful measurement row, but it is not the row the ladder wants. On Q5_K_S 64k, it scores 93.47% at the 99.9% tail, only 0.31 points above the same-run q5_0 control at 93.16%. It costs 40.6% of bf16 KV, so the extra size does not buy much on that run.

The better trade is q6_0 / q5_0. It keeps most of the q6 K benefit, drops the footprint to 37.5%, and is the first q6_0 row that makes sense as an actual preset. But on Q5_K_S the gain over q5_0 / q5_0 is almost a tie: 93.29% vs 93.16% at the 99.9% tail, with mean KLD also nearly tied, 0.002820 vs 0.002826. From this data the benefits are not obvious.

The reason to keep it is the lower-weight-precision run. On IQ4_XS, q6_0 / q5_0 opens a clearer gap over q5_0 / q5_0: 98.08% vs 97.22% at 64k, and 97.81% vs 97.35% at 128k. This does not prove a full mechanism for why the gap changes, but it is enough for a recommendation: if the weights are already compressed and you want more KV headroom than q5 gives, q6 K is the next useful spend.

13.2 The q5 V Range Is Cheap, But Not Free

The easiest trap in this follow-up is the Q5_K_S tail wobble around q5 V. Symmetric q6_0 scores 93.47%, q6_0 / q5_1 scores 93.50%, and q6_0 / q5_0 scores 93.29%. The 0.03-point "gain" from dropping V to q5_1 is not a result to build a recommendation around. Mean KLD moves the other way: 0.002614 for symmetric q6_0, 0.002781 with q5_1 V, and 0.002820 with q5_0 V.

IQ4_XS makes it less ambiguous. At 64k, the 99.9% tail goes 98.39% → 98.23% → 98.08% as V moves from q6_0 to q5_1 to q5_0. At 128k, it goes 98.11% → 97.85% → 97.81%. So the q5 V range is still a controlled loss. It just happens to be small enough that one Q5 tail column can wobble by more than the real difference between adjacent q5 V tiers.

The useful boundary is below q5. q6_0 / q4_1 drops to 92.19% on Q5_K_S, 1.10 points below q6_0 / q5_0. On IQ4_XS, the same step costs 1.37 points at 64k and 0.86 points at 128k. q6_0 / q4_0 is even easier to reject: it has the same 34.4% footprint as q5_0 / q5_0, but loses on all three follow-up configs.

13.3 turbo4 Still Loses the Tie-Breaker

turbo4 is still not a normal V tier. The confusing row is q6_0 / turbo4: on Q5_K_S, it scores 91.93% at the 99.9% tail, slightly ahead of q6_0 / q4_0 at 91.55%. If that were the only column, it would look like turbo4 V had beaten q4_0 V.

The rest of the row rejects that reading. Mean KLD is worse with turbo4, 0.003748 vs 0.003288, and the IQ4_XS runs move in the same direction at the tail: 96.02% vs 96.52% at 64k, then 96.18% vs 96.87% at 128k. This is the kind of case where the 99.9% tail is still the right warning metric, but mean KLD stops one lucky tail from becoming a recommendation.

The q8 and q5 turbo4 pairs tell the same story. q8_0 / turbo4 gets a tiny Q5_K_S tail edge over q8_0 / q4_0 from the older table, 92.33% vs 92.18%, but loses on IQ4_XS. q5_0 / turbo4 is smaller than q5_0 / q4_0, but the q4 V row is the better recommendation whenever that extra 1.2% of bf16 KV fits.

turbo3_tcq V is below the precision floor for these mixed presets. At q6_0 K, q6_0 / turbo3_tcq lands at 87.68% on Q5_K_S, below the same-run symmetric q4_0 control at 88.87%. That does not remove symmetric turbo3_tcq from the extreme-compression ladder, but it does mean turbo V should not be mixed into ordinary q5/q6/q8 precision rows.

13.4 Updated Preset Ladder

The preset ladder only needs two edits. At the top, q8_0 / q6_0 replaces q8_0 / q5_1. The Q5_K_S gain is small, 94.33% vs 94.21% at the 99.9% tail, but the size cost is also small: 46.9% vs 45.3% of bf16 KV. At this tier, paying 1.6 points of KV size for the stronger V side is cleaner than keeping q5_1 as the main fidelity preset.

In the middle, q6_0 / q5_0 is added as an optional headroom row. It is not a new default. If q5_0 / q5_0 already gives enough quality, stay there. Use q6_0 / q5_0 when you want a stronger K side and cannot justify the jump to q8_0 K.

The rejected q6 rows are not close calls. q6_0 / q5_1 saves too little compared with symmetric q6 and is larger than the useful q6_0 / q5_0 row. q6_0 / q4_1 crosses the V cliff. q6_0 / q4_0 has the same size as q5_0 / q5_0 and loses to it. q6_0 / turbo4 has one attractive Q5_K_S tail ordering, but mean KLD and both IQ4_XS runs reject it. q6_0 / turbo3_tcq belongs below the precision floor, not in the normal ladder.

The preset rule is simple: use q8_0 / q6_0 when you want the high-end cache tier, use q6_0 / q5_0 only when q5_0 / q5_0 feels too tight, and do not chase sub-q5 V just because K is stronger. The q6 follow-up adds one useful middle rung, not a new philosophy for the whole ladder.

14. bf16-K Follow-Up Benchmarks

This follow-up keeps K at bf16 and walks V through the same cache tiers used elsewhere in the article. The point is narrow: these rows isolate how much damage comes from V quantization when the K side is left at the reference format.

Same format as §11.2. KL precision and 99.9% precision are measured against the bf16 / bf16 baseline for each weight/context configuration. Rows are sorted by total KV cache size descending.

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

15. bf16-K Follow-Up Analysis

15.1 What The bf16-K Rows Test

The useful part of this follow-up is not a new preset ladder, but a V-side isolation check. These pairs keep K at the reference format and make V do all the compression work, so they answer a cleaner question: once attention scores have the strongest K side available in this benchmark, where does V start to become the limiting error?

That also makes these rows awkward as recommendations. A bf16 K side is expensive: at 64k it already takes 2048 MiB before V is counted. Even the smallest row in this set, bf16 / turbo3_tcq, still uses 60.2% of the full bf16 cache. So the entire point is to see which V tiers still matter after K has stopped being the bottleneck.

15.2 q8 V Does Not Earn Full K

bf16 / q8_0 is the cleanest non-recommendation in the table. On Q5_K_S 64k, it scores 94.50% at the 99.9% tail, which is the same broad quality band as symmetric q8_0 / q8_0 at 94.62% and q8_0 / q6_0 at 94.33%. The size is the problem: bf16 / q8_0 uses 76.6% of the full cache, while q8_0 / q8_0 uses 53.1% and q8_0 / q6_0 uses 46.9%.

The IQ4_XS rows do not rescue the trade. bf16 / q8_0 scores 98.81% at 64k and 98.52% at 128k, but it still costs 76.6% of the full cache. If q8 K is already enough to reach the high-end tail band, keeping K at 16 bits mostly buys comfort for the benchmarker, not a useful user-facing preset.

15.3 q6 V Is The High-End Ceiling

The q6 row shows where the V side stops needing more precision. On Q5_K_S, bf16 / q6_0 scores 94.59%, effectively tied with bf16 / q8_0 at 94.50%. On IQ4_XS, dropping V from q8 to q6 costs 0.09 points at 64k and 0.19 points at 128k. That is small enough to treat q6 V as the ceiling for the practical high-end row.

The comparison that matters is bf16 / q6_0 against q8_0 / q6_0. The full-K row is 70.3% of bf16 KV, while the q8-K row is 46.9%. On Q5_K_S, that extra 23.4 points of cache size buys 0.26 points of tail precision. On IQ4_XS, it buys 0.16 points at 64k and 0.05 at 128k. That is a validation margin, not a preset margin.

15.4 q5 V Is The Practical Middle

The q5 rows are where V quantization becomes visible without falling out of the normal quality range. On Q5_K_S, bf16 / q5_0 scores 93.57%, one point below bf16 / q6_0. On IQ4_XS, the same drop is smaller: 98.19% at 64k and 97.93% at 128k. That repeats the earlier pattern from §9 and §13: lower-weight-precision runs are less sensitive to same quants.

The full-K q5 rows are still too large to recommend directly. bf16 / q5_0 uses 67.2% of the cache, while q6_0 / q5_0 uses 37.5% and lands in the same practical band. The q5 V tier is useful, but with q6 or q5 K, not 16-bit K.

15.5 q4 V Is The Drop Full K Cannot Hide

q4 V shows the limit of the full-K diagnostic. On Q5_K_S, bf16 / q4_0 scores 91.68% and bf16 / q4_1 scores 92.07%. Those rows are larger than every normal q8/q6/q5 preset in the ladder, but they still sit below the q5 V range. Once V falls to q4, reference-format K can soften the loss but cannot make the row high end.

The IQ4_XS rows are less harsh but keep the same boundary. At 128k, bf16 / q5_0 is 97.93%, bf16 / q4_1 is 96.99%, and bf16 / q4_0 is 96.58%. That is not a collapse, but it is a clear step down at the same q5-to-q4 V boundary that showed up in the earlier asymmetric tables.

15.6 turbo V Stays Below The Normal Ladder

turbo4 looks better with bf16 K than it looked as a symmetric preset, but that is not enough to make it a normal recommendation. On Q5_K_S, bf16 / turbo4 scores 92.52%, above bf16 / q4_0 and bf16 / q4_1. The price is still 62.9% of the full cache, and the earlier sections already showed why turbo4 loses once other factors are included.

bf16 / turbo3_tcq is clearer. It scores 88.43% on Q5_K_S, 92.28% on IQ4_XS 64k, and 91.75% on IQ4_XS 128k. Full K helps compared with fully compressed turbo rows, but it does not turn turbo V into a q5/q6-quality tier. Turbo modes still belong in the extreme-compression part of the ladder, not in the ordinary precision presets.

15.7 What This Changes

This follow-up does not add any bf16 / V presets and stays exclusively diagnostic. Holding K at bf16 shows the V-side ceiling, but it also makes the rows too large to compete with the q8/q6/q5 ladder.

V has a ceiling around q6, q5 is the practical middle, and below q5 the K side cannot fully bail the row out. That strengthens the section 13 ladder: spend the high-end budget on q8_0 / q6_0, not on bf16 / q8_0 or bf16 / q6_0.

16. bf16 vs f16 Follow-Up Benchmarks

This follow-up runs a narrow Q5_K_S 32k Wikitext path using f32 KL baseline. The point is to answer the smaller question left open by the earlier bf16-baseline tables: which 16-bit KV format between f16 and bf16 stays closer to f32?

The PPL rows compare symmetric f32, f16, and bf16 cache. The KL rows compare those same symmetric rows plus two diagnostic f32-K rows, f32 / f16 and f32 / bf16, which isolate the V side while keeping K at the reference format. Sizes are for total K+V cache at 32k context.

16.1 Perplexity

Q5_K_S 32k

16.2 KL Divergence

Same idea as §11.2, but measured against a f32 baseline instead of a bf16 baseline. Since the reference row has mean KLD of zero, precision vs f32 is 100 * exp(-meanKLD). Rows are sorted by total KV cache size descending.

Q5_K_S 32k

17. bf16 vs f16 Follow-Up Analysis

17.1 Why The f32 Baseline Matters

The earlier tables used bf16 as the KL reference because it was the practical uncompressed cache format. That was fine for judging q8, q6, q5, q4, and turbo rows, but it made the f16 vs bf16 question awkward: if bf16 is the baseline, then bf16 automatically gets the cleanest comparison. The f32 follow-up removes that tilt: both 16-bit formats are measured as deviations from the same f32 logit file.

17.2 bf16 Beats f16 At The Same Size

The practical symmetric comparison is direct. bf16 / bf16 and f16 / f16 both use 2048 MiB at 32k context, exactly half of the f32 / f32 cache. On PPL, bf16 lands at 6.2147 while f16 lands at 6.2384. On KL, bf16 has lower mean KLD, 0.013964 vs 0.014504, and a much better 99.9% tail, 1.798395 vs 2.314729.

The V-side isolation rows point the same way. With K held at f32, f32 / bf16 gets mean KLD 0.012111 and 99.9% KLD 1.544926. f32 / f16 gets mean KLD 0.015022 and 99.9% KLD 2.224951. That rules out a softer reading where symmetric bf16 merely got lucky as a whole-cache row, as on this benchmark bf16 V is also cleaner than f16 V.

17.3 Precision Wins, Speed Is Mostly A Wash

There is no useful VRAM trade-off between f16 and bf16. Both are 16-bit cache formats, both use the same 2048 MiB at 32k, and both cut the f32 cache in half. If the hardware supports bf16 properly, the precision result is the deciding factor.

The elapsed times do not give f16 a strong counterargument. In the PPL run, bf16 / bf16 finished in 376.55 seconds and f16 / f16 in 381.02 seconds. In the KL run, bf16 / bf16 took 380.41 seconds and f16 / f16 took 374.74 seconds. The mixed rows show the same small wobble: f32 / bf16 took 395.52 seconds and f32 / f16 took 389.81 seconds. That is not enough to justify taking the worse KLD row as a speed preset.

17.4 What This Changes

This resolves the practical f16 vs bf16 KV-cache question for this benchmark: bf16 is the better 16-bit default. It has the same memory footprint as f16, no clear speed penalty in these runs, and lower divergence from the f32 baseline.