Home › Articles › KV Cache Quantization Benchmarks: KVarN, Precision Tail

KV Cache Quantization Benchmarks: KVarN, Precision Tail

413 cache configurations benchmarked with KLD across Qwen 3.6 27B and Gemma 4 31B, turned into a recommendation ladder from high end to extreme compression.

1. 1. What Is On The Table
2. 2. Bit Width Is The Main Lever
3. 3. KVarN Moves The Ladder
4. 4. Where The Bits Go Between K And V
5. 5. What An Exact Tail Buys
6. 6. Choosing For Qwen
7. 7. Gemma Is A Different Decision
8. 8. The Recommendation Ladder
   1. 8.1 Qwen
   2. 8.2 Qwen Standard-Only
   3. 8.3 Gemma
9. 9. What This Does Not Prove
10. 10. Method And Full Benchmark Data
    1. 10.1 Qwen 3.6 27B Q5_K_S 64k
    2. 10.2 Gemma 4 31B Q5_K_S 16k

1. What Is On The Table

The complete declared matrix contains 238 Qwen rows and 175 Gemma rows, all tested on BeeLlama v0.4.0. It crosses 48 standard K/V pairs and 15 KVarN pairs with a precision tail, a configurable run of the most recent tokens kept exact while the older body stays quantized, plus one BF16 baseline per model.

KVarN quantizes in groups of 128 tokens, and BeeLlama floors its tail at one full group, so a raw tail request of 0 resolves to an effective 128-token exact suffix. The tables display that floored request as 128*, and KVarN has no tail 0 rows because that configuration cannot exist. That fixes the comparison structure used throughout: standard tail 0 is a diagnostic control, and standard tail 128 versus KVarN 128* is the matched-suffix comparison the format claims rest on.

Qwen 3.6 27B Q5_K_S is measured at 65536 context. Its standard pairs use tail 0, 128, 1024, and 2048; KVarN uses 128*, 1024, and 2048. Gemma 4 31B Q5_K_S is measured at 16384 context. Its standard pairs use 0, 128, and 1024; KVarN uses 128* and 1024, which already covers the window visible to its sliding-window layers.

The comparison is broader than symmetric presets, as it also includes near-balanced asymmetric steps available in the default BeeLlama build, up to two bits of difference between K and V. That makes it possible to test whether a saved V bit creates a useful intermediate step or simply weakens the cache.

2. Bit Width Is The Main Lever

Across both models, moving the body out of the lowest bit widths changes KLD far more than adjusting a tail at the top of the ladder. On Qwen at tail 0, symmetric standard q2_0 records 0.019374 median KLD, q4_0 0.001846, and q8_0 0.000909. A longer exact suffix can rescue part of a weak body, but it does not help preserve the older context.

The _1 variants and asymmetric pairs fill memory gaps, yet they do not overturn that ordering. The first decision is how credible the whole retained body must be. Only then is it useful to decide whether the next memory increment belongs to V precision, K precision, or an exact recent window.

3. KVarN Moves The Ladder

KVarN rotates and normalizes each 128-token group before quantizing it; the implementation article covers the method and where it comes from. On Qwen, every tested symmetric KVarN width records lower median KLD than its same-width standard counterpart at the matched 128-token suffix, by between 2.6% at six bits and 12.6% at three bits. kvarn4 records 0.001111 in 1184.00 MiB against 0.001206 in 1160.00 MiB for q4_0: cleaner, and 24.00 MiB larger, because KVarN's always-resident F16 group costs 48.00 MiB where a standard BF16 tail 128 overlay costs 8.00 MiB.

Each KVarN width lands part way up the standard ladder rather than replacing a full tier. kvarn4 lands between q4_0 and q5_0. kvarn5 comes within 0.000014 of q6_0, 0.000927 against 0.000913 median KLD, while using 232.00 MiB less. kvarn6 edges past q8_0, 0.000889 against 0.000901, while using 488.00 MiB less.

4. Where The Bits Go Between K And V

One-bit asymmetry is usually the useful compromise. On Qwen, kvarn5-kvarn4 at 128* uses 1312.00 MiB and records 0.000995 median KLD, placing it between symmetric kvarn4 and kvarn5. The same pattern appears in standard pairs such as q5_0-q4_0. These rows save memory while keeping the precision closer to what the K cache tier would give.

Two-bit gaps are less convincing, though the matrix offers only one matched-memory check. At tail 128*, kvarn4-kvarn2 uses the same 928.00 MiB as symmetric kvarn3 but records 0.003570 median KLD instead of 0.001797. On that evidence, balance is the default and a one-bit V reduction is the normal intermediate step.

5. What An Exact Tail Buys

The tail's return depends on the body. Qwen standard q2_0 falls from 0.019374 at tail 0 to 0.004648 at 1024, while q8_0 moves from 0.000909 to 0.000897. The second thousand exact tokens reduces q2_0 further to 0.003696, but q8_0 changes by two millionths. KVarN follows the same curve from a stronger starting point, so its tail contribution is smaller still, though even the widest standard body remains a measurable step below BF16.

Treat the short-tail rows as synthetic readings. Teacher-forced Wikitext scores the next token at every position, so the newest tokens are always in play as decisive context, and even a 128-token exact suffix is rewarded far beyond what most real work would give it. In an agent session the tokens that matter (an earlier instruction, a function signature, a tool result) routinely sit hundreds or thousands of positions back.

The practical tail choices therefore start at 1024 and 2048; tail 128 stays in the matrix as the matched control that keeps standard and KVarN comparable, not as a recommended setting. The same reward shapes the Pareto frontier: of the 28 Qwen rows that no other row beats on both memory and median KLD, 21 carry a 1024- or 2048-token tail.

A short tail also changes what a comparison means. Standard tail 0 remains useful as a deployable low-memory row and as a diagnostic control, but it is not the matched comparator for KVarN, whose intrinsic newest group is exact. Even the matched comparison, standard 128 against KVarN 128*, mixes two remaining differences beyond the formats themselves: standard tails use BF16 in these runs while KVarN tails use F16, and non-sliding-window KVarN layers carry a permanent exact prefix sink. Section 9 weighs those caveats.

6. Choosing For Qwen

The balanced Qwen starting point is kvarn5-kvarn4 with tail 1024. It records 0.000936 median KLD and 0.089469 at P99.9 in 1360.00 MiB: median within 3% of q8_0 tail 0 (0.000909, with 0.093029 at P99.9) for 816.00 MiB less persistent memory. The tradeoff is a slower prefill path.

kvarn6-kvarn5 tail 1024 is the more conservative body, at 0.000886 and 1616.00 MiB; kvarn4-kvarn3 tail 1024 is the compact step at 0.001112 and 1104.00 MiB. kvarn3-kvarn2 tail 1024 reaches 848.00 MiB, but its 0.002424 median and much heavier upper tail make it emergency compression rather than a coding default.

Standard pairs are practical alternatives. They process prompts faster and have a simpler portability. At tail 1024, standard q5_0-q4_0 records 0.000964 median KLD at 865.18 tokens/s and 1376.00 MiB, against 0.000936 at 816.51 tokens/s and 1360.00 MiB for kvarn5-kvarn4. Try both on real tasks with long context and only then make a decision.

7. Gemma Is A Different Decision

Gemma's 50 sliding-window layers and ten full-attention layers turn tail 1024 into a structural boundary in these runs. The visible window becomes exact in most layers, so body-width differences survive mainly in the full-attention layers while memory and prefill change sharply. The frontier composition flips between the two models: 23 of the 28 Qwen frontier rows are KVarN, against 6 of the 29 on Gemma.

kvarn5-kvarn4 tail 1024 strictly dominates q8_0 tail 0 on memory and median KLD alike: 1290.86 versus 1317.50 MiB and 0.036086 versus 0.037100. It is not the general recommendation because its measured prefill is 722.84 tokens/s against 932.09, a 22.4% deficit. Standard q8_0 tail 0 is the safer default when throughput and older-context coverage matter; the exact-window KVarN row is for workloads that value the newest 1024 tokens enough to pay that cost.

One further practical note. Across every matched pair, cache quantization costs Gemma far more than Qwen: the shared kvarn5-kvarn4 tail 1024 row reads 0.036086 median KLD against Qwen's 0.000936, and same-top agreement with the BF16 baseline lands at 80.5% against Qwen's 97.976%. KLD is measured against each model's own baseline, so this is cache damage rather than absolute model quality, but the practical consequence is that the same configuration costs Gemma a much wider accuracy spread than it costs Qwen.

Where the workload allows a model choice, Qwen's cache quantizes better and gives more room to tune, with the wider KVarN ladder, longer tail support, and cheaper asymmetric steps. For setups with limited VRAM, using Qwen with a quantized cache is the safer way. If sticking with Gemma is non-negotiable, avoid quantized KV cache whatsoever.

8. The Recommendation Ladder

The ladder turns the matrix into presets, interleaving standard and KVarN rows so both paths stay visible, and it spans the full spread on purpose. The right tier is the one that fits the card: a 24 GB setup and an 8 GB setup are different problems, and at the low end four or even three bits may be the only way a long context fits at all.

8.1 Qwen

The high end starts around 1600 MiB, the balanced band sits near 1300 to 1600 MiB, and below that the low end exists to allow you to fit more context, but without quality expectations.

The asymmetric pairs do the quiet work in the middle. Every rung on the KVarN side is exactly 128.00 MiB, which makes the descent directly comparable: kvarn6 to kvarn6-kvarn5 costs 0.8% of median KLD, then 1.2% to kvarn5, 4.3% to kvarn5-kvarn4, 6.2% to kvarn4, 11.9% to kvarn4-kvarn3, and 18.3% to kvarn3. The first four rungs together, 512.00 MiB from kvarn6 down to kvarn4, cost 13.1%, barely more than the single rung below kvarn4 costs on its own. Below kvarn3 the arithmetic stops being gentle: one more rung costs 84.2%, and the next 57.2%.

At the top the decisions become more complicated due to the noise, driven by extremes skewing the mean and making double-checking median tricky. The mean graph makes that visible.

8.2 Qwen Standard-Only

For setups without KVarN or the tail feature, the same matrix reduces to a standard-only ladder on Qwen 3.6 27B Q5_K_S at 64k. Prefill throughput stays within 882.61 to 888.39 tok/s across every standard tail 0 row, so the only trade on offer here is quality for memory.

Three cliffs shape this ladder, and each one opens directly below a symmetric _0 row. Median KLD rises only 27% across the 768.00 MiB from q8_0 down to q5_0, then the next 64.00 MiB step to q5_0-q4_1 costs 24% on its own. Below q4_0, one 64.00 MiB step to q4_0-q3_1 costs 70%, and below q3_0 the same size step to q3_1-q2_0 costs 161%. When the budget is tight, q5_0, q4_0, and q3_0 are the rows to land on.

q6_0 anchors the high end. At 1664.00 MiB it holds 0.00096 median KLD, which is 512.00 MiB below q8_0 for 5.6% more error, and the frontier rows in between buy that error back at a poor rate: 0.6% for 64.00 MiB at q6_1-q6_0, 4.2% for 320.00 MiB at q8_0-q6_1. It also carries the cleanest 99.9% tail on this ladder, 0.091134 against 0.093029 for q8_0, so the widest cache here is not the one with the best worst case.

8.3 Gemma

Gemma's ladder is the mirror image of Qwen's, and the reason is cost, not coverage. Gemma's 50 sliding-window layers store the tail's exact tokens in native precision across the full 1024-token window, so the tail overlay is far heavier than on Qwen: a flat 96.00 MiB on Qwen at any body width, but 243.36 to 693.36 MiB on Gemma, growing as the body shrinks. At q2_0 the tail overlay alone (693.36 MiB) costs more than the body it sits on (387.50 MiB). KVarN's intrinsic F16 group carries the same penalty.

That cost buys nothing this benchmark can resolve. Gemma's cache damage is already severe at the top of the range: q8_0 tail 0 agrees with the BF16 baseline on 80.445% of top tokens against the baseline's own 99.995%, so a fifth of top-token agreement is gone before any tier is chosen. Twenty configurations match or beat that number: they span 1290.86 to 1628.98 MiB and 719.54 to 932.09 tok/s, yet every one of them lands between 80.445% and 80.859% same-top, a 0.414 point spread against a per-row uncertainty of 0.103 points.

The same three symmetric _0 rows anchor the cliffs here as on Qwen. Median KLD rises only 14.9% across the 310.00 MiB from q8_0 down to q6_0, then one 38.75 MiB step past q6_0 costs 21.7%. The same size step past q5_0 costs 70.9%, past q4_0 it costs 130.5%, and past q3_0 it costs 129.3%. Gemma's cliffs are steeper than Qwen's and its same-top agreement falls with them, from 79.496% at q6_0 to 76.802% at q5_0, 71.347% at q4_0, and 61.353% at q3_0, so the rows to land on are the same ones and the penalty for overshooting them is larger.

9. What This Does Not Prove

This is a KLD benchmark, not a direct agentic-coding evaluation. It does not ask a model to retrieve an old identifier, preserve repository policy, select a tool, emit valid structured output, or repair a failed edit. A row's median KLD is evidence about distribution drift on Wikitext, not a percentage of coding ability retained. The ladder above ranks configurations by that metric, so the distance between two rungs may matter far more or far less on a real workload.

The tail length inherits the same problem. These ladders settle on 1024 because that is where the KLD curve flattens, but Wikitext scores every position more or less alike, while a real session leans on particular earlier spans: a function signature, a system prompt, a tool schema. A workload whose load-bearing tokens sit further back than the exact suffix reaches will not show that failure in any column here. The best play is to adjust tail size to match the task.

10. Method And Full Benchmark Data

The matrix uses one RTX 3090 on Windows 11, BeeLlama v0.4.0. Both models use Q5_K_S weights and Wikitext-2 raw with batch size 2048, micro-batch size 512, seed 1, FlashAttention, unified KV, no mmap, mlock, and no host offload. KLD is calculated against a matching BF16-cache baseline. Tok/s is prompt processing measured inside the run. Filters operate on published tail and on cache family, so selecting 128 includes standard tail 128 and KVarN 128*.

KV cache cells show the persistent total. Where a cache pair has a shorter reference row, a smaller (Δ...) gives the difference from that tail 0 total. KVarN 128* is its own requested-0 baseline and carries no delta, because its intrinsic exact group is already inside the total: on Qwen that group accounts for 48.00 MiB where a standard tail 128 overlay costs 8.00 MiB. Longer KVarN tails are measured from that 128* row, so KVarN and standard deltas begin from different baselines and should be read within a family rather than across the two.

10.1 Qwen 3.6 27B Q5_K_S 64k

* KVarN raw request 0 is the canonical effective tail 128 result and is displayed as 128*. Explicit requested-128 aliases are excluded.

10.2 Gemma 4 31B Q5_K_S 16k

* KVarN raw request 0 is the canonical effective tail 128 result and is displayed as 128*. Gemma tail 2048 is outside this declared quality matrix.