Home › Articles › KV Cache Precision Tail: Implementation and Benchmarks

KV Cache Precision Tail: Implementation and Benchmarks

A small exact recent tail can recover much of a low-bit KV cache's lost precision, as shown on Qwen 3.6 27B and Gemma 4 31B across standard and KVarN formats.

1. 1. Why Keep Recent Tokens Exact
2. 2. What A Precision Tail Does
3. 3. How To Read The Results
4. 4. The Effect Of Tail Size
5. 5. Standard Quants Versus KVarN
6. 6. Practical Recommendations
7. 7. Why Gemma Is Harder
8. 8. What This Does Not Prove
9. 9. Method And Full Benchmark Data
   1. 9.1 Qwen 3.6 27B Q5_K_S 64k
   2. 9.2 Gemma 4 31B Q5_K_S 16k
   3. 9.3 Decode Throughput By Context Depth

1. Why Keep Recent Tokens Exact

Quantizing the whole KV cache treats every retained token alike, even though the newest tokens are often carrying the unfinished sentence, the current code block, or the tool output the model is about to act on. BeeLlama v0.4.0 adds a new precision tail feature: a configurable exact suffix that gives that recent span full-precision storage while leaving older history in the selected low-bit body.

The effect is easy to see in the Qwen controls. Standard q4_0 starts at 0.001846 median KLD and 1152.00 MiB with tail 0. tail 128 moves the median to 0.001206 for 1160.00 MiB; tail 1024 reaches 0.001057 for 1248.00 MiB; tail 2048 reaches 0.001019 for 1312.00 MiB. The first exact tokens do most of the work, and the second thousand buy a smaller change.

2. What A Precision Tail Does

The tail changes storage precision, not the attention pattern. Full-attention layers still attend across the exact suffix and quantized body. Sliding-window layers still observe their configured window. As tokens arrive, the boundary advances: once a token falls behind the exact span it is stored in the quantized body.

Standard caches in these runs use a BF16 tail. KVarN quantizes in groups of 128 tokens, and each group is quantized only when its last token arrives. The newest tokens have not filled a group yet, so they stay in F16 until they do. BeeLlama floors the tail at one full group, so a raw request of 0 becomes an effective 128-token exact suffix. The tables display that minimum as 128*, and KVarN has no tail 0 rows, as they are impossible to achieve.

3. How To Read The Results

The full tables are deliberately symmetric: K and V use the same type in every non-BF16 row. That covers all eleven symmetric standard types, including the _1 variants, and all six symmetric KVarN types, which isolates body width and tail size before the broader practical review introduces asymmetric pairs. Qwen includes standard tails 0, 128, 1024, and 2048 plus KVarN tails 128*, 1024, and 2048. Gemma stops at 1024 because that already covers its visible sliding window.

Median KLD ranks the typical scored position. Mean and upper percentiles show whether a similar median hides a heavier error tail, and section 5 puts that check to work. Same-top probability asks whether the most likely token agrees with the BF16-cache baseline. Tok/s is prompt processing from the quality run, not generation speed. Persistent KV memory sums the row-matched runtime cache buffers.

4. The Effect Of Tail Size

Low-bit bodies have the largest recoverable error. On Qwen, standard q2_0 falls from 0.019374 median KLD at tail 0 to 0.007521 at 128, 0.004648 at 1024, and 0.003696 at 2048. KVarN2 begins from a stronger intrinsic 128* configuration at 0.007108, then falls to 0.003811 and 0.002820. At eight bits the curve is almost flat: standard q8_0 moves from 0.000909 at tail 0 to 0.000895 at 2048, while kvarn8 is 0.000871 at 128* and 1024 and 0.000877 at 2048.

Tail 128 is valuable here as a control and fairness point. It is not automatically the best working size. Teacher-forced Wikitext continually gives the most recent text a chance to matter, which can flatter even a short exact suffix. For a real long-context workload, tail 1024 is the more practical first step when recent source or tool output is privileged, while tail 2048 and above are a task-specific purchase after that.

5. Standard Quants Versus KVarN

On Qwen, KVarN records lower median KLD than the standard cache at the same nominal width and published tail. At 128 tokens, kvarn4 records 0.001111 against 0.001206 for standard q4_0. At 1024 the comparison is 0.000994 against 0.001057. At two bits the gap widens instead, from 5.5% at tail 128 to 18.0% at 1024 and 23.7% at 2048, and three bits behaves the same way. The longer exact suffix helps the low-bit KVarN body more than it helps the low-bit standard body.

Those are median rankings, and the mean sometimes disagrees. At tail 1024, kvarn4 records the lower median against q4_0, 0.000994 against 0.001057, and the higher mean, 0.003059 against 0.002881; kvarn5 against q5_0 behaves the same way. This is partually a matter of noise and is hard to draw conclusions from, because even doing the same benchmarks at different batch sizes would results in mean extremes shifting significantly.

Memory runs the other way as the tail grows. The KVarN body is smaller than the standard body at the same width, 1136.00 MiB against 1152.00 at four bits, while its intrinsic exact group costs 48.00 MiB where a standard tail 128 overlay costs 8.00 MiB. That leaves KVarN 24.00 MiB larger at tail 128. At tail 1024 and 2048 both sides carry the same overlay, only the body difference remains, making every Qwen KVarN row 16.00 MiB smaller.

The standalone KVarN article uses the same baseline structure. Standard tail 0 is a diagnostic control for suffix effects; standard tail 128 versus KVarN 128* compares deployable configurations at the same suffix length. The comparison is not a pure quantizer ablation because KVarN uses F16 for the suffix and retains an exact prefix sink on non-SWA layers.

6. Practical Recommendations

For Qwen, a low-bit body benefits materially from the first 1024 exact tokens. At five bits and above the tail's measured contribution shrinks, though the body is still a measurable step below BF16 at every width. On the KLD axis alone the tail is often the better purchase: q4_0 at tail 1024 records 0.001057 for 1248.00 MiB, while q5_0 at tail 0 records 0.001154 for 1408.00 MiB. This review still prefers body precision for workloads that value old and recent tokens equally, because teacher-forced Wikitext rewards the recent span more heavily than a long agent session does.

A balanced starting point is a KVarN4 or KVarN5 body with tail 1024 when the backend supports it; standard q4/q5 is useful when prefill speed or portability matters more. Tail 2048 makes sense when the newest two thousand tokens are genuinely the privileged working set, such as a long compiler trace or a source file just loaded into context. It is less persuasive for retrieval-first workloads where an instruction or definition from much earlier must survive.

The standard path is faster in Qwen prefill throughout these symmetric rows. At the four-bit matched point it processes 871.68 tokens/s for standard q4 tail 128 versus 822.51 for KVarN4 128*. Within every symmetric Qwen series, measured prefill throughput also falls as the published tail grows. Generation throughput stayed solid, as shown separately below.

7. Why Gemma Is Harder

Gemma's 50 sliding-window layers see 1024 tokens while ten layers use full attention. In these benchmark artifacts, tail 1024 makes the complete visible window exact in most layers, alongside a large KLD improvement and a sharp memory/throughput transition. Standard q4 moves from 0.134091 at tail 0 to 0.044082 at 1024, while KVarN4 moves from 0.058185 at 128* to 0.037354 at 1024.

The exact window also weakens the normal memory ladder. Standard q4 tail 1024 uses 1240.86 MiB against 1250.86 for KVarN4 at the same tail, while q8 tail 0 uses 1317.50 MiB and keeps much higher prefill throughput. Gemma therefore needs a separate decision: exact-window configurations are compelling when recent context dominates, while a standard high-bit body is a strong general default.

8. What This Does Not Prove

KLD measures how the full output distribution shifts at every scored position, so it is sensitive to cache changes that perplexity can hide, but it is not coding accuracy, long-range retrieval, instruction retention, tool reliability, or error accumulation. A tail that cuts median KLD in half on Wikitext may do far less for a task where the model needs to recall a function signature from 3000 tokens ago, or far more for a task where the last few lines of context carry most of the signal.

Tail 128 produces some of the largest KLD improvements in these tables because even a small exact suffix keeps the newest tokens in full precision format. Real tasks are not scored token-by-token across an encyclopedic corpus. Instruction following, code completion, and tool use depend on retaining specific earlier context, and the tokens that matter may sit well beyond 128 positions back. A working tail for those tasks likely starts at 1024 or 2048 and may need to be even larger, depending on how the model uses its context window.

9. Method And Full Benchmark Data

The quality matrix uses one RTX 3090 on Windows 11, BeeLlama v0.4.0, CUDA 13.1 build. Both models use Q5_K_S weights and Wikitext-2 raw with batch size 2048, micro-batch size 512, seed 1, FlashAttention, unified KV, no mmap, mlock, and no host offload. KLD is calculated against a matching BF16-cache baseline.

KV cache cells show the persistent total. Where a cache pair has a shorter reference row, a smaller (Δ...) gives the difference from that tail 0 total. KVarN 128* is its own requested-0 baseline and carries no delta, because its intrinsic exact group is already inside the total: on Qwen that group accounts for 48.00 MiB where a standard tail 128 overlay costs 8.00 MiB. Longer KVarN tails are measured from that 128* row, so KVarN and standard deltas begin from different baselines and should be read within a family rather than across the two.

9.1 Qwen 3.6 27B Q5_K_S 64k

* KVarN requested tail 0 resolves to an effective 128-token F16 suffix and is displayed as 128*. Standard tails use BF16.

9.2 Gemma 4 31B Q5_K_S 16k

* KVarN requested tail 0 resolves to an effective 128-token F16 suffix and is displayed as 128*. tail 1024 covers Gemma's visible sliding window; tail 2048 is outside this quality matrix.

9.3 Decode Throughput By Context Depth