Home › Articles › KV Cache Precision Tail: Implementation and Benchmarks

KV Cache Precision Tail: Implementation and Benchmarks

Precise recent KV cache recovers low-bit cache fidelity, and KVarN delivers stronger precision at lower VRAM cost, as demonstrated on Qwen 3.6 27B.

1. Why Keep Recent Tokens Exact
2. What A Precision Tail Does
3. How To Read The Results
4. The Effect Of Tail Size
5. Standard Quants Versus KVarN
6. Practical Recommendations
7. Qwen Performance And Implementation Status
8. Why Gemma Is Harder
9. What This Does Not Prove
10. Method And Full Benchmark Data
    1. Qwen 3.6 27B Q5_K_S 64k
    2. Gemma 4 31B Q5_K_S 16k
    3. Decode Throughput By Context Depth

1. Why Keep Recent Tokens Exact

The earlier KVarN benchmark established a useful one-tier shift: KVarN4 approached ordinary q5 quality at roughly q4 memory, KVarN5 approached q6, and KVarN6 reached the q8 quality ceiling without paying for a q8 cache. This benchmark adds a second control over that tradeoff. Instead of quantizing every cached token, BeeLlama can keep the newest N tokens in BF16 or F16 and quantize only the older body.

That changes the practical question. A 64k context does not always need 64k tokens of full-precision KV. Many conversations and agent workloads need the full history to remain retrievable, but need the latest instructions, tool results, code, or document fragment to remain especially precise. A 1024-token tail targets that case directly: old context keeps the normal quantized-cache memory cost, while the newest 1024 rows remain exact.

On Qwen 3.6 27B, the effect is largest in the low-bit rows. The median Kullback-Leibler divergence (KLD) of q4_0 falls from 0.001846 to 0.001057 with a 1024 tail, a 42.7% reduction, while persistent KV grows from 1156.50 to 1252.50 MiB. At 64k, that is a 96 MiB tail instead of replacing the entire 4112 MiB BF16 cache. Lower-bit caches gain even more; higher-bit caches quickly reach a point where there is little error left for the tail to remove.

2. What A Precision Tail Does

The cache is divided conceptually into an exact recent region and an older quantized body. Attention still sees the whole valid context. The tail does not truncate old tokens, summarize them, or make long-context retrieval local-only. It changes the representation: the newest N K/V rows are read from an exact BF16/F16 store, while older rows are read from the selected standard quant or KVarN records, and the two attention results are combined under one softmax.

Standard quantized caches use a BF16 tail by default in these runs. KVarN uses F16 because its native staging and record path is F16-based. In this article, exact means stored in an unquantized floating-point format; it does not mean that the standard and KVarN tails use the same format. At tail 0, KVarN keeps the currently incomplete part of its 128-token staging group exact, so the exact recent span varies from 0 to 127 rows as that group fills. The non-SWA layout also keeps a permanent exact sink group. A KVarN tail-0 row is therefore not physically identical to a standard cache with no exact rows; explicit 1024- and 2048-token settings provide a fixed exact recent region in addition to the record layout.

This distinction matters for real workloads. Retrieval from 20k or 50k tokens ago still depends on the selected quantizer, so a tail does not cure poor long-range fidelity. What it can do is preserve the latest 1k, 2k, or another user-selected span without multiplying the whole KV budget by two or three. A coding session might keep the current file and latest tool output exact; a document workflow might keep the latest source section exact while older references remain compressed. The tail size is therefore a workload control, not a universal constant.

The low-bit mean-KLD graph shows that basic effect before the finer statistical distinctions matter. At q2-q4, moving from tail 0 to 1024 shifts both standard and KVarN points sharply downward for a modest move to the right: less divergence for a relatively small increase in persistent KV memory.

3. How To Read The Results

The main metric in this article is median KLD, the typical per-token divergence from the matching full-BF16-cache baseline. Mean KLD remains in the full results and in the high-end comparison below, with 90th- and 99th-percentile KLD as additional checks. That is a change from my earlier KVarN article, where mean KLD carried more of the ranking.

The reason is visible in the new logs. Qwen's high-bit median is about 0.0009 and its 99th percentile about 0.019, but isolated maximum values reach 13-22. With roughly 131,000 scored positions, one KLD-20 token contributes about 0.00015 to the mean by itself. That is larger than several complete q5/q6/q8 differences in the chart. The mean is still a valid corpus average, but it is noisy because a few extreme positions can reorder otherwise similar high-bit caches.

The KLD +/- field needs a careful reading. llama-perplexity calculates it from the distribution of per-token KLD values in one run. It is not run-to-run variation and not a paired confidence interval between two cache configurations. It quantifies how much the token-level values vary inside that corpus, which is useful context, but overlapping values do not prove that two configurations are statistically tied.

Median KLD is therefore the clearest main signal for relative cache and tail quality: it is stable enough here to show the expected low-bit improvements and high-bit saturation. Mean KLD answers a different, still useful question—the average divergence across all scored tokens—but a handful of extreme positions can move it more than the entire difference between q5, q6, and q8. The p90 and p99 values show whether the typical result extends into the worse parts of the distribution. At p99.9 and maximum KLD, the benchmark is primarily describing rare-event behavior.

Read the last three KVarN rows as saturation, not as proof that five bits are intrinsically more precise than eight. Median, p90 and p99 put KVarN6 and KVarN8 at effectively the same ceiling, while the outlier-sensitive mean changes their order. The standard rows tell the same story: q8 has the best median, q6 has a fractionally better p99, and neither advantage is large enough to matter as much as VRAM or speed.

4. The Effect Of Tail Size

Qwen gives a clean low-bit tail curve. More bits improve median KLD across the standard-cache rows, and larger tails improve every standard cache. KVarN follows the same pattern until the high-bit results reach a plateau. The more aggressive the body quantization, the more useful it is to keep the newest rows out of that body.

KVarN8's 0.7% increase from tail 1024 to 2048 is six millionths of KLD, inside the same high-bit plateau seen elsewhere. It does not establish that a larger exact tail damages an 8-bit cache. It establishes that this benchmark cannot rank changes that small reliably from a single corpus run.

The default recommendation is straightforward: start with a 1024-token tail. It captures most of the KLD benefit, costs 48 MiB on Qwen KVarN or 96 MiB on the standard caches tested here, and usually reduces throughput by only a few percent. A 2048 tail is easier to justify with q2/q3, where the second 1024 tokens still produce a visible precision gain, or when the workload itself needs roughly 2k recent tokens protected. At q5 and above, Wikitext KLD has mostly saturated after 1024.

This is of course just a benchmark result, not a universal limit. KLD evaluates one fixed teacher-forced Wikitext scenario in prefill-sized batches. A support agent that must preserve the latest long customer message, or a coding agent carrying a 3k-token source file, may benefit from a 2048 or 4096 tail even when this aggregate KLD barely moves. Larger sizes are available so the user can match the exact region to the workload. But the cost grows with the tail, and the Qwen measurements show the expected few-percent throughput loss, so there is no reason to allocate a larger exact region without a use case for it.

5. Standard Quants Versus KVarN

At tail 0, KVarN beats the corresponding standard cache at every tested bit width on Qwen median KLD. The KVarN allocation is about 32 MiB larger because its structured records and exact staging region carry fixed overhead, but the precision gain is large at 2-4 bits and gradually shrinks as both families approach the high-bit floor.

The more useful comparison is precision per VRAM. KVarN4 at 1188.38 MiB slightly beats q5_0 at 1413.50 MiB on median KLD. KVarN5 at 1445.38 MiB beats q6_0 at 1670.50 MiB. KVarN6 at 1702.38 MiB slightly beats q8_0 at 2184.50 MiB. This reproduces the recommendation ladder from the previous KVarN article with a cleaner typical-token metric.

This is not a renamed standard quantizer. BeeLlama keeps KVarN in a separate structured cache backend with 128-token records, Hadamard rotation, iterative variance normalization, dedicated scale metadata, F16 staging, custom CUDA store kernels, and native KVarN FlashAttention dispatch. Supporting it means integrating allocation, cache updates, attention, rollback, save/restore, and architecture-specific cache layouts. The current implementation targets Qwen 3.6 and Gemma 4 rather than claiming universal model support; broader architecture coverage will need explicit work in future versions.

6. Practical Recommendations

For Qwen, these symmetric results can be combined with the earlier asymmetric recommendation ladders. They do not replace those ladders because K/V pairs such as 5/4 and 6/5 were not retested with tails here, but they make the main decision points clearer.

KVarN8 is mainly a ceiling check. Its median is excellent, but KVarN6 is already at the same practical KLD level for 514 MiB less at tail 1024. The same logic applies to ordinary q8_0 versus q6_0: if VRAM matters, the tiny high-end median difference is hard to justify. If compatibility or maximum conservatism matters more than memory, q5_0/q6_0 with a 1024 tail remain sensible standard-cache choices.

Asymmetric pairs may improve several of these rows. Earlier work found that keeping K one step above V often produced a better allocation, but this article does not assume the old ranking automatically survives the new tail path. That is the next benchmark rather than a claim hidden inside this one.

7. Qwen Performance And Implementation Status

The v0.3.2 preview established KVarN's precision but used a raw integration whose speed numbers were not ready to recommend. The current Qwen path is materially different: dedicated CUDA FlashAttention kernels consume KVarN records directly, and exact-tail attention has kernel coverage for various geometries.

On the tested Qwen 3.6 27B, RTX 3090, and CUDA configuration, the benchmark behavior now looks like a usable optimization rather than a precision-only prototype. Standard Qwen prefill stays around 856-884 tok/s across the tested q4 tail sizes. KVarN4 prefill is slower at 808-821 tok/s, but decode at 64k is competitive: 25.30 tok/s at tail 0 versus 24.63 for q4_0, and 25.00 versus 24.10 at tail 1024. Larger tails carry the expected cost rather than causing a collapse.

The prefill and decode measurements come from different benchmark builds, so the two tok/s columns should not be compared to each other. Within each column, however, they show the intended pattern. A 1024 tail costs about 0.8-2.2% on these q4 rows; a 2048 tail costs about 1.5-4.3%. That is a real performance price, but a controlled one.

8. Why Gemma Is Harder

Gemma 4 uses a materially different cache layout from Qwen. The tested 31B model has 60 cache-bearing attention layers: 50 use a 1024-token interleaved sliding-window-attention (iSWA) ring and 10 use full attention. With n_swa=1024 and ub=512, BeeLlama (and upstream llama.cpp) allocates a 1536-row physical SWA ring so the 1024 visible rows and in-flight ubatch rows can coexist safely.

A requested 1024 tail therefore covers the complete visible SWA window. For standard caches, BeeLlama promotes that entire 50-layer ring to BF16 instead of keeping a quantized body plus an equally large overlay. For KVarN it uses a full F16 SWA ring. This is why the VRAM numbers jump toward the same point regardless of the original bit width: q4_0 grows from 703.12 to 1685.62 MiB, while KVarN4 moves from 1563.59 to 1695.47 MiB. At KVarN8, replacing the records and staging allocation with the plain F16 ring is actually smaller, so the total falls from 2188.59 to 2020.47 MiB.

That behavior is correct for the current representation, but it largely erases the memory ladder that makes KVarN plus a tail attractive on Qwen. It also makes performance inconsistent. Gemma KVarN4 prefill is 692.34 tok/s at tail 0 and 738.67 with the exact SWA ring, well below the 898.58 tok/s standard q4_0 tail-1024 run. I do not recommend KVarN or precision tails for Gemma 4 as a general optimization in this version.

Tail 1024 is the only obvious Gemma preset to test first because it matches the SWA visibility window exactly. Larger requested tails cannot make those 50 SWA layers see more than 1024 tokens; they can only extend the tail on the 10 full-attention layers. That may help a specialized workload, but it no longer has the simple Qwen cost/benefit curve.

The precision results are still valuable as an independent check. Gemma's standard-cache medians improve with bit width, KVarN beats the matching standard quant, and a 1024 tail strongly rescues q2-q4. KVarN6 and KVarN8 with tail 1024 are effectively tied rather than strictly ordered. For example, q4_0 falls from 0.134091 to 0.045341 median KLD, while KVarN4 falls from 0.058185 to 0.037166. The quality mechanism works; the current SWA storage and kernel economics do not yet make it a good general recommendation. I will continue looking for a representation that keeps the exact visible ring without losing most of the VRAM and throughput advantage in future BeeLlama versions.

9. What This Does Not Prove

KLD is a sensitive distribution-fidelity benchmark, not a 1:1 simulation of generation. It teacher-forces Wikitext tokens and evaluates prompt-sized batches against stored BF16-cache baseline logits. Real generation normally adds one token at a time, sampling can make two runs diverge after the first changed choice, and user-visible quality depends on what those choices mean. These tables establish local cache fidelity and expose large regressions; they do not replace retrieval tests, reasoning tasks, coding workloads, or human evaluation.

The high-bit mean graph has a second limitation. Its q5-q8 points are close enough that a few extreme token positions can change their order, so it is not a reliable picture of the underlying precision ladder. In particular, q8 sharing or exceeding q6's mean KLD, and some tail-2048 points moving upward, should be read as outlier sensitivity in this run rather than evidence that more bits or more exact rows reduce general cache fidelity.

The exact tail also does not make the full context exact. A fact retrieved from the quantized body is still subject to the selected cache format. The tail is useful when a workload has a privileged recent region, and its value will vary with attention patterns, prompt structure, and the amount of recent material that must remain precise. The Wikitext result places the broad efficiency knee at 1024 tokens on Qwen; other workloads may put it elsewhere.

Finally, this is not the complete precision-per-VRAM ladder. The symmetric rows make the tail and KVarN effects easy to see, but asymmetric K/V pairs may offer better final presets. Model support is also not exhaustive because KVarN is a separate structured cache implementation with custom kernels, not a generic extension automatically inherited by every llama.cpp architecture. The Qwen results demonstrate that the design works on this non-SWA cache layout. Gemma shows why each new cache layout still needs its own engineering and validation.

10. Method And Full Benchmark Data

The compact tables above carry the analysis. The complete KLD, percentile, VRAM, same-top-p, prefill, and decode results follow for reproducibility and for readers who want a different tradeoff than the recommendations.

The KLD and prefill runs used BeeLlama v0.4.0 commit 41d41b0ce5 on one power limited RTX 3090 (24 GB). The corpus was wiki.test.raw from Wikitext-2 raw. The seed was fixed to 1. Every candidate was compared with a baseline generated by the same executable, model, corpus, context, -b 2048, and -ub 512. The baseline used BF16 K and V and llama.cpp's standard v1 KLD-baseline format.

The candidate command followed this template; MODEL, CONTEXT, CACHE, BASELINE, and TAIL identify the row:

llama-perplexity -m MODEL.gguf -ngl all -b 2048 -ub 512 \
  --ctx-size CONTEXT --cache-type-k CACHE --cache-type-v CACHE \
  --flash-attn on --seed 1 --no-mmap --mlock --no-host --kv-unified \
  -f wiki.test.raw --kl-divergence-base BASELINE.kld --kl-divergence \
  --kv-tail-tokens TAIL

The baseline used the same template with bf16 K/V, tail 0, and without --kl-divergence, causing llama-perplexity to write the reference logits. The percentage columns labelled exp(-ΔKLD) are derived similarity proxies, not task accuracy or a probability that the quantized cache is correct. KLD +/- is the standard error calculated over per-token KLD values within a run; it is not repeat-run variance.

10.1 Qwen 3.6 27B Q5_K_S 64k

Each row uses the symmetric K/V cache type and tail size shown. The default tail representation is bf16 for standard quantized caches and f16 for KVarN; the bf16 cache appears at tail 0 only. Tok/s is prompt-processing throughput from the same runs. KV cache (MiB) is persistent KV VRAM from llama-bench --kv-memory on the same BeeLlama CUDA build; tail rows show combined (tail-0 base + tail delta).

10.2 Gemma 4 31B Q5_K_S 16k

Gemma's iSWA layers cap the visible precision tail at n_swa=1024, so a requested 2048 tail resolves to 1024 on those layers and only tail 0 and 1024 are shown. The default tail representation is bf16 for standard caches and f16 for KVarN; the bf16 cache appears at tail 0 only. Tok/s is prefill throughput; KV cache (MiB) is persistent KV VRAM from llama-bench --kv-memory, with tail rows shown as combined (tail-0 base + tail delta).

10.3 Decode Throughput By Context Depth

Decode (token generation) throughput comes from separate decode benchmark runs with BeeLlama build 44ac8f800 on one power limited RTX 3090. Each cell is average tokens per second over 128 generated tokens and five repetitions from llama-bench at the given context depth, with -b 2048 -ub 512 and symmetric K/V. Tail representation is bf16 for q4_0 and f16 for KVarN; bf16 is measured at tail 0 only. Gemma was measured through 16k and Qwen through 64k.