Home › Articles › KVarN KV Cache: Implementation and Benchmarks

KVarN KV Cache: Implementation and Benchmarks

KVarN, a Variance-Normalized KV-Cache Quantization from Huawei, improves KLD vs standard llama.cpp quants at every tested width on Qwen 3.6 27B and Gemma 4 31B.

1. 1. Where KVarN Comes From
2. 2. What KVarN Stores
   1. 2.1 Sink, Body, And Tail
   2. 2.2 What This Implementation Changes
3. 3. Matching The Baseline
4. 4. Where The Standard Ladder Runs Out
5. 5. Changing Only The Format
6. 6. Splitting K And V
7. 7. Gemma Changes The Tradeoff
8. 8. What This Does Not Prove
9. 9. Method
10. 10. Full Benchmark Data
    1. 10.1 Qwen 3.6 27B Q5_K_S 64k
    2. 10.2 Gemma 4 31B Q5_K_S 16k

1. Where KVarN Comes From

KVarN is a KV cache quantizer from Huawei, published as KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks (arXiv 2606.03458), with a reference implementation released as a vLLM fork. Its starting observation is about how KV quantizers are usually judged. Most are measured on prefill-shaped work, where a whole context is scored in one pass, while a reasoning model spends most of its time generating one token at a time. The paper argues that quantization error behaves differently in that second regime, building up across timesteps, and that the buildup is driven mainly by badly estimated per-token scales.

The proposed fix needs no calibration data and has two stages. A Hadamard rotation is applied per head to K and V after RoPE. Rotating mixes every channel of a head into every other, so a single outlier channel no longer stretches the quantization range for the whole head. The query is rotated by the same matrix, which leaves the QK product unchanged, so attention can run in the rotated frame and the result is rotated back once per token rather than per stored token. A variance normalization then rescales both axes of each matrix before rounding, which is the stage aimed at the token-scale errors directly. The method runs that normalization as 16 Sinkhorn iterations. On the paper's own benchmarks the combination sets a two-bit state of the art on MATH500, AIME24, and HumanEval.

The subject here is BeeLlama's implementation of that method rather than the reference, and the measurement is teacher-forced Wikitext KLD rather than the reasoning benchmarks the paper reports. Those are different questions, and a result in one does not transfer to the other.

2. What KVarN Stores

2.1 Sink, Body, And Tail

KVarN is a structured K/V cache format rather than another standard block type, and its precision depends on where a token sits in the sequence. The first 128 tokens stay exact permanently, which preserves the attention-sink behavior models rely on. Everything after that is quantized in groups of 128 tokens, and a group is quantized at the moment its last token arrives. Each group is rotated and normalized before quantization, and the layout stores the quantized payload with its scale metadata. Attention paths that support the format read these records directly; others convert to a standard cache layout first.

The newest tokens have not filled a group yet, so they stay exact until they do. That trailing exact span is what this article calls the tail: a run of the most recent tokens held at higher precision than the rest of the cache. A standard cache treats every token alike, so a tail there is a feature you switch on. In KVarN it is a consequence of the group size, and it is never entirely absent.

In the published design the tail is nothing more than that partial group. It grows from empty toward the group size and collapses each time a group seals, so the number of exact recent tokens is whatever has arrived since the last flush. The paper names the region R and sets it to 128 alongside a group size of 128, and the reference implementation sizes its F16 side buffer for two blocks per request per layer: the permanent sink and the one in-progress tail.

2.2 What This Implementation Changes

BeeLlama does not let the region shrink. It floors the effective tail at one full group and then routes KVarN through the same precision-tail machinery that serves standard caches, so the cache holds at least 128 recent tokens exact rather than however many happen to have arrived. That floor is why a request of --kv-tail-tokens 0 resolves to 128, and it is what makes a stated suffix length comparable between the two families at all. It is a deviation from the reference, and it moves in KVarN's favor, so the rows below describe this arrangement rather than the published one.

Two further details matter for the comparison. This implementation keeps the prefix sink on non-sliding-window layers only; sliding-window layers have no permanent sink, though they still carry their local exact suffix. And the matched tables hold suffix length constant while other properties still differ: standard tails use BF16 in these runs, KVarN tails use F16, and non-SWA KVarN carries the prefix sink.

3. Matching The Baseline

Because KVarN always carries that exact suffix, a comparison against a tail-free standard cache does not result in an honest measurement of quantizers. Wikitext KLD scores the next token at every position, so an exact recent region is worth a great deal to it, and the standard cache can be given the same region for a known price.

No quantizer changed between those columns; the cache simply stopped compressing its newest 128 tokens. What that is worth depends on how good the body underneath already is, from 61.2% at two bits down to 0.9% at eight, and Gemma runs the same shape from 55.0% to 0.4%. None of it is a claim about output. Only 128 tokens of a 65536-token cache change hands, and scoring every position rewards exact recent context far more heavily than most real work does. The tail sweep is its own subject, covered in the precision-tail article; here it exists only to make the two families comparable.

That gradient is why an unmatched comparison misleads worst exactly where KVarN is most interesting. Place a KVarN two-bit row beside a tail-free standard two-bit row and most of the visible gap is the suffix, not the format. At eight bits the same confound is worth under 1% on either model.

So every comparison from here on holds the suffix constant, which gives three published row types. Standard tail 0 is the diagnostic control above. Standard tail 128 is the matched comparator. KVarN tail 128* is the deployable configuration measured against it, and it has no tail 0 counterpart because that configuration can not exist.

4. Where The Standard Ladder Runs Out

First, let's take a look at where KVarN lands among the standard tiers. Both sides of every row below carry the same 128-token suffix. At four bits kvarn4 sits between standard q4_0 and q5_0 at 1184.00 MiB, gaining precision over its own width for less memory than a full standard step up to 1416.00.

The absolute KLD deltas between standard tiers shrink as the type widens. On Qwen one bit from q4_0 to q5_0 buys 0.000236 of median KLD, the next bit to q6_0 buys 0.000057, and the two bits from q6_0 to q8_0 buy 0.000012. Gemma runs the same curve at a coarser scale, 0.033947 then 0.007506 then 0.001672. That non-linearity reflects how fast lower bit widths degrade, not how close the top of the ladder sits to BF16. q8_0 itself agrees with the BF16 baseline on the top token 98.0% of the time on Qwen and 80.5% on Gemma, so every width is a precision tier worth improving, and the question at each is whether KVarN delivers that improvement for comparable memory.

Across all 48 standard pairs at this suffix, on both models, the best row is q8_0 itself: 0.000901 on Qwen and 0.036941 on Gemma. No mixture of standard K and V types goes lower at any memory size, so q8_0 sets the standard floor.

Five Qwen configurations and three Gemma ones sit under that floor. Qwen kvarn6-kvarn5 clears the entire standard family at 1568.00 MiB against q8_0's 2184.00, saving 616.00 MiB over a row it beats outright. On Gemma kvarn6 does it at 1318.98 against 1428.36, saving 109.38 MiB.

The exact margin below q8_0 is harder to pin down. One teacher-forced pass per configuration does not separate close rows with statistical confidence, and there is no standard tier above q8_0 to calibrate the distance against. That these KVarN configurations read lower than the standard best is established by the data; a tighter bound on how much lower would take either more runs or a benchmark with headroom above the standard formats.

On Qwen most KVarN rows also undercut the cheapest standard row matching their median, six of them by exactly 104.00 MiB. On Gemma none do: every Gemma KVarN row a standard cache can match is matched more cheaply, so the format pays there only at the top, where no standard configuration can match it.

5. Changing Only The Format

Section 4 compares configurations of different sizes, which leaves room to read KVarN's position as bought with memory rather than earned by the format. The same-width rows close that off, because they change one variable. At four bits Qwen kvarn4 records a median KLD of 0.001111 against q4_0's 0.001206 while its quantized body is the smaller of the two, 1136.00 MiB against 1152.00. The 24.00 MiB that makes the KVarN total larger is exact overlay, which neither configuration compresses. Whatever produces the difference is happening inside a body with fewer bytes in it.

KVarN improves the Qwen median at every tested symmetric width, but not by a smooth function of width. The gain peaks at 12.6% at three bits and is smallest at 2.6% at six. Neither extreme sits at an endpoint: two bits returns 5.5%, eight bits 3.3%, and four bits 7.9%.

Those percentages are smaller than the effect they stand for, and the control in section 3 shows why. Making the newest 128 tokens exact moves the Qwen four-bit median by 0.000640. Swapping the quantizer underneath moves the same median by 0.000095. This benchmark responds nearly seven times more strongly to which tokens are exact than to which format compresses the rest, and both rows already include the exact suffix. The same-width column is a reading taken on ground that tilts away from the thing being measured.

The untilted version cannot be run, because KVarN has no tail-free configuration and the table without a suffix does not exist. Nor does the tilt have a consistent sign, which rules out correcting for it: across the tail sweep the Qwen advantage widens with a longer tail at two and three bits and narrows at eight. Read these percentages as what this measurement can see, not as the true difference between the formats.

6. Splitting K And V

Width is not the only dial. K and V can take different types, which fills the gaps between whole widths, and the balance between them often matters more than the total. kvarn5-kvarn4 uses 1312.00 MiB and records 0.000995 median KLD, between kvarn4 at 1184.00 and kvarn5 at 1440.00, filling a gap that whole widths leave open.

A one-bit step lands where you would want it. Wherever a two-bit pair meets a symmetric row at the same allocation the symmetric row wins, and by a wide margin at the bottom: kvarn4-kvarn2 records 0.00357 against 0.001797 for kvarn3 at the same 928.00 MiB, roughly twice the error for the same memory.

Three pairs are worth keeping as presets on this evidence. kvarn6-kvarn5 at 1568.00 MiB and 0.000899 is the quality-first compressed choice, sitting inside the same unresolved group as kvarn6 for 128.00 MiB less. kvarn5-kvarn4 is the balanced middle. kvarn3-kvarn2 at 800.00 MiB and 0.00402 is an emergency-compression step, and the two-bit end improves its own tier without reliably replacing the next standard tier. The pattern across the whole matrix, symmetric proof points with near-balanced asymmetric rows filling the staircase between them, is the subject of the full KV cache matrix.

7. Gemma Changes The Tradeoff

Gemma behaves differently. Its 50 sliding-window layers and ten full-attention layers create a discontinuity when an exact suffix covers the visible 1024-token window, and even at 128 tokens the accounting has already diverged from Qwen's. Standard q4_0 uses 808.36 MiB at median KLD 0.080066, while kvarn4 tail 128* uses 1008.98 MiB at 0.058185: a 27.3% lower median at four bits, against 7.9% on Qwen, for 200.62 MiB more.

That 200.62 MiB is fixed across all six widths, as the 24.00 MiB is on Qwen, but it is charged somewhere else. Gemma's exact overlay is 110.86 MiB on both sides of the comparison, so the entire premium is in the quantized body: 898.12 MiB against 697.50 at four bits, where the Qwen body ran 16.00 MiB in the other direction. Prefill is more expensive too, falling from 6.9% at two bits to 14.0% at eight against 5.1% to 6.7% on Qwen. Longer tails change the memory arithmetic again, which is why the practical comparison belongs with the full tail sweep rather than here.

Two bits reverses the outcome outright. At the matched suffix kvarn2 tail 128* records 2.091697 median KLD against 1.329847 for standard q2_0, 57.3% worse for 200.62 MiB more. This is the only symmetric width in either model where a KVarN row records a higher median KLD than its same-width standard counterpart at the same published tail, and on Gemma the advantage only starts at three bits.

This is why the recommendation cannot be copied between models. Qwen offers a smooth body-width ladder for a near-fixed 24.00 MiB and about 6% of its prefill throughput. Gemma charges 200.62 MiB and as much as 14.0% of throughput for the same move, so a standard high-bit body can remain the better default there even where the compressed row records the lower median.

8. What This Does Not Prove

This is one teacher-forced Wikitext-2 pass per configuration, not a coding-agent evaluation. It does not measure retrieval of an old symbol, adherence to repository instructions, correct tool selection, valid JSON, edit quality, or accumulated autoregressive error. KLD, perplexity, same-top probability, prefill throughput, and decode throughput are related observations, not interchangeable quality scores.

Every row here holds its newest 128 tokens exact, so none of them says much about a decision made tens of thousands of tokens earlier. The KVarN prefix sink, the tail floor this implementation adds on top of the published design, F16 versus BF16 tail representation, one GPU, and the unrecovered binary hash of the quality runs are further limits. Close high-bit rows are best treated as a band.

9. Method

The quality matrix uses one RTX 3090 on Windows 11, BeeLlama v0.4.0, CUDA 13.1 build. The two models are Qwen 3.6 27B Q5_K_S at 65536 context and Gemma 4 31B Q5_K_S at 16384 context. Both use Wikitext-2 raw, batch size 2048, micro-batch size 512, seed 1, FlashAttention, unified KV, no mmap, mlock, and no host offload. Every row is measured against its model's BF16-cache baseline. The complete standalone view contains BF16, all 48 standard pairs at tails 0 and 128, and all 15 selected KVarN pairs at tail 128* for each model.

Median KLD ranks the typical scored position, and the mean and upper percentiles show whether a similar median hides a heavier error tail. Same-top probability asks whether the most likely token still agrees with the BF16-cache baseline. Tok/s is prompt processing measured inside the run, not generation speed.

KV cache cells show the persistent total, reconstructed from each selected run's runtime buffer accounting. Where a cache pair also has a tail 0 row, a smaller (Δ...) gives the difference from that tail 0 total. KVarN tail 128* is its own requested-0 baseline and carries no delta, because its exact overlay is already inside the total.

10. Full Benchmark Data

10.1 Qwen 3.6 27B Q5_K_S 64k

* KVarN compresses in 128-token groups and keeps the newest group exact in F16. A raw tail request of 0 therefore publishes as tail 128*. Non-SWA KVarN also keeps an exact group-0 sink.

10.2 Gemma 4 31B Q5_K_S 16k

* KVarN compresses in 128-token groups and keeps the newest group exact in F16. A raw tail request of 0 therefore publishes as tail 128*. Gemma's sliding-window layers do not use the permanent non-SWA prefix sink.