Home › Articles › KVarN KV Cache: Implementation and Benchmarks

KVarN KV Cache: Implementation and Benchmarks

Tests on Qwen 3.6 27B show how an experimental BeeLlama KVarN preview moves q4-class KV memory into higher tier and beats all the alternatives in the same weight class.

1. What Changed
2. What KVarN Is
3. The BeeLlama Preview
4. The Setup
5. How To Read The Results
6. The Main Result
7. The K/V Matrix
8. TurboQuant And TCQ
9. What This Does Not Prove
10. Practical Takeaway
11. Benchmark Data
    1. Perplexity
    2. KL Divergence
12. Reference Data
    1. Perplexity
    2. KL Divergence

Disclaimer: this is still a narrow benchmark. The KVarN path tested here is a preview implementation in BeeLlama v0.3.2, not a mature optimized runtime. The precision numbers are the meaningful part of the article, but may be subject to change with future updates. The tok/s column is prompt-processing behavior on a very raw preview of this specific fork, not a final verdict on KVarN's decode or generation speed.

1. What Changed

The original KV-cache benchmark had a pretty clear sub-6-bit story: normal q4/q5 cache quantization was stronger than TurboQuant at similar practical sizes, while TCQ made the 2-3 bit turbo modes less bad. You could save memory, or you could preserve the output distribution, and the tradeoff was mostly where you expected it to be.

KVarN changes that curve. On the main Q5_K_S 64k comparison, kvarn4-kvarn4 uses 27.9% of the bf16 KV-cache footprint and lands around the q5_0/q5_1 quality tier. kvarn4-kvarn3 uses 24.8%, below ordinary q4_0, and still beats q4_0 on mean KLD. kvarn3-kvarn3 sits near the turbo3 memory class and cuts the turbo3_tcq mean KLD by a third.

That does not make KVarN lossless. The bf16 KLD reference is still far away. But for VRAM-constrained long-context setups, "q5-ish quality at q4-ish memory" is the result people are happy to see for low-bit KV-cache methods. Where TurboQuant promised that kind of shift, KVarN is the first thing in these benches that actually looks like it.

2. What KVarN Is

KVarN is a KV-cache quantization method from the paper KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks. The paper's central claim is that current KV-cache quantizers make bad token-scale errors during autoregressive decoding, and those errors accumulate over time, while KVarN reduces the outlier errors that matter most.

The mechanism has two parts. First, KVarN applies a Hadamard rotation in the channel dimension, which spreads outlier energy across coordinates before quantization. That part is familiar from QuaRot-style weight work and from the broader low-bit KV-cache literature. Second, it applies dual-axis variance normalization across both dimensions of a KV tile, so both token and channel scale variation are brought under control before round-to-nearest quantization. The paper describes this as a Sinkhorn-inspired variance-normalization step.

In plain terms: ordinary per-axis quantization can preserve one shape of scale variation and still blow up a few token norms. KVarN spends extra metadata on a second scale so the quantizer can keep those token magnitudes closer to where they were. The paper argues that the worst few percent of quantization errors do disproportionate damage, especially in decode-heavy reasoning tasks, so suppressing those outliers matters more than lowering average reconstruction error everywhere.

3. The BeeLlama Preview

BeeLlama is my llama.cpp fork for long-context experiments. The earlier benchmark used it for q6_0, TurboQuant and TCQ cache modes. This KVarN run is the same kind of fork-level experiment, but the KVarN path is much younger. It was wired in as experimental cache types named kvarn2, kvarn3 and kvarn4, plus asymmetric K/V pairs.

The BeeLlama implementation tested here follows the same broad KVarN structure: 128-token groups, 2/3/4-bit payloads, Hadamard rotation, iterative variance normalization, and stored scale/zero-point metadata. The measured sizes in this article are from the actual BeeLlama run logs, which matters because this preview stores auxiliary values as fp16 records, while the paper reports a more optimized accounting with lower auxiliary precision.

KVarN is also not just one more GGUF-style cache quant like q8_0, q5_0, q4_0, or the turbo types. In BeeLlama, kvarn2, kvarn3, and kvarn4 are CLI pseudo-types that select a separate structured KVarN cache backend. The underlying cache tensors are kept as an fp16 staging path plus KVarN records, and the KVarN configuration is stored separately from normal type_k/type_v because each record spans a full 128-token K/V tile.

That is why there are no rows like q8_0-kvarn4 in the current preview. They are not impossible in principle, but they would require a real hybrid-cache architecture: one side allocated and served by the normal KV path, the other side by KVarN, with attention graph routing, CUDA kernels, state save/load, rollback, prompt cache, seq_cp/seq_rm, DFlash backup, SWA/iSWA, and multi-sequence behavior all updated for split ownership. That would be quite complex to implement, and it is unclear whether the partial compression would be worth the extra risk.

Any speed claims also get a caveat. The paper's speed story is about an optimized vLLM-oriented decode path, while the tok/s values in this article are prompt-processing throughput from llama-perplexity. BeeLlama's current path is a raw llama.cpp integration with custom KVarN store/materialize operations and conservative memory handling. If it is slower in prompt processing today, that is operationally true for this build, but it says little about optimized generation speed.

4. The Setup

The hardware and model setup matches the original article as closely as possible. Hardware: one RTX 3090 with 24 GB VRAM, Ryzen 7 5700X3D, and 32 GB system RAM. Model: Qwen 3.6 27B. The main comparison is Q5_K_S weights at 64k context, because that is the cleaner weight quant for this purpose. IQ4_XS at 64k and 128k is included mostly as a tie-breaker and noise check; it is useful, but I do not treat it as the primary ranking signal.

The PPL run tested only symmetric KVarN cache types: kvarn4, kvarn3, and kvarn2. The KLD run tested the full 3x3 K/V KVarN matrix: kvarn4-kvarn4, kvarn4-kvarn3, kvarn4-kvarn2, and so on down to kvarn2-kvarn2. KLD also included the control rows from the original benchmark to ensure there's no significant drift from the engine changes.

All KLD rows are measured against a bf16 KV-cache baseline using llama-perplexity --kl-divergence on Wikitext-2. That is not the paper's pseudo-decode evaluation and it is not an end-to-end reasoning benchmark. It is a local distribution-fidelity test. I like it here because it catches output-distribution movement that PPL tends to hide.

The tok/s column is prompt-processing/prefill throughput from the benchmark run, not generation speed. The table sizes are the actual KV-cache sizes reported by the run logs. KVarN could still behave differently during decode, including faster after optimized kernels; these tables do not decide that question.

5. How To Read The Results

PPL makes KVarN look nearly lossless at the top settings. On Q5_K_S 64k, bf16 PPL is 5.4800 and kvarn4 is 5.4807. That looks like nothing happened. But KLD says something did happen: bf16 mean KLD is 0.000375, while kvarn4-kvarn4 is 0.002974. That is not fp16 precision at the distribution level.

That distinction matters. The KVarN paper reports near-fp16 behavior on generative benchmarks such as MATH500, AIME24 and HumanEval. That can be true at the task-score level while still being false at the tensor or distribution level. At 2-4 bits plus stored scales, there is no mathematical reason to expect fp16-equivalent cache reconstruction. What KVarN can do is move the errors into less damaging shapes and preserve the task behavior better than older cache methods.

For this article, the main metric is mean KLD on Q5_K_S 64k, with 99.9% KLD as the tail check and IQ4_XS as a tie-breaker. If a row only wins PPL, I do not count that as much.

6. The Main Result

Here is the core Q5_K_S 64k comparison. The table is deliberately smaller than the full reference data (available at the end of the article), because this is the part that explains the curve shift.

kvarn4-kvarn4 is the headline. It almost matches q5_1 on mean KLD, beats q5_0, and uses less memory than q4_0. The q5_1 edge is real but narrow: 0.002911 versus 0.002974. In exchange, q5_1 uses 37.5% of bf16 KV while kvarn4-kvarn4 uses 27.9%. For a VRAM-constrained long-context setup, that is not a practical win for q5_1, it's a quality tie-breaker for people with memory to spare.

kvarn4-kvarn3 is an interesting middle point. It is not simply "smaller q4_0". It sits between q4_0 and q5_0-q4_0: worse than q5_0-q4_0 on the two 64k mean-KLD rows, basically tied on the 128k IQ4_XS row, and consistently better than q4_0. Its memory footprint is lower than both: 24.8% on the 64k runs, versus 28.1% for q4_0 and 31.3% for q5_0-q4_0.

kvarn3-kvarn3 is the compact sweet spot if you were previously looking at turbo3_tcq. On the main row it uses 21.7% instead of 20.3% of bf16 KV, but the mean KLD drops from 0.007978 to 0.005349 and the 99.9% KLD drops from 0.227104 to 0.168135. That is a large quality improvement for a small memory increase.

kvarn2-kvarn2 is not the revolution. It is useful as the low endpoint, and it beats turbo2_tcq on Q5 mean KLD, but the tails are not clean enough to make it the row I would recommend first.

7. The K/V Matrix

The full 3x3 matrix shows why KVarN should not be treated as a single symmetric setting. The first rule is familiar: K matters more than V. At the same total size, reducing K hurts more than reducing V.

The second rule is more important: do not overreact and dump all the budget into K. At 21.7% memory, kvarn3-kvarn3 crushes both asymmetric extremes. kvarn2-kvarn4 damages K too much, while kvarn4-kvarn2 damages V too much. K deserves priority, but V cannot be abandoned.

That is why the two settings that matter most are kvarn4-kvarn4 and kvarn4-kvarn3. The first gives the q5-like result. The second keeps K at 4 bits, trims V to 3, and creates a memory tier that ordinary q4/q5 cache quantization did not have.

8. TurboQuant And TCQ

The turbo comparison is where KVarN looks most like the "TurboQuant that was promised". That is not a formal claim about the TurboQuant paper, but a practical read of this benchmark. The old turbo modes were attractive because they reached memory sizes normal q4/q5 could not. The problem was quality leakage. But KVarN reaches the same broad memory bands with much better KLD.

On the main Q5 row, kvarn4-kvarn3 is smaller than turbo4 and better: 24.8% memory and 0.003824 mean KLD versus 25.8% and 0.004760. On the IQ4_XS rows, the same comparison holds directionally. kvarn3-kvarn3 is slightly larger than turbo3_tcq, but much better: 0.005349 versus 0.007978 mean KLD on Q5, with a much better 99.9% tail.

The 2-bit end is less clean. kvarn2-kvarn2 beats turbo2_tcq on Q5 mean KLD, but it's... bad. The tails are still large, and the IQ4_XS checks are noisy enough that I would treat it as an emergency compression setting, not the main result.

9. What This Does Not Prove

First, this does not prove KVarN has fp16 precision. It obviously does not in KLD. kvarn4-kvarn4 is around the q5 cache tier, not around bf16. That is still strong because it gets there at q4-class memory, but the distribution has moved. If an article says "fp16-level" without specifying task score, not tensor fidelity or KLD, it is being loose with the phrase.

Second, this does not prove the paper's reasoning-benchmark results. The paper is about decode-time accumulation and reports results on generative tasks such as MATH500, AIME24, HumanEval and IFEval. This page uses Wikitext PPL and KL divergence against stored bf16 baseline logits. KLD is a strong local fidelity signal, and it is much more sensitive than PPL, but it is not a replacement for running the actual long-generation tasks.

Third, this does not settle speed. The current BeeLlama implementation is too raw, and these tok/s values are prompt-processing numbers rather than generation numbers. The preview path allocates the right cache shapes and produces meaningful quality data, but it is not the final optimized kernel. I would not use today's implementation neither to reject KVarN nor to claim a speedup.

10. Practical Takeaway

For VRAM-constrained long context, KVarN looks like a real frontier shift. The old sub-6-bit recommendation was a ladder of compromises: q5 if you could afford it, q4 if you needed memory, turbo3_tcq if you needed even more memory and accepted quality loss. KVarN rearranges that.

If memory allows it, kvarn4-kvarn4 is the row I would test first. It is the cleanest q5-class-at-q4-memory result. If memory is tighter, kvarn4-kvarn3 is the interesting new middle tier: below q4 memory, above q4 fidelity, approaching q5_0-q4_0. If the goal is to beat turbo-class memory without turbo-class error, kvarn3-kvarn3 is the better compact candidate.

11. Benchmark Data

Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

KL Divergence

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

12. Reference Data

Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

KL Divergence

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k