Home › Articles › KVarN KV Cache: Implementation and Benchmarks

KVarN KV Cache: Implementation and Benchmarks

Tests on Qwen 3.6 27B show how an experimental BeeLlama KVarN preview shifts KV-cache quality down a memory tier: q5 quality at q4 size, q6 at q5, and q8 at q6.

1. What Changed
2. What KVarN Is
3. The BeeLlama Preview
4. The Setup
5. How To Read The Results
6. The Main Result
7. The K/V Matrix
8. TurboQuant And TCQ
9. What This Does Not Prove
10. Practical Takeaway
11. Benchmark Data
    1. Perplexity
    2. KL Divergence
12. Reference Data
    1. Perplexity
    2. KL Divergence
13. KVarN 5/6/8 Follow-Up Benchmarks
    1. Perplexity
    2. KL Divergence
14. KVarN 5/6/8 Follow-Up Analysis
    1. The One-Step Cheaper Ladder
    2. KVarN5: Symmetric Proof, 5/4 Value
    3. KVarN6: 6/6 Proof, 6/5 Value
    4. KVarN8 Mostly Confirms Saturation
    5. K And V Still Need Balance
    6. Updated Preset Ladder
15. KL Divergence Graphs

Disclaimer: this is still a narrow benchmark. The KVarN path tested here is a preview implementation in BeeLlama v0.3.2, not a mature optimized runtime. The precision numbers are the meaningful part of the article, but may be subject to change with future updates. The tok/s column is prompt-processing behavior on a very raw preview of this specific fork, not a final verdict on KVarN's decode or generation speed.

1. What Changed

The original KV-cache benchmark had a pretty clear sub-6-bit story: normal q4/q5 cache quantization was stronger than TurboQuant at similar practical sizes, while TCQ made the 2-3 bit turbo modes less bad. You could save memory, or you could preserve the output distribution, and the tradeoff was mostly where you expected it to be.

KVarN changes that curve. On the main Q5_K_S 64k comparison, kvarn4-kvarn4 uses 27.9% of the bf16 KV-cache footprint and lands around the q5_0/q5_1 quality tier. kvarn4-kvarn3 uses 24.8%, below ordinary q4_0, and still beats q4_0 on mean KLD. kvarn3-kvarn3 sits near the turbo3 memory class and cuts the turbo3_tcq mean KLD by a third.

That does not make KVarN lossless. The bf16 KLD reference is still far away. But for VRAM-constrained long-context setups, "q5-ish quality at q4-ish memory" is the result people are happy to see for low-bit KV-cache methods. Where TurboQuant promised that kind of shift, KVarN is the first thing in these benches that actually looks like it.

The 5/6/8 follow-up at the end widens that result into a cleaner ladder. kvarn5-kvarn5 behaves like a q6-class row at q5-class memory, while kvarn6-kvarn6 reaches the q8-class ceiling around q6 memory. That is the stronger claim: KVarN is not merely good for its bit width, it often matches the next ordinary cache tier while paying for the cheaper one.

2. What KVarN Is

KVarN is a KV-cache quantization method from the paper KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks. The paper's central claim is that current KV-cache quantizers make bad token-scale errors during autoregressive decoding, and those errors accumulate over time, while KVarN reduces the outlier errors that matter most.

The mechanism has two parts. First, KVarN applies a Hadamard rotation in the channel dimension, which spreads outlier energy across coordinates before quantization. That part is familiar from QuaRot-style weight work and from the broader low-bit KV-cache literature. Second, it applies dual-axis variance normalization across both dimensions of a KV tile, so both token and channel scale variation are brought under control before round-to-nearest quantization. The paper describes this as a Sinkhorn-inspired variance-normalization step.

In plain terms: ordinary per-axis quantization can preserve one shape of scale variation and still blow up a few token norms. KVarN spends extra metadata on a second scale so the quantizer can keep those token magnitudes closer to where they were. The paper argues that the worst few percent of quantization errors do disproportionate damage, especially in decode-heavy reasoning tasks, so suppressing those outliers matters more than lowering average reconstruction error everywhere.

3. The BeeLlama Preview

BeeLlama is my llama.cpp fork for long-context experiments. The earlier benchmark used it for q6_0, TurboQuant and TCQ cache modes. This KVarN run is the same kind of fork-level experiment, but the KVarN path is much younger. The first pass tested experimental cache types named kvarn2, kvarn3 and kvarn4, plus asymmetric K/V pairs; the follow-up extends the same path to kvarn5, kvarn6 and kvarn8.

The BeeLlama implementation tested here follows the same broad KVarN structure: 128-token groups, 2/3/4-bit payloads, Hadamard rotation, iterative variance normalization, and stored scale/zero-point metadata. The measured sizes in this article are from the actual BeeLlama run logs, which matters because this preview stores auxiliary values as fp16 records, while the paper reports a more optimized accounting with lower auxiliary precision.

KVarN is also not just one more GGUF-style cache quant like q8_0, q5_0, q4_0, or the turbo types. In BeeLlama, kvarn2, kvarn3, and kvarn4 are CLI pseudo-types that select a separate structured KVarN cache backend. The underlying cache tensors are kept as an fp16 staging path plus KVarN records, and the KVarN configuration is stored separately from normal type_k/type_v because each record spans a full 128-token K/V tile.

That is why there are no rows like q8_0-kvarn4 in the current preview. They are not impossible in principle, but they would require a real hybrid-cache architecture: one side allocated and served by the normal KV path, the other side by KVarN, with attention graph routing, CUDA kernels, state save/load, rollback, prompt cache, seq_cp/seq_rm, DFlash backup, SWA/iSWA, and multi-sequence behavior all updated for split ownership. That would be quite complex to implement, and it is unclear whether the partial compression would be worth the extra risk.

Any speed claims also get a caveat. The paper's speed story is about an optimized vLLM-oriented decode path, while the tok/s values in this article are prompt-processing throughput from llama-perplexity. BeeLlama's current path is a raw llama.cpp integration with custom KVarN store/materialize operations and conservative memory handling. If it is slower in prompt processing today, that is operationally true for this build, but it says little about optimized generation speed.

4. The Setup

The hardware and model setup matches the original article as closely as possible. Hardware: one RTX 3090 with 24 GB VRAM, Ryzen 7 5700X3D, and 32 GB system RAM. Model: Qwen 3.6 27B. The main comparison is Q5_K_S weights at 64k context, because that is the cleaner weight quant for this purpose. IQ4_XS at 64k and 128k is included mostly as a tie-breaker and noise check; it is useful, but I do not treat it as the primary ranking signal.

The PPL run tested only symmetric KVarN cache types: kvarn4, kvarn3, and kvarn2. The KLD run tested the full 3x3 K/V KVarN matrix: kvarn4-kvarn4, kvarn4-kvarn3, kvarn4-kvarn2, and so on down to kvarn2-kvarn2. KLD also included the control rows from the original benchmark to ensure there's no significant drift from the engine changes.

The follow-up tables at the end use the same three model/context configurations, but only walk the higher KVarN range where the earlier matrix suggested a useful preset could exist: symmetric kvarn5, kvarn6, kvarn8 for PPL, and K≥V pairs for KLD.

All KLD rows are measured against a bf16 KV-cache baseline using llama-perplexity --kl-divergence on Wikitext-2. That is not the paper's pseudo-decode evaluation and it is not an end-to-end reasoning benchmark. It is a local distribution-fidelity test. I like it here because it catches output-distribution movement that PPL tends to hide.

The tok/s column is prompt-processing/prefill throughput from the benchmark run, not generation speed. The table sizes are the actual KV-cache sizes reported by the run logs. KVarN could still behave differently during decode, including faster after optimized kernels; these tables do not decide that question.

5. How To Read The Results

PPL makes KVarN look nearly lossless at the top settings. On Q5_K_S 64k, bf16 PPL is 5.4800 and kvarn4 is 5.4807. That looks like nothing happened. But KLD says something did happen: bf16 mean KLD is 0.000375, while kvarn4-kvarn4 is 0.002974. That is not fp16 precision at the distribution level.

That distinction matters. The KVarN paper reports near-fp16 behavior on generative benchmarks such as MATH500, AIME24 and HumanEval. That can be true at the task-score level while still being false at the tensor or distribution level. At 2-4 bits plus stored scales, there is no mathematical reason to expect fp16-equivalent cache reconstruction. What KVarN can do is move the errors into less damaging shapes and preserve the task behavior better than older cache methods.

For this article, the main metric is mean KLD on Q5_K_S 64k, with 99.9% KLD as the tail check and IQ4_XS as a tie-breaker. If a row only wins PPL, I do not count that as much.

6. The Main Result

Here is the core Q5_K_S 64k comparison. The table is deliberately smaller than the full reference data (available at the end of the article), because this is the part that explains the curve shift.

kvarn4-kvarn4 is the headline. It almost matches q5_1 on mean KLD, beats q5_0, and uses less memory than q4_0. The q5_1 edge is real but narrow: 0.002911 versus 0.002974. In exchange, q5_1 uses 37.5% of bf16 KV while kvarn4-kvarn4 uses 27.9%. For a VRAM-constrained long-context setup, that is not a practical win for q5_1, it's a quality tie-breaker for people with memory to spare.

kvarn4-kvarn3 is an interesting middle point. It is not simply "smaller q4_0". It sits between q4_0 and q5_0-q4_0: worse than q5_0-q4_0 on the two 64k mean-KLD rows, basically tied on the 128k IQ4_XS row, and consistently better than q4_0. Its memory footprint is lower than both: 24.8% on the 64k runs, versus 28.1% for q4_0 and 31.3% for q5_0-q4_0.

kvarn3-kvarn3 is the compact sweet spot if you were previously looking at turbo3_tcq. On the main row it uses 21.7% instead of 20.3% of bf16 KV, but the mean KLD drops from 0.007978 to 0.005349 and the 99.9% KLD drops from 0.227104 to 0.168135. That is a large quality improvement for a small memory increase.

kvarn2-kvarn2 is not the revolution. It is useful as the low endpoint, and it beats turbo2_tcq on Q5 mean KLD, but the tails are not clean enough to make it the row I would recommend first.

7. The K/V Matrix

The full 3x3 matrix shows why KVarN should not be treated as a single symmetric setting. The first rule is familiar: K matters more than V. At the same total size, reducing K hurts more than reducing V.

The second rule is more important: do not overreact and dump all the budget into K. At 21.7% memory, kvarn3-kvarn3 crushes both asymmetric extremes. kvarn2-kvarn4 damages K too much, while kvarn4-kvarn2 damages V too much. K deserves priority, but V cannot be abandoned.

That is why the two settings that matter most are kvarn4-kvarn4 and kvarn4-kvarn3. The first gives the q5-like result. The second keeps K at 4 bits, trims V to 3, and creates a memory tier that ordinary q4/q5 cache quantization did not have.

8. TurboQuant And TCQ

The turbo comparison is where KVarN looks most like the "TurboQuant that was promised". That is not a formal claim about the TurboQuant paper, but a practical read of this benchmark. The old turbo modes were attractive because they reached memory sizes normal q4/q5 could not. The problem was quality leakage. But KVarN reaches the same broad memory bands with much better KLD.

On the main Q5 row, kvarn4-kvarn3 is smaller than turbo4 and better: 24.8% memory and 0.003824 mean KLD versus 25.8% and 0.004760. On the IQ4_XS rows, the same comparison holds directionally. kvarn3-kvarn3 is slightly larger than turbo3_tcq, but much better: 0.005349 versus 0.007978 mean KLD on Q5, with a much better 99.9% tail.

The 2-bit end is less clean. kvarn2-kvarn2 beats turbo2_tcq on Q5 mean KLD, but it's... bad. The tails are still large, and the IQ4_XS checks are noisy enough that I would treat it as an emergency compression setting, not the main result.

9. What This Does Not Prove

First, this does not prove KVarN has fp16 precision. It obviously does not in KLD. kvarn4-kvarn4 is around the q5 cache tier, not around bf16. That is still strong because it gets there at q4-class memory, but the distribution has moved. If an article says "fp16-level" without specifying task score, not tensor fidelity or KLD, it is being loose with the phrase.

Second, this does not prove the paper's reasoning-benchmark results. The paper is about decode-time accumulation and reports results on generative tasks such as MATH500, AIME24, HumanEval and IFEval. This page uses Wikitext PPL and KL divergence against stored bf16 baseline logits. KLD is a strong local fidelity signal, and it is much more sensitive than PPL, but it is not a replacement for running the actual long-generation tasks.

Third, this does not settle speed. The current BeeLlama implementation is too raw, and these tok/s values are prompt-processing numbers rather than generation numbers. The preview path allocates the right cache shapes and produces meaningful quality data, but it is not the final optimized kernel. I would not use today's implementation neither to reject KVarN nor to claim a speedup.

10. Practical Takeaway

For VRAM-constrained long context, KVarN looks like a real frontier shift. The old sub-6-bit recommendation was a ladder of compromises: q5 if you could afford it, q4 if you needed memory, turbo3_tcq if you needed even more memory and accepted quality loss. KVarN rearranges that.

In the original 2/3/4-bit sweep, kvarn4-kvarn4 is the cleanest result: q5-class fidelity at q4-class memory. If memory is tighter, kvarn4-kvarn3 creates a useful below-q4 tier, and kvarn3-kvarn3 is the better compact candidate if you were previously looking at turbo3_tcq.

The follow-up changes the higher-end recommendation. The symmetric rows are the clean proof of the tier shift, but the near-balanced asymmetric rows are often the practical value presets. If the cache budget can reach the mid-30% range, kvarn5-kvarn5 is the q6-class proof row and kvarn5-kvarn4 is the tighter value row. If the goal is the q8 fidelity tier without the q8 footprint, kvarn6-kvarn6 is the cleaner row, while kvarn6-kvarn5 is the aggressive smaller variant. kvarn8-kvarn8 mostly belongs as a ceiling check, not the row to reach for first.

11. Benchmark Data

Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

KL Divergence

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

12. Reference Data

Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

KL Divergence

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

13. KVarN 5/6/8 Follow-Up Benchmarks

This follow-up was run on updated BeeLlama v0.3.2 Preview that extends the KVarN matrix above the original 2/3/4-bit sweep. The same three model/context configurations are used: Q5_K_S 64k, IQ4_XS 64k, and IQ4_XS 128k, each measured against its bf16 KV-cache baseline.

Perplexity

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

These PPL rows are only a sanity check. As in the main article, PPL barely moves across the useful cache range, so the recommendation should come from the KLD tables below rather than from the median PPL column.

KL Divergence

Q5_K_S 64k

IQ4_XS 64k

IQ4_XS 128k

14. KVarN 5/6/8 Follow-Up Analysis

The first KVarN run already had the important low-bit result: kvarn4-kvarn4 was not merely a better q4 row, it reached the normal q5 tier while using less memory than ordinary q4_0. The new rows repeat that shape higher up the ladder. The symmetric rows prove the shift cleanly, and the asymmetric pairs are the ones with highest value per MiB.

There are two different questions: one is whether KVarN can match a higher ordinary quant tier at lower memory, and the other is which pairs you should actually run. kvarn5-kvarn5 is the clean q6-class proof, but kvarn5-kvarn4 is the sharper mid-tier value row. kvarn6-kvarn6 is the conservative q8-class proof, while kvarn6-kvarn5 is a more aggressive variant.

14.1 The One-Step Cheaper Ladder

The clean read is that KVarN shifts the useful cache ladder down by about one ordinary memory tier. This is not true for every possible K/V split. It holds for balanced rows and for the one-bit asymmetric rows that keep K slightly higher than V. The table below separates proof rows from value rows so the result does not get flattened into a symmetric-only story.

KVarN is not just better within its own nominal bit width. In the useful range, it often gives the quality tier above the memory tier it occupies. The row a user picks, however, may be one step asymmetric rather than symmetric.

14.2 KVarN5: Symmetric Proof, 5/4 Value

kvarn5-kvarn5 should not be sold as q8-class, but it is the first row that looks q6-class while staying in the q5 memory band. The Q5_K_S 64k row is mixed because mean KLD is slightly worse than q6_0, 0.002705 vs 0.002614, but the 99.9% tail is better, 0.083457 vs 0.090800.

The more practical mid-tier row is kvarn5-kvarn4. On the main Q5_K_S row, it uses 31.1% of the bf16 KV cache and scores 0.002824 / 0.093313. That is better than q5_1 at 37.5%, better than q5_0 at 34.4%, and better than the old asymmetric q5_0-q4_0 at 31.3%. Overall, it's a very solid mid-tier recommendation.

So kvarn5 has two jobs. kvarn5-kvarn5 proves the q6-at-q5-memory claim. kvarn5-kvarn4 is the row I would highlight when talking about value per size, because it sits around the old q5_0-q4_0 memory band while beating ordinary q5.

14.3 KVarN6: 6/6 Proof, 6/5 Value

The q8 comparison starts to become real at kvarn6. Against symmetric q8_0, kvarn6-kvarn6 is effectively tied on the main Q5_K_S row: 0.002338 / 0.078797 vs 0.002328 / 0.078709, but it comes at 40.4% of bf16 KV instead of 53.1%.

Against optimized usual quant rows, kvarn6-kvarn6 is the clean claim. It beats q8_0-q6_0 on Q5_K_S 64k and IQ4_XS 64k while using about 40% instead of 47%. kvarn6-kvarn5 is not as clean on the main Q5 row because its mean KLD is worse than q8_0-q6_0, 0.002602 vs 0.002499, but its 99.9% tail is better, 0.079818 vs 0.081616, and it uses 37.3% instead of 46.9%. That makes it a great aggressive value row on the high end.

14.4 KVarN8 Mostly Confirms Saturation

kvarn8-kvarn8 is sort of meaningless. On Q5_K_S 64k, it scores 0.002361 / 0.076809, while ordinary q8_0 scores 0.002328 / 0.078709. On IQ4_XS 128k, it scores 0.000387 / 0.012994 against q8_0 at 0.000482 / 0.014951. Those are q8-class numbers, and sometimes better than q8, but the footprint is also q8-class: 52.9% at 64k and 52.6% at 128k.

The asymmetric kvarn8 rows do not change the recommendation much. kvarn8-kvarn5 can beat ordinary q8_0 on the IQ4 rows at lower memory, but kvarn6-kvarn6 and kvarn6-kvarn5 already reach the same useful tier with less cache. Once kvarn6 has reached q8-class fidelity, extra K bits mostly become a diagnostic check rather than a preset.

That makes kvarn8 useful mostly as a ceiling check. It shows that the KVarN path can sit at the top of the measured distribution-fidelity range, but it also shows why that top row is not necessary. If kvarn6-kvarn6 already reaches the q8 tier at roughly q6 memory, kvarn8-kvarn8 mostly answers a diagnostic question rather than a preset question.

14.5 K And V Still Need Balance

The follow-up also puts a limit on the old K-first rule. K still deserves priority when there is one extra bit to spend, but dumping bits into K while starving V stops working quickly. At the same 34.2% footprint on Q5_K_S 64k, kvarn5-kvarn5 scores 0.002705 / 0.083457, kvarn6-kvarn4 slips to 0.002831 / 0.091507, and kvarn8-kvarn2 collapses to 0.009494 / 0.325652. More K does not rescue a bad V side.

The practical rule is simple: K can take the extra bit, but V should stay within one bit unless the row has data proving otherwise. The useful asymmetric rows are near-balanced, especially kvarn4-kvarn3, kvarn5-kvarn4, and kvarn6-kvarn5. The rows with a two-bit or larger gap are mostly negative evidence.

14.6 Updated Preset Ladder

With the follow-up included, the preset ladder changes more than the original 2/3/4-bit article suggested. The old ladder had kvarn4-kvarn4 at the top because that was the largest tested KVarN row. The higher-bit sweep makes kvarn4-kvarn4 a memory-saving row, not the quality ceiling.

In these runs, KVarN shifts the practical ladder down by roughly one memory tier: q5 quality at q4 memory, q6 quality at q5 memory, and q8-class quality around q6 memory. The symmetric rows show the ladder, and the one-bit asymmetric rows are often where the best value per size sits. kvarn8 reaches the ceiling, but kvarn6 is the reason the ceiling matters.

15. KL Divergence Graphs

The graphs below plot mean and 99.9% KL divergence against KV-cache size, with separate plots for the full range and the sub-4-bit range. Each point is one cache configuration from the tables above.