Home › Articles › KV Cache Quantization on Gemma 4 31B: Non-QAT vs QAT

KV Cache Quantization on Gemma 4 31B: Non-QAT vs QAT

Does QAT make Gemma 4 31B usable with a quantized KV cache? Standard tail-free cache establishes the result before a precision tail and KVarN test how far it extends.

1. 1. What Is Being Compared
2. 2. Standard Cache
3. 3. QAT Moves The Usable Boundary
4. 4. Precision Tail
5. 5. KVarN
6. 6. The Standard Tail-Free Answer
7. 7. Limits And Secondary Comparisons
8. 8. Method And Full Benchmark Data
   1. 8.1 Gemma 4 31B Q4_0 (non-QAT) 32k
   2. 8.2 Gemma 4 31B QAT Q4_0 32k

1. What Is Being Compared

The question here is whether a QAT model merely survives low-bit weights, or also becomes more tolerant of a separately quantized KV cache. In a typical fake-quantization workflow, the forward pass rounds and clamps values to imitate a deployment format, then dequantizes them for the remaining computation. Optimization still updates floating-point tensors and typically uses a straight-through estimator to approximate gradients through nondifferentiable rounding. Unlike post-training quantization, this gives the model an opportunity to adapt its weights and internal values to the simulated error.

Google describes Gemma 4 QAT in those broad terms, but does not document KV-cache quantization as part of this checkpoint's training objective. Still, a cache benefit can emerge if QAT changes those tensors or the model's downstream sensitivity to their perturbation, but this benchmark cannot identify the internal cause. It measures the checkpoint's end-to-end response; it does not show that Google trained a quantized cache.

The two Q4_0 files come from different releases. The file used here as non-QAT, gemma-4-31B-it-Q4_0.gguf, was obtained from Unsloth's Gemma 4 31B GGUF repository. The QAT side is Google's gemma-4-31B-it-qat-q4_0-gguf. Both have Q4_0 weights, so the primary comparison is between checkpoints at the same nominal weight width.

The benchmark runs on BeeLlama v0.4.3 rather than upstream llama.cpp, because BeeLlama exposes the additional cache configurations: standard cache quantization, a configurable precision tail, and KVarN. Relative to each checkpoint's own BF16-cache run, standard quantization with no tail changes only the runtime cache type. A precision tail also changes which tokens stay exact, and KVarN changes the quantizer and requires an exact recent group.

Mean KLD measures the average shift in the output distribution; same-top agreement counts positions where a cache row and its BF16-cache control choose the same highest-probability token. Those BF16 controls regenerate slightly differently: mean KLD is 0.000000 for non-QAT and 0.000035 for QAT, while same-top agreement is 99.998% and 99.967%. The discrepancy matters as a measurement floor, but it is too small to explain the result below. Even the smallest QAT standard-cache mean KLD, 0.015078 at Q8, is about 431 times that floor.

2. Standard Cache

This is the cleanest test, where K and V use the same standard type. If QAT matters to cache sensitivity, it has to appear here before either of BeeLlama's specialized cache features can take credit.

The Q8 row establishes the scale of the difference. Cache quantization is already severe here for the non-QAT checkpoint: its mean KLD reaches 0.305575 and its highest-probability token agrees with the BF16-cache run at only 85.115% of measured positions. The QAT checkpoint at the same Q8 cache records 0.015078 mean KLD and 94.870% same-top agreement. That gap is far beyond the control floor.

The benefit persists as the cache narrows, but it does not stay proportionally constant. The non-QAT/QAT mean-KLD ratio contracts from 20.3x at Q8 to 9.7x at Q4 and 3.9x at Q2. QAT makes the checkpoint more cache-tolerant; it does not make cache width irrelevant. At Q2, the QAT row still disagrees with its BF16-cache baseline on 51.341% of measured positions.

3. QAT Moves The Usable Boundary

QAT Q4 against non-QAT Q8 asks whether the checkpoint difference can buy a whole cache tier. On mean KLD and same-top agreement, it can. QAT Q4 records 0.090504 and 86.337% at 1057.50 MiB; non-QAT Q8 records 0.305575 and 85.115% at 1997.50 MiB. The QAT Q4 cache is 940.00 MiB smaller, yet it has 3.4 times less mean divergence and slightly more argmax agreement.

The median adds an important qualification. QAT Q4 records 0.041659 median KLD, above non-QAT Q8 at 0.019254, even though its mean and upper percentiles are much lower. The typical small shift is larger on QAT Q4, while non-QAT Q8 produces the heavier error tail and more argmax changes. QAT Q4 therefore does not dominate non-QAT Q8 on every statistic, but it wins the average-distribution and decision comparisons that drive the practical claim.

That cross-tier result is the first reason to answer the article's question with a qualified yes. QAT has moved standard Q4 from a row that loses 28.368 percentage points of same-top agreement from its control to one that loses 13.630 points, while using barely more than half the memory of non-QAT Q8.

Q3 prevents the conclusion from becoming too broad. Its QAT mean KLD of 0.286372 is also slightly below non-QAT Q8, but same-top agreement is only 73.017%. Matching one aggregate metric does not make the configurations equivalent. The Q4 row clears both tests; Q3 clears only the mean, and Q2 clears neither. In this matrix, QAT moves the plausible standard-cache boundary to Q4, not all the way to the bottom of the ladder.

4. Precision Tail

A precision tail keeps the newest tokens at higher precision while older tokens remain in the selected quantized body. It changes storage precision, not the attention pattern: full-attention layers still see the whole retained context, and sliding-window layers keep their normal window. As the sequence advances, tokens that fall behind the exact suffix move into the quantized body. These standard-cache runs use a 1024-token BF16 tail.

Gemma makes that particular length consequential. 50 of its layers use a 1024-token sliding window, so the tail makes the complete visible window exact in those layers; ten full-attention layers still combine the exact suffix with a much longer quantized body. This is no longer the clean cache-only comparison from section 2. It deliberately privileges recent context, and it consumes additional memory.

The cost begins well before 1024 tokens. In the earlier Gemma Q5_K_S cache matrix, adding only 128 exact tokens to standard Q4 raised persistent cache memory from 697.50 to 808.36 MiB, a 110.86 MiB or 15.9% increase. The equivalent 128-token overlay cost only 8.00 MiB on Qwen in that study. On Gemma, any exact suffix is replicated through 50 sliding-window layers, and low-bit bodies make its fixed cost proportionally worse.

At standard Q4, the tail lowers QAT mean KLD from 0.090504 to 0.025147 and raises same-top agreement from 86.337% to 92.997%. The non-QAT checkpoint improves too, from 0.880436 to 0.445013 and from 71.630% to 81.346%, but it remains far behind. The tail therefore strengthens the QAT result instead of creating it.

The recovery curve also develops a limit. At Q4, the tail removes 72.21% of QAT mean KLD and 49.46% of non-QAT mean KLD. At Q8, where the QAT body is already close to its control, it removes only 6.83%. At Q2, proportional recovery is almost the same for QAT and non-QAT, 67.82% and 67.33%, yet the QAT row still reaches only 70.712% same-top agreement. A large recovery percentage can describe a large improvement from a bad starting point; it is not itself evidence of a good final configuration.

The Q4 tail result looks like the natural recommendation until memory is brought back into the comparison. Standard Q4 grows from 1057.50 MiB at tail 0 to 1600.86 MiB at tail 1024. At that larger footprint, QAT Q4 records 0.025147 mean KLD and 92.997% same-top agreement.

QAT Q6 with no tail uses less memory, 1527.50 MiB, and records better values on both metrics: 0.022552 mean KLD and 93.640% same-top agreement. It dominates Q4 with tail 1024 inside this matrix. If old and recent tokens matter equally, spending the memory on a wider body is the stronger standard-cache choice.

The tail still represents a different policy rather than wasted memory. It buys exact recent context while accepting a narrower old-context body, which can be useful when the current code block, prompt continuation, or tool output matters more than distant context. Teacher-forced WikiText-2 repeatedly scores the next token from its immediate predecessors, so this benchmark is especially responsive to that policy. For Gemma, the 1024-token tail is a specialized recency setting, not the default route to an efficient cache.

5. KVarN

KVarN cannot participate in the article's main tail-free comparison. It rotates K and V within each head, applies variance normalization, and quantizes in groups of 128 tokens. A group is quantized only when its final token arrives, so the newest incomplete group remains exact in F16. BeeLlama therefore floors KVarN's effective tail at one complete group: a request for tail 0 becomes 128*, and a true KVarN tail 0 is impossible.

On Gemma, that minimum has a material cost. The earlier matched-tail study measured standard Q4 at tail 0 at 697.50 MiB, standard Q4 at tail 128 at 808.36 MiB, and KVarN4 at its intrinsic 128* tail at 1008.98 MiB. The first 128 exact tokens alone added 110.86 MiB to the standard cache. KVarN4 then cost another 200.62 MiB at the same suffix length because its Gemma body was larger, leaving the minimum KVarN4 configuration 311.48 MiB, or 44.7%, above tail-free standard Q4. KVarN starts by paying Gemma's expensive exact-suffix tax; it cannot offer an alternative that avoids it.

The present matrix tests KVarN only with a 1024-token F16 tail, so it expands that mandatory exact group until all 50 sliding-window layers hold their complete visible window exactly. It also retains an exact prefix sink in non-sliding-window layers, while the standard comparator uses a BF16 tail. At four bits, KVarN improves the QAT checkpoint from standard Q4 tail 1024's 0.025147 mean KLD and 92.997% same-top agreement to 0.015083 and 95.111%, for 10.00 MiB more. That is a practical family-to-family result, not a pure test of the body quantizer.

The memory result now controls how the quality result should be read. KVarN4 at 1610.86 MiB is 553.36 MiB, or 52.3%, larger than tail-free standard Q4. It is also 83.36 MiB larger than tail-free standard Q6. Its better KLD may come from the KVarN body, the exact sliding window, or both; this matrix cannot separate them. KVarN4 is therefore a specialized recency-focused follow-up, not evidence that KVarN is an efficient default for Gemma.

KVarN8 is the most favorable case because replacing part of an eight-bit body with exact storage has a smaller relative memory penalty than doing so with a narrow body. It uses 2250.86 MiB, 253.36 MiB more than tail-free standard Q8, while moving mean KLD from 0.015078 to 0.010614 and same-top agreement from 94.870% to 95.894%. That can justify a quality-first experiment when memory is secondary. It does not alter the answer to the article's main question: on the non-QAT checkpoint even KVarN4 remains at 0.320820 mean KLD and 84.925% same-top agreement, and KVarN cannot reproduce QAT's tail-free Q4 result.

6. The Standard Tail-Free Answer

The benchmark supports a qualified yes, not a finished deployment verdict. Standard Q4 with no tail is the decisive row: on mean KLD and same-top agreement it beats non-QAT Q8 while using 940.00 MiB less cache, and it reaches that result without exact recent tokens or a specialized quantizer. Its 86.337% same-top agreement still leaves substantial measurable drift, so “usable” here means credible enough for workload testing.

Q4 is the aggressive memory tier, not the automatic default. Q5 cuts its mean KLD by more than half and adds 4.797 percentage points of same-top agreement for 235.00 MiB. Q6 makes another smaller but still useful step for the same 235.00 MiB. The final jump to Q8 costs 470.00 MiB and buys only 0.007474 mean KLD and 1.230 same-top points over Q6. On this evidence, tail-free Q6 is the strongest standard balance; Q4 is the lowest tier with a defensible result.

Q2 remains outside the recommendation. QAT reduces its cache sensitivity substantially, and the tail removes 67.82% of its starting mean KLD, but the final standard Q2 tail-1024 row still agrees with BF16 on only 70.712% of measured top tokens. QAT shifts the boundary; it does not remove one.

7. Limits And Secondary Comparisons

The conclusion concerns cache sensitivity, not overall model quality. Each checkpoint has its own BF16-cache baseline, so a lower KLD says that changing the cache perturbed that checkpoint less. It does not say that the answers are better.

The direct PPL(Q) values expose a related limitation. The BF16-cache rows record 17581.27 for non-QAT and 1704.44 for QAT on this WikiText-2 pass. Unlike KLD, those direct perplexities do not depend on the stored divergence baselines; the large gap confirms that the two checkpoints are behaviorally different before cache quantization is applied.

An adjacent Gemma Q5_K_S matrix provides secondary context. Its non-QAT Q4 cache at 16k records 0.901654 mean KLD and 71.347% same-top agreement, close to this article's 32k non-QAT Q4_0 checkpoint at 0.880436 and 71.630%. The context, model file, and cache allocation differ, so this is not a controlled Q4_0-versus-Q5_K_S comparison. It does show that the QAT effect here is much larger than the difference visible between those two non-QAT runs.

The Qwen weight-quant comparison is a warning against turning that observation into a universal rule. At 64k context and Q4 cache, Q5_K_S records 0.004711 mean KLD while the lower-precision IQ4_XS weights record 0.002759.

This is one teacher-forced WikiText-2 pass per configuration. It does not test coding accuracy, long-range retrieval, instruction retention, tool use, structured output, or autoregressive error accumulation. The tail is especially workload-dependent because teacher forcing gives exact recent tokens a direct role at every scored position. A session that depends on information far behind the 1024-token suffix can behave differently.

8. Method And Full Benchmark Data

The matrix uses one RTX 3090 on Windows 11 and BeeLlama v0.4.3 with CUDA 13.1. Both checkpoints use Q4_0 weights and WikiText-2 raw at 32768 context with batch size 2048, micro-batch size 512, seed 1, FlashAttention, unified KV, no mmap, mlock, and no host offload. KLD is measured against a matching BF16-cache baseline.

The symmetric matrix contains BF16, standard Q8_0 through Q2_0 at tail 0 and 1024, and KVarN8 through KVarN2 at tail 1024. Standard tails use BF16; KVarN tails use F16 and KVarN's group routing. Persistent KV memory is reconstructed from each selected run's non-zero KV and KVarN buffers. Prefill speed is calculated from the 294912 evaluated tokens divided by the prompt-evaluation time reported in the matching run log.

Each checkpoint is measured against its own BF16-cache baseline. Cross-table KLD magnitudes compare sensitivity to cache quantization, not distance between the checkpoints or model quality. KV cache (MiB) is the persistent total from the selected run's runtime buffer accounting, including the full-attention body/tail and Gemma's sliding-window allocation; it is not total GPU VRAM. Tok/s is prefill throughput calculated from total evaluated tokens divided by prompt-evaluation time. Elapsed (s) is wall-clock run time from the persisted benchmark record.

8.1 Gemma 4 31B Q4_0 (non-QAT) 32k

8.2 Gemma 4 31B QAT Q4_0 32k