+<article class="kv-calc"><nav aria-label="Breadcrumb"><a href="/">ml0x</a> / <a href="/calculators/">Calculators</a> / <span>KV cache calculator</span></nav><h1>KV cache calculator</h1><p class="byline">By <a href="https://zovo.one/lifetime?utm_source=ml0x&utm_medium=inline&utm_campaign=calculators-kv-cache-size-calculator" rel="author">Michael Lip</a></p><aside class="s2o" data-cro="sprint-20261115-ml0x-triage-top" data-mr-placement="s2_top_kv-cache" aria-label="Training Run Triage offer"><style>.s2o{margin:14px 0 18px;padding:12px 14px;border:1px solid var(--line-strong,#D8D8D6);border-left:4px solid var(--ink,#1A1A1A);border-radius:8px;background:var(--surface,#fff);color:var(--ink,#1A1A1A);font:15px/1.5 var(--font-body,sans-serif);max-width:760px}.s2o p{margin:0 0 8px}.s2o-a{display:flex;flex-wrap:wrap;align-items:center;gap:6px 18px}.s2o-btn{display:inline-flex;align-items:center;min-height:48px;padding:0 18px;border-radius:999px;background:var(--ink,#1A1A1A);color:#fff!important;font-weight:600;text-decoration:none}.s2o-btn:hover{background:#333}.s2o-a a:not(.s2o-btn){display:inline-flex;align-items:center;min-height:48px;font-weight:600}.s2o .s2o-n{margin:4px 0 0;font-size:13px;color:var(--ink-muted,#6a6a6a)}@media print{.s2o{display:none}}</style><p><b>Sizing GPU memory for a real run?</b> Training Run Triage has the full memory guide (the four terms, what moves with batch size), offline KV cache and model memory calculators, a memory budget script and 6 more guides.</p><p class="s2o-a"><span data-cro="sprint-20261116-direct"><a class="s2o-btn" data-cta="s2_top_buy_triage_kv-cache" href="https://buy.stripe.com/bJe00kbYK4Jyccd8QR43S0s" rel="noopener">Buy Training Run Triage, $99 once</a></span><a data-cta="s2_top_triage_page_kv-cache" href="/triage/">See what is inside</a></p><div class="s3-paybox" data-cro="sprint-20261116-paybox"><p><strong>$99</strong> once - Training Run Triage: 7 PDF guides (NaN loss, learning rate and schedule, optimizer, metrics that disagree, GPU memory) and 4 scripts, with a source for every default</p><p>Download page immediately after payment, no account or licence key · 60-day refund, no questions asked, and you keep the files</p><p>Secure checkout by Stripe - the page shows BeLikeNative, our company account</p></div><form class="s3-notready" data-cro="sprint-20261116-notready" data-offer="ml0x-notready" data-price="99" novalidate><label>Not ready? Email me this link <input type="email" name="email" required placeholder="you@email.com" autocomplete="email"></label><input type="text" name="hp" value="" tabindex="-1" autocomplete="off" aria-hidden="true" style="position:absolute;left:-9999px;width:1px;height:1px;opacity:0"><button type="submit">Email me</button><small>We email you this link once and one reminder. Unsubscribe any time.</small><span class="s3-nr-msg" role="status"></span></form><style>.s3-paybox{margin:.5rem 0;padding:.5rem .75rem;border:1px solid rgba(0,0,0,.15);border-radius:8px;font-size:13px;line-height:1.4}.s3-paybox p{margin:.15rem 0}.s3-notready{font-size:13px;margin:.25rem 0 .75rem;position:relative}.s3-notready input[type=email]{max-width:200px;padding:4px 6px;font-size:14px}.s3-notready small{display:block;opacity:.75}</style><script>(function(){if(window.__s3nr)return;window.__s3nr=1;document.querySelectorAll('form.s3-notready').forEach(function(f){f.addEventListener('submit',function(e){e.preventDefault();var hp=f.querySelector('[name=hp]'),em=f.querySelector('[name=email]'),msg=f.querySelector('.s3-nr-msg');if(hp&&hp.value)return;var email=(em&&em.value||'').trim();if(!email||(em.checkValidity&&!em.checkValidity())){if(msg)msg.textContent='Please enter a valid email.';return;}var oid=f.getAttribute('data-offer')||'ml0x-notready',price=f.getAttribute('data-price')||'';fetch('https://fleet-leads.lipmichal.workers.dev/lead',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({email:email,offer_id:oid,page:location.href,price:price,hp:''})}).then(function(r){return r.ok?r.json().catch(function(){return{ok:true};}):{ok:false};}).then(function(j){if(j&&j.ok){if(msg)msg.textContent='Check your inbox to confirm.';f.reset();if(typeof window.gtag==='function')window.gtag('event','generate_lead',{site:'ml0x.com',offer_id:oid,price:Number(price),currency:'USD'});}else if(msg){msg.textContent='That did not work. Please try again.';}},function(){if(msg)msg.textContent='That did not work. Please try again.';});});});})();</script><p class="s2o-n">Stripe checkout, then a download page. 60-day refund, and you keep the files.</p></aside><section id="tool"><h2>Size the cache for your model</h2><div class="kvc-grid"><label for="kvc-model">Model</label><select id="kvc-model"><option value="llama31_8b" data-bpt="131072" data-gqa="4">Llama 3.1 8B</option><option value="llama32_1b" data-bpt="32768" data-gqa="4">Llama 3.2 1B</option><option value="qwen25_0_5b" data-bpt="12288" data-gqa="7">Qwen2.5-0.5B</option><option value="qwen25_1_5b" data-bpt="28672" data-gqa="6">Qwen2.5-1.5B</option><option value="qwen25_3b" data-bpt="36864" data-gqa="8">Qwen2.5-3B</option><option value="qwen25_7b" data-bpt="57344" data-gqa="7">Qwen2.5-7B</option><option value="qwen25_14b" data-bpt="196608" data-gqa="5">Qwen2.5-14B</option><option value="qwen25_32b" data-bpt="262144" data-gqa="5">Qwen2.5-32B</option><option value="qwen25_72b" data-bpt="327680" data-gqa="8">Qwen2.5-72B</option><option value="qwen3_8b" data-bpt="147456" data-gqa="4">Qwen3-8B</option><option value="qwen3_14b" data-bpt="163840" data-gqa="5">Qwen3-14B</option><option value="qwen3_32b" data-bpt="262144" data-gqa="8">Qwen3-32B</option><option value="mistral_7b" data-bpt="131072" data-gqa="4">Mistral 7B v0.1</option><option value="mistral_nemo_12b" data-bpt="163840" data-gqa="4">Mistral NeMo Base 2407</option><option value="mixtral_8x7b" data-bpt="131072" data-gqa="4">Mixtral 8x7B v0.1</option><option value="phi3_mini" data-bpt="393216" data-gqa="1">Phi-3-mini-4k-instruct</option><option value="phi4" data-bpt="204800" data-gqa="4">phi-4</option><option value="olmo2_7b" data-bpt="524288" data-gqa="1">OLMo-2-1124-7B</option><option value="smollm2_1_7b" data-bpt="196608" data-gqa="1">SmolLM2-1.7B</option><option value="custom">Custom model (enter your own config values)</option></select><label for="kvc-prec">KV precision</label><select id="kvc-prec"><option value="2">bf16 or fp16 (two bytes per element)</option><option value="1">fp8 KV cache (one byte per element)</option><option value="0.5">int4 KV cache (half a byte per element)</option></select><label for="kvc-ctx">Context length in tokens</label><input id="kvc-ctx" type="number" min="1" max="100000000" step="1" value="131072"><label for="kvc-batch">Concurrent sequences</label><input id="kvc-batch" type="number" min="1" max="1000000" step="1" value="1"></div><fieldset id="kvc-custom" hidden><legend>Custom model</legend><p>Any model can be sized here, including one absent from the list above. Open its <code>config.json</code> and copy three keys across.</p><label for="kvc-layers">num_hidden_layers</label><input id="kvc-layers" type="number" min="1" max="10000" step="1"><label for="kvc-kvheads">num_key_value_heads</label><input id="kvc-kvheads" type="number" min="1" max="100000" step="1"><label for="kvc-headdim">head_dim (use the published field if the file has one, otherwise hidden_size / num_attention_heads)</label><input id="kvc-headdim" type="number" min="1" max="100000" step="1"><label for="kvc-attnheads">num_attention_heads (optional, used only to show you what the wrong method would have said)</label><input id="kvc-attnheads" type="number" min="1" max="100000" step="1"></fieldset><div id="kvc-out" role="status" aria-live="polite"></div><details><summary>Paste a config.json instead</summary><p>Paste the raw file. The parser reads <code>num_hidden_layers</code>, <code>num_key_value_heads</code> and <code>head_dim</code>, falls back to <code>hidden_size</code> divided by <code>num_attention_heads</code> when no <code>head_dim</code> is published, and refuses architectures the formula does not describe rather than approximating them.</p><textarea id="kvc-json" rows="6" placeholder="{ "num_hidden_layers": ..., "num_key_value_heads": ..., "head_dim": ... }"></textarea><button type="button" id="kvc-parse">Read this config</button><p id="kvc-note" role="status" aria-live="polite"></p></details><p class="disclaimer">All computation runs client side. Nothing you enter leaves your browser.</p></section><section><h2>The formula, and where it comes from</h2><pre><code>kv_bytes = 2 (one K tensor, one V tensor)
0 commit comments