KV Cache Size Calculator

Estimate per-device cache memory for tensor-parallel inference.

Live estimate

Model & workload

Model-specific runtime options appear here.

Memory footprint

Per device size --
= -- GB
Per sequence --
Per token --
Bytes per element --
Total cache size all devices --
Calculation --

--

Model details