← Rachel's Space

Where each LLM speed and memory technique fits

Training vs. inference, prefill vs. decode, and how much VRAM each lever saves.

LLM pipeline stagesBuild-time lane, request path lane, and GPU engine lane with animated tokens flowing from gateway to decode. Build time (offline, before serving) Request path (before the GPU) GPU engine (weights + KV live in VRAM) ArchitectureGQA, MLA, SWA, MoE Train / tuneQAT, distill, fp8 CompressPrune, PTQ, KV quant Eval gateFlip rate, rollback GatewayBudgets, fallback Cache + trimSemantic, compress RouterSmall vs big, cascade SchedulerContinuous batching PrefillCompute-bound KV cacheVRAM capacity Decode ↻Bandwidth-bound Scale + streamTP, PP, EP, replicas
speed VRAM cost Click a box or filter, then a technique for details
Pick a technique to see where it runs, which bottleneck it fixes, and its cost.
VRAM and decode-speed lab
8B dense model, 32 layers, head dim 128, one 80 GB GPU at 3.35 TB/s. Rough first-order estimates, not benchmarks.
Context
Batch (seqs)
Weights KV cache Fragmentation waste Activations / runtime | 80 GB limit
VRAM per GPU
KV per token
Decode, per seq
Total throughput

Sources: neurarch-ai/awesome-llm-system-design topics 04, 11, 17; rasbt/LLMs-from-scratch ch04.