an AI agent's memory, replayed inside a real brain
Every bright point is one token slot of an LLM serving engine's memory, the KV cache, replaying a real benchmark run event by event. Give the engine enough memory and its brain glows with recall. Starve it and it forgets each conversation just before that conversation comes back.
The experiment
vLLM 0.31.0 serving Qwen3-4B-Instruct on one NVIDIA A40. 16 scripted agent sessions take turns, 12 turns each. Every turn re-sends the session's whole history (about 3,000–3,400 tokens), with a 0.5 s "tool call" pause before the next turn and 4 requests in flight. Prefix caching lets a turn reuse the KV cache of its own previous turn, if that cache is still on the GPU.
The only thing that changes between the two brains is how much GPU memory the KV cache gets: 4.5 GiB holds 32,768 tokens, and 10 GiB holds 72,816. The 16 sessions' histories need 54,173. At 4.5 GiB the pool is big enough for every request that is actually running, but not for all the idle sessions waiting their turn. LRU eviction always throws out the session that has waited longest, which, because the sessions take turns in a circle, is exactly the one that is up next. There were zero preemptions in every run: all the forgetting happens to sessions that are idle.
Full method, pre-registration and raw data: github.com/dundysm/inference-under-pressure.
What you are seeing
| on screen | what it is | source |
|---|---|---|
| the brain's folds | A real human cortex: the Colin27 MRI, one person scanned 27 times, segmented and surfaced at 0.5 mm (1.2M vertices, 2,029 cm² of cortex). Drawn as 450,000 dim points, shaded by how deep each point sits in a fold. | anatomy |
| a point of light | One token slot of the KV cache, fixed to one spot on the cortex. The pool size is the number of lights. | measured |
| pale, condensing | Tokens being computed into memory. A prompt's tokens drift in and settle over that request's measured time to first token; decode tokens land almost at once. | measured timing · replay slot |
| gold blink | A prefix-cache hit: history read back from memory instead of recomputed. | measured |
| warm ember | Memory that has been recalled keeps some of the gold, a little warmer each time it is reused (up to three). Warmth is earned by real reuse. | measured |
| dim, breathing | Remembered but idle: cached and evictable, waiting for its session. | replay |
| spark, then dust | Forgotten: the block sparks, and a plume tinted by the session it belonged to drifts off while the slot is rewritten with someone else's tokens. | replay |
| cyan | One session (agent 5), highlighted so you can follow its memory. | measured |
Which point on the cortex a memory slot sits on is an artistic choice, made once: each block's 16 tokens form one grain of cortex, and the grains are chained nearest-neighbour across the surface, so block b sits next to block b+1. vLLM's free queue tends to hand out runs of neighbouring blocks, so a session's memory lands as one territory; under pressure, as the LRU recycles slots, you watch that territory fragment. The fragmentation is real; the map is not. The anatomy carries no data. Everything that moves does.
How the replay works, and how it was checked
vLLM reports, per request, how many prompt tokens came from the cache. It does not log which GPU block was evicted when. To animate each block, tools/replay.py replays vLLM v1's block manager over the measured request timeline, using the real start time, time to first token, end time and size of every request:
Prompt blocks are allocated at request start, and decode blocks at the first token. At request end, all of a request's blocks go to the tail of an LRU free queue in reverse order. New allocations evict from the head. Which blocks are reused comes straight from the measured cached-token counts. The replay decides only when everything else is evicted.
The check: every block the engine actually reused must still be in the replayed cache at that moment. It was, for every reuse in all 15 runs, with 0 inconsistencies. On its own, without being told the measured reuse, the replay predicts the measured cached tokens exactly for 192/192 requests at every budget from 2.5 to 6 GiB, and 168–171/192 at 8 and 10 GiB. Each miss there is one 16-token block, the one holding the end of the previous prompt plus the model's reply. In those requests the next scripted prompt happened to start with the same tokens the model had generated.
| KV budget | pool (tokens) | resumed reuse | forgotten blocks | replay exact |
|---|---|---|---|---|
| 2.5 GiB | 18,192 | 0.5% | 37,384 | 192 / 192 |
| 3 GiB | 21,840 | 0.5% | 37,158 | 192 / 192 |
| 4.5 GiB | 32,768 | 0.5% | 36,481 | 192 / 192 |
| 6 GiB | 43,680 | 0.5% | 35,805 | 192 / 192 |
| 8 GiB | 58,240 | 98.9% | 77 | 168 / 192 |
| 10 GiB | 72,816 | 98.9% | 0 | 171 / 192 |
Reuse is the measured median over resumed turns. Forgotten blocks count evictions of reusable prompt blocks in one run. One run per budget is shown. 4.5 and 10 GiB come from a pre-registered 5-repetition cohort: the forgotten counts are identical in all five repetitions (36,481 and 0), and the replay's exact matches at 10 GiB range from 168 to 171 of 192.
Credits
Brain: Colin27 average brain (Holmes et al., 1998), MNI / McConnell Brain Imaging Centre, via TemplateFlow. © 1993–2009 Louis Collins, McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, used under its permissive licence. Engine: vLLM. Rendering: three.js, WebGL.
by @dundysm · code and data: github.com/dundysm/forgetting