Learning a Spatial Memory Codec for Video World Models

01 · Results

On returns to earlier places, our bidirectional model scores 2.16 dB higher PSNR and 55% lower rotation error than the strongest baseline on each metric.

Every method sees the same earlier observations and the same starting frame, then generates the future as the camera returns. The clips span driving, campus and city walks, web videos, scene captures and game worlds. Our video model is pose-conditioned and trained jointly with the memory.

Our autoregressive model also outperforms every baseline in its group, and every bidirectional baseline on average.

The autoregressive model generates chunk by chunk and writes each finished chunk to memory before generating the next. Against the strongest autoregressive baseline on each metric, it gains 3.28 dB PSNR, with 35% lower LPIPS and 91% lower rotation error.

02 · How it works

A learned codec writes each chunk once and decodes only the entries a view needs.

A world model must recall a place after its earlier views have left the recent context. The cost of keeping every past frame in context grows with the history, and decoding a whole archive for each view is slow. The codec stores compact entries, gives each one a landmark, and reads back only a few entries per view.

The animation uses illustrative values.

Each landmark is a small address, and the reader picks entries by their scores.

A landmark holds a few learned summaries of its entry, with the entry's pose and time. The reader scores every landmark against the target view, keeps the top $K$ , and decodes only those entries, weighted by their scores.

1

Index

\ell_{c,p} = \Big( \{ (m_{c,p,h},\, e_{c,p,h}) \}_{h=1}^{H},\; P_{c,p},\; \tau_{c,p} \Big)

Entry $p$ of chunk $c$ gets $H$ learned summaries $m$ with their entropies $e$ , its camera pose $P$ and its time $\tau$ .

2

Read

S = \operatorname{TopK}_b\, s_{t,b},

\pi_b = \frac{\exp(g\, s_{t,b})}{\sum_{b' \in S} \exp(g\, s_{t,b'})}, \quad b \in S

The relevance scores $s_{t,b}$ compare the target query with every landmark. The decoder reads the $K$ selected entries $S$ , mixed by the weights $\pi_b$ in its first layers.

Compact entries outperform per-frame storage, and a sparse read matches a dense one with five times fewer tokens.

At the same read budget, the codec reaches 20.28 dB PSNR on held-out views. Storing every frame as its own tokens reaches 15.53 dB. A dense read of the whole archive reaches 20.24 dB while reading 32,768 tokens instead of 6,144.

03 · Details

Reads stay fast up to 1.65 hours, and quality holds on archives of nearly two hours.

The codec is trained on at most 128 observations. On driving footage, the query views and their relevant history stay fixed while unrelated drives are added.

With independent drives combined into nearly two hours, quality stays within 0.3 dB of reading only the relevant views.

92 to 100% of the opened entries come from the relevant chunks, so unrelated history is rarely read. A read takes about 44 ms for archives of up to 1.65 hours, excluding video generation. Each chunk is encoded once, in about 87 ms.

Storage follows the content of each chunk.

For the same requested quality, the codec keeps on average 0.63 entries per chunk on web videos. Scene captures, with more dynamic detail and faster camera motion, take 2.9.