Storage, write rules, forgetting rules, and the tradeoff none of them escapes.
There’s a small paradox at the heart of in-context learning.
A Transformer’s KV cache never learns anything. It’s a list. Keys and values go in, sit there untouched, and get dropped when the context ends. Nothing in that list adapts. The only thing that adapts is the read: a query comes in, softmax picks what to pull out. And yet we say Transformers learn in context, and empirically they do. Show one a few examples and it starts doing the task.
So where is the learning?
The answer is that memory and learning aren’t separate things here. Any model that eats a stream of tokens is running an online algorithm. It decides what to keep, how to fold in what just arrived, and what to drop. The KV cache is the laziest possible version of that algorithm. Write rule: append. Forgetting rule: never. All the intelligence got pushed downstream into retrieval.
That reframing does a lot of work. The last three years gave us linear attention, DeltaNet, Mamba, RWKV, RetNet, test-time training, Titans, and a pile of retrieval hybrids. The papers mostly argue about perplexity and throughput. Read them as memory algorithms instead and the zoo shrinks. What’s left is a design space with three axes, and a question continual learning has been chewing on for forty years: how plastic can a memory be before it stops retaining anything?
Every memory mechanism answers three questions
Three questions pin down any memory in a sequence model. The answers turn out to be close to independent.
Where does it live? As a raw list of key–value pairs. In a fixed-size matrix. In a fixed-size state vector. In the weights of a small model that keeps training at inference time. Or outside the network, in an index.
How is it written? By appending. By stacking outer products on top of each other. With a gate that decides how much of the new input gets in. With an error-correcting update that checks what the memory already predicts for this key. By a gradient step. Or selectively, where something like surprise decides whether to write at all.
How is it forgotten? Never. By decay, where old content shrinks a little every step. By overwrite, where a new value displaces the old one at the same key. By interference, where a fixed-size store blurs things together whether you want it to or not. By compression. Or by an explicit delete.
Those columns being independent is what makes the grid worth drawing. Most of the novelty of the last three years is a new path through it, not a new column. That’s not a knock. Some of those paths work much better than what came before. But it means the comparisons worth making run along the axes, not between brand names.
The families below are sorted by how much the memory is allowed to learn. Start with the one where it learns nothing.
Attention keeps everything, and pays for it
A Transformer’s memory after $t$ tokens is
\[M_t = \{(k_1, v_1), \dots, (k_t, v_t)\},\]and reading it is soft nearest-neighbor lookup. Raw storage, append, no forgetting. If something made it into the context, it’s still in there. The only question is whether attention can find it.
You pay for that in the obvious way. Memory grows with the context, and every read scans all of it. Sliding-window attention, sparse attention, landmark tokens, Transformer-XL, the Compressive Transformer: each one changes the storage or the forgetting rule and leaves the read alone. Sliding windows add a hard cutoff. Drop anything older than $W$. Landmark attention keeps the whole history but changes the index, tagging each block with a token so the model can find the block first and attend inside it after. The Compressive Transformer swaps forgetting for compression, squeezing old chunks into summary vectors that become memory slots in their own right. In every case the read stays attention. What changes is what there is to attend over.
Reach for this family when you care more about keeping information than about the bill. It’s also the only family where memory and learning stay cleanly apart. Nothing in the memory adapts. Only the read does.
Which leads to the question the rest of the field has been answering. What happens when the memory is too small to keep everything?
A matrix blurs; the delta rule corrects
Collapse the list into a fixed-size matrix. Linear attention swaps the softmax kernel for a feature map and accumulates
\[M_t = M_{t-1} + v_t k_t^\top, \qquad y_t = M_t q_t.\]Storage is now a $d \times d$ associative memory, the write is additive, and forgetting happens by accident. Every outer product lands on top of the last one. Query the memory and you get back a weighted sum of every value whose key looks like your query. Similar keys blur. Write the same key twice with different values and you get both back, added together. Nothing in that update can say this key means something else now.
The delta rule fixes it. Make the write depend on what’s already in there:
\[M_t = M_{t-1} + \beta_t \big(v_t - M_{t-1} k_t\big) k_t^\top.\]The term in parentheses is a prediction error. What the memory returns for $k_t$ today, minus what it should return. Schlag, Irie and Schmidhuber introduced this update when they showed linear Transformers are fast weight programmers in disguise. Yang and colleagues later made it practical by parallelizing it over sequence length, which is where the name DeltaNet comes from. Forgetting is now overwrite. Write a key again and the old association gets corrected instead of buried. $\beta_t$ sets how hard.
This is where the framing starts paying for itself. $M$ is a linear model. The write rule is its training algorithm: one step of online gradient descent on squared error. In-context learning here isn’t like online learning. It is online learning, running in the hidden state.
Linear attention does Hebbian learning. DeltaNet does error-corrected learning. So when the two behave differently on recall tasks with reused keys, that isn’t an architecture mystery. It’s the gap between two learning rules that people understood decades before either paper.
That’s one way to shrink a memory. There’s a second, and it starts from a different picture of what a state is.
A state vector forgets by compressing
The other lineage keeps a fixed-size state too. It just doesn’t treat it as a key–value table. It treats it as a compressed trajectory:
\[h_t = A_t h_{t-1} + B_t x_t.\]S4, Mamba, Mamba-2 and the selective SSMs live here. So do RWKV, HGRN, RetNet and gated linear attention. What they share is a multiplicative term on the previous state. RetNet uses a fixed decay,
\[S_t = \gamma S_{t-1} + k_t v_t^\top,\]which is linear attention with a forgetting rule bolted on. Mamba makes the transition input-dependent, so the model picks per token how much of the past survives. Gated DeltaNet does the obvious thing once you see write and forgetting as separate knobs: delta write, gated forget. RWKV-7 lands somewhere similar from the other side, with a generalized delta rule driving its state.
The research question here is a different one. Nobody asks whether Mamba can overwrite an association. They ask how much history a $d$-dimensional state can hold, and what the gate learns to throw away. The failure mode isn’t blur between similar keys. It’s information that never made it into the state at all. Arora and colleagues pinned this down in the Zoology work: recall quality tracks recurrent state size, and you don’t get recall and throughput for free at the same time. Jelassi and colleagues came at it from the other end and showed Transformers beat SSMs at plain copying, which is about as pure a test of “did you keep it” as you can build.
Keep the two pictures separate even when the equations line up. DeltaNet is memory as an online learner. Mamba is memory as a dynamical system. They point at different diagnostics, different ablations, different stories about why a run went wrong. Conversations go sideways when two people use the same equation to mean different things.
If the memory is a learner, give it a better learner
Push the learner reading one step further. DeltaNet’s memory is a linear model trained by one gradient step per token. Why linear?
Test-time training says it doesn’t have to be. The memory becomes a small network $f_\theta$, and every token triggers an update:
\[\theta_t = \theta_{t-1} - \eta \nabla_\theta \mathcal{L}(x_t; \theta), \qquad y_t = f_{\theta_t}(q_t).\]Make $f$ linear and $\mathcal{L}$ squared reconstruction, and DeltaNet drops out. Make it an MLP and the memory can hold nonlinear structure. Wang and colleagues turned this into a recipe: pick an associative-memory objective, pick an optimizer, read off an architecture. That’s roughly where the field has landed, even if the papers still arrive one architecture at a time.
Titans push somewhere else. Not on how expressive the memory is, but on what gets written. The long-term memory module still updates by gradient descent, but the learning rate scales with surprise, roughly the gradient magnitude. Tokens the memory already predicts barely move it. Novel tokens write hard. Add momentum so surprise carries for a few steps, plus weight decay as a forgetting gate, and the memory can tell “this is new, keep it” from “this is routine, skip it.”
That second axis is the interesting one. Uniform updates versus importance-weighted updates. It’s the least explored knob in the space, and the place where continual learning has the most to say and gets listened to the least. Every architecture above writes on every token. Biological memory doesn’t. Neither does anyone who takes decent notes.
It also needs numbers and doesn’t have many. The obvious experiment: a stream whose key distribution shifts halfway through, a linear-attention memory and a TTT memory at matched state size. Don’t score final accuracy. Score two curves. How fast does each pick up the new regime, and how much of the old one can it still answer? The shapes should differ more than the endpoints do. That’s a guess, not a result.
Outside the network, forgetting becomes a retrieval problem
One family refuses the premise. Why compress the past into the network at all?
kNN-LM, the Memorizing Transformer, RETRO, RAG, most agent memory systems: they write events to a store outside the model,
\[m_i = (k_i, v_i), \qquad i^* = \arg\max_i \; \mathrm{sim}(q, k_i),\]and look them up at read time. Capacity is basically unbounded. Nothing gets overwritten by accident. RETRO took it to the limit and retrieved from a database of trillions of tokens. The Memorizing Transformer showed you can bolt a kNN lookup onto an attention layer and let the model learn to use it.
The hard problems don’t go away. They move to the edge of the store. What’s worth writing? How do you index it so the right thing comes back? Should the model retrieve at all, or trust its weights? What happens when a memory is stale, or when two memories disagree? Over long horizons those questions matter more than capacity. A store with a bad write policy is a growing pile of noise. A store with a bad read policy is a library with no catalog.
In the three-axis language: unlimited storage, a write rule nobody has specified, and a forgetting rule quietly handed to a similarity function. That’s a strange place to park, and it won’t hold.
It’s the stability–plasticity dilemma in an architecture costume
Line the families up and the same shape shows up every time. Each one is good on one side of a single tradeoff and pays for it on the other.
| Substrate | Plasticity | Retention | Failure mode |
|---|---|---|---|
| KV attention | Perfect within context | Perfect within context | Context is finite; nothing survives past it |
| Linear attention | High | Poor | Interference; values returned in superposition |
| DeltaNet | High | Moderate | Overwrite destroys the old value at a reused key |
| SSM / gated RNN | Gate-controlled | Gate-controlled | Compression loss; a wrong gate is irreversible |
| Test-time training | Very high | Fragile | Catastrophic forgetting in the fast weights |
| Titans-style | Selective | Improved | Only as good as the surprise signal’s calibration |
| External memory | Unbounded | Strong | Retrieval is the bottleneck; staleness, contradiction |
This is the stability–plasticity dilemma. Grossberg named it in the 1980s and continual learning has been paying for it ever since. Define
\[\begin{aligned} \text{Plasticity} &= \text{how fast new information is acquired}, \\ \text{Retention} &= \text{how much old information survives}, \end{aligned}\]and every architecture above is a point on that plane. Sweep its hyperparameters and it becomes a curve.
That’s a better comparison than another Mamba-versus-Transformer-versus-TTT benchmark, because you can control it. Build an online task whose statistics shift partway through. Keys get reassigned. Popularity follows a heavy tail. A new key family shows up in a burst. Junk arrives that should be ignored. Then ask each substrate two things: how fast does it pick up the new regime, and how much of the old one does it still get right? The answers aren’t scalars. They’re curves, and the curve is the architecture’s signature.
A few predictions fall out, all cheap to test:
- Linear attention should lose retention as the number of distinct keys grows, whether or not anything gets reassigned. Its failure is interference.
- DeltaNet shouldn’t care much about distinct-key count, but should lose retention as the reassignment rate climbs. Its failure is overwrite.
- A TTT memory with an MLP should win on plasticity under shift and lose on retention, unless you regularize it. The regularization that helps should look like continual learning’s: decay toward an anchor, elastic penalties in the spirit of EWC, replay.
- Surprise-gated writes should help on streams full of routine tokens and hurt when surprise is miscalibrated. Junk that’s novel but worthless is the obvious failure case, and real data is full of it.
Maybe worth studying more
Going in, the differences between these architectures looked like efficiency differences. That’s how the papers frame them. Coming out, efficiency looks more like the constraint and memory policy like the actual content. A few questions kept showing up no matter which family I read, and none of them have clean answers yet.
How much can a fixed state hold? For SSMs this is the whole ballgame. We have real capacity results for the linear case and mostly intuition for the selective one.
What deserves to be written? Surprise is one signal. Reward, downstream utility and predicted future relevance are others. None of them is obviously right. It’s the same question as what an agent should put in its scratchpad, and it’s just as open there. An earlier post on agent memory ran into the same wall from the other side.
Is forgetting a bug or a feature? In classical continual learning it’s the bug with a name. In a memory that has to survive a shifting stream, decay is doing real work. It encodes a prior: recent information matters more. Architectures that can’t forget, plain linear attention and unbounded stores among them, aren’t obviously better than the ones that can. Over long horizons they’re probably worse.
Can you compose substrates? The most interesting recent systems are hybrids. Attention for sharp short-term recall. A recurrent or fast-weight state for compressed medium-term context. An external index for the long tail. That’s roughly the human split between working, semantic and episodic memory, which is either encouraging or a sign of pattern-matching. Either way the question stops being which substrate wins. It becomes how to coordinate the write and forgetting rules across several at once, and nobody has a good answer to that yet.
A sequence architecture is an online learning algorithm for its own memory. The axis that matters most isn’t FLOPs or context length. It’s where the design sits on the plasticity–retention frontier. Most papers report the first two. Almost none report the third.
Notes
References:
- Katharopoulos et al., Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, ICML 2020 — arXiv:2006.16236
- Ba et al., Using Fast Weights to Attend to the Recent Past, NIPS 2016 — arXiv:1610.06258
- Schlag, Irie & Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers, ICML 2021 — arXiv:2102.11174
- Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length (DeltaNet), NeurIPS 2024 — arXiv:2406.06484
- Yang, Kautz & Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule, ICLR 2025 — arXiv:2412.06464
- Yang et al., Gated Linear Attention Transformers with Hardware-Efficient Training, ICML 2024 — arXiv:2312.06635
- Gu, Goel & Ré, Efficiently Modeling Long Sequences with Structured State Spaces (S4), ICLR 2022 — arXiv:2111.00396
- Gu & Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, COLM 2024 — arXiv:2312.00752
- Dao & Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2), ICML 2024 — arXiv:2405.21060
- Sun et al., Retentive Network: A Successor to Transformer for Large Language Models, 2023 — arXiv:2307.08621
- Peng et al., RWKV: Reinventing RNNs for the Transformer Era, Findings of EMNLP 2023 — arXiv:2305.13048
- Peng et al., RWKV-7 “Goose” with Expressive Dynamic State Evolution, 2025 — arXiv:2503.14456
- Qin, Yang & Zhong, Hierarchically Gated Recurrent Neural Network for Sequence Modeling (HGRN), NeurIPS 2023 — arXiv:2311.04823
- Sun et al., Learning to (Learn at Test Time): RNNs with Expressive Hidden States (TTT), 2024 — arXiv:2407.04620
- Behrouz, Zhong & Mirrokni, Titans: Learning to Memorize at Test Time, NeurIPS 2025 — arXiv:2501.00663
- Behrouz et al., It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization, 2025 — arXiv:2504.13173
- Behrouz et al., ATLAS: Learning to Optimally Memorize the Context at Test Time, 2025 — arXiv:2505.23735
- Wang, Nguyen & Duvenaud, Test-Time Regression: A Unifying Framework for Designing Sequence Models with Associative Memory, 2025 — arXiv:2501.12352
- Dai et al., Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, ACL 2019 — arXiv:1901.02860
- Rae et al., Compressive Transformers for Long-Range Sequence Modelling, ICLR 2020 — arXiv:1911.05507
- Mohtashami & Jaggi, Landmark Attention: Random-Access Infinite Context Length for Transformers, NeurIPS 2023 — arXiv:2305.16300
- Munkhdalai, Faruqui & Gopal, Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention, 2024 — arXiv:2404.07143
- Khandelwal et al., Generalization through Memorization: Nearest Neighbor Language Models (kNN-LM), ICLR 2020 — arXiv:1911.00172
- Wu et al., Memorizing Transformers, ICLR 2022 — arXiv:2203.08913
- Borgeaud et al., Improving Language Models by Retrieving from Trillions of Tokens (RETRO), ICML 2022 — arXiv:2112.04426
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020 — arXiv:2005.11401
- Arora et al., Zoology: Measuring and Improving Recall in Efficient Language Models, ICLR 2024 — arXiv:2312.04927
- Arora et al., Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff (Based), ICML 2024 — arXiv:2402.18668
- Jelassi et al., Repeat After Me: Transformers are Better than State Space Models at Copying, ICML 2024 — arXiv:2402.01032
- Kirkpatrick et al., Overcoming Catastrophic Forgetting in Neural Networks (EWC), PNAS 2017 — arXiv:1612.00796
The stability–plasticity dilemma predates all of this; the standard reference is Grossberg’s adaptive resonance work from the 1970s and 1980s, later summarized in Carpenter and Grossberg’s ART papers. Positions in the plasticity–retention figure are illustrative rather than measured, and the reused-key figure assumes $\beta = 1$ and unit-norm keys so that the update is a clean replacement.
@article{reneejia2026memoryalgorithm,
title = {Every Sequence Model Is a Memory Algorithm},
author = {Renee Jia},
journal = {renee-jia.github.io},
year = {2026},
url = {https://renee-jia.github.io/research%20blog/ai%20research/every-sequence-model-is-a-memory-algorithm/}
}