Storage, write rules, forgetting rules, and the tradeoff none of them escapes.
There’s a small paradox at the heart of in-context learning.
A Transformer’s KV cache never learns anything. It’s a list. Keys and values go in, sit there untouched, and get dropped when the context ends. Nothing in that list adapts. The only thing that adapts is the read: a query comes in, softmax picks what to pull out. And yet we say Transformers learn in context, and empirically they do. Show one a few examples and it starts doing the task.
So where is the learning?
The answer is that memory and learning aren’t separate things here. Any model that eats a stream of tokens is running an online algorithm. It decides what to keep, how to fold in what just arrived, and what to drop. The KV cache is the laziest possible version of that algorithm. Write rule: append. Forgetting rule: never. All the intelligence got pushed downstream into retrieval.
That reframing does a lot of work. The last few years gave us linear attention, DeltaNet, Mamba, RWKV, RetNet, test-time training, Titans, and a pile of systems built around retrieval. The papers mostly argue about perplexity and throughput. Read them as memory algorithms instead and the zoo shrinks. What’s left is a design space with three axes, and a question continual learning has been stuck on for forty years: how plastic can a memory be before it stops retaining anything?
Every memory mechanism answers three questions
Three questions pin down any memory in a sequence model. The answers turn out to be close to independent.
Where does it live? As a raw list of key–value pairs. In a fixed-size matrix. In a fixed-size state vector. In the weights of a small model that keeps training at inference time. Or outside the network, in an index.
How is it written? By appending. By stacking outer products on top of each other. With a gate that decides how much of the new input gets in. With an error-correcting update that checks what the memory already predicts for this key. By a gradient step. Or selectively, where something like surprise decides whether to write at all.
How is it forgotten? Never. By decay, where old content shrinks a little every step. By overwrite, where a new value pushes out the old one at the same key. By interference, where a fixed-size store blurs things together whether you want it to or not. By compression. Or by never being retrieved again.
Those columns being independent is what makes the grid worth drawing. Most of the novelty of the last three years is a new path through it, not a new column. That’s not a criticism. Some of those paths work much better than what came before. But it means the comparisons worth making run along the axes, not between brand names.
The families below are sorted by how much the memory is allowed to learn. Start with the one where it learns nothing.
Attention keeps everything, and pays for it
A Transformer’s memory after $t$ tokens is
\[M_t = \{(k_1, v_1), \dots, (k_t, v_t)\},\]and reading it is soft nearest-neighbor lookup. Raw storage, append, no forgetting. If something made it into the context, it’s still in there. The only question is whether attention can find it.
You pay for that in the obvious way. Memory grows with the context, and every read scans all of it. Sliding-window attention, sparse attention, landmark tokens, Transformer-XL, the Compressive Transformer: each one changes the storage or the forgetting rule and leaves the read alone. Sliding windows add a hard cutoff. Drop anything older than $W$. Landmark attention keeps the whole history but changes the index. Each block gets a tag token, so the model finds the block first and attends inside it after. The Compressive Transformer swaps forgetting for compression, squeezing old chunks into summary vectors that become memory slots in their own right. In every case the read stays attention. What changes is what there is to attend over.
Reach for this family when you care more about keeping information than about the bill. It’s also the family where memory and learning stay most cleanly apart. Nothing in the memory adapts. Only the read does.
Which leads to the question the rest of the field has been answering. What happens when the memory is too small to keep everything?
A matrix blurs; the delta rule corrects
Collapse the list into a fixed-size matrix. Linear attention swaps the softmax kernel for a feature map and accumulates
\[M_t = M_{t-1} + v_t k_t^\top, \qquad y_t = M_t q_t.\]Storage is now a $d \times d$ associative memory. The write is additive. Nothing is ever erased, so forgetting happens by accident, as blur. Every outer product lands on top of the last one. Query the memory and you get back a weighted sum of every value whose key looks like your query. Similar keys blur. Write the same key twice with different values and you get both back, added together. Nothing in that update can say this key means something else now.
The delta rule fixes it. Make the write depend on what’s already in there:
\[M_t = M_{t-1} + \beta_t \big(v_t - M_{t-1} k_t\big) k_t^\top.\]The term in parentheses is a prediction error. What the memory returns for $k_t$ today, minus what it should return. Schlag, Irie and Schmidhuber introduced this update when they showed linear Transformers are secretly fast weight programmers. Yang and colleagues later made it practical by running it in parallel across the sequence, which is where the name DeltaNet comes from. Forgetting is now overwrite. Write a key again and the old association gets corrected instead of buried. $\beta_t$ sets how hard.
This is where the framing starts paying for itself. $M$ is a linear model. The write rule is its training algorithm: one step of online gradient descent on squared error. In-context learning here isn’t like online learning. It is online learning, running in the hidden state.
Linear attention does Hebbian learning. DeltaNet does error-corrected learning. So when the two behave differently on recall tasks with reused keys, that isn’t an architecture mystery. It’s the gap between two learning rules that people understood decades before either paper.
That’s one way to shrink a memory. There’s a second, and it starts from a different picture of what a state is.
A state vector forgets by compressing
The other family keeps a fixed-size state too. It just doesn’t treat it as a key–value table. It treats it as a compressed summary of everything so far:
\[h_t = A_t h_{t-1} + B_t x_t.\]S4, Mamba, Mamba-2 and the selective SSMs live here. So do RWKV, HGRN, RetNet and gated linear attention. What they share is that the old state gets multiplied by something before the new input is added. RetNet uses a fixed decay,
\[S_t = \gamma S_{t-1} + k_t v_t^\top,\]which is linear attention with a forgetting rule added on. Mamba makes the transition input-dependent, so the model picks per token how much of the past survives. Gated DeltaNet does the obvious thing once you see write and forgetting as separate knobs: delta write, gated forget. RWKV-7 lands somewhere similar from the other side, with a generalized delta rule driving its state.
The research question here is a different one. Nobody asks whether Mamba can overwrite an association. They ask how much history a $d$-dimensional state can hold, and what the gate learns to throw away. The failure mode isn’t blur between similar keys. It’s information that never made it into the state at all. Arora and colleagues pinned this down in the Zoology work. Recall quality tracks recurrent state size, and you don’t get recall and throughput for free at the same time. Jelassi and colleagues came at it from the other end. They showed Transformers beat SSMs at plain copying, about as pure a test of “did you keep it” as you can build.
Keep the two pictures separate even when the equations line up. DeltaNet is memory as an online learner. Mamba is memory as a dynamical system. They suggest different checks, different ablations, different stories about why a run went wrong. Conversations go wrong when two people use the same equation to mean different things.
If the memory is a learner, give it a better learner
Push the learner reading one step further. DeltaNet’s memory is a linear model trained by one gradient step per token. Why linear?
Test-time training says it doesn’t have to be. The memory becomes a small network $f_\theta$, and every token triggers an update:
\[\theta_t = \theta_{t-1} - \eta \nabla_\theta \mathcal{L}(x_t; \theta), \qquad y_t = f_{\theta_t}(q_t).\]Make $f$ linear and $\mathcal{L}$ squared error, and DeltaNet drops out. Make it an MLP and the memory can hold nonlinear structure. Wang and colleagues turned this into a recipe: pick an associative-memory objective, pick an optimizer, read off an architecture. That’s roughly where the field has landed, even if the papers still arrive one architecture at a time.
Titans push somewhere else. Not on how expressive the memory is, but on what gets written. The long-term memory still updates by gradient descent. But the size of each write is set by surprise: how far off the memory’s prediction for the current token was. Tokens the memory already predicts barely move it. Novel tokens write hard. Add momentum so surprise carries for a few steps. Add weight decay as a forgetting gate. Now the memory can tell “this is new, keep it” from “this is routine, skip it.”
That second axis is the interesting one. Uniform updates versus importance-weighted updates. It’s the least explored knob in the space. It’s also where continual learning has the most to say, and gets listened to the least. Every architecture above writes on every token. Biological memory doesn’t. Neither does anyone who takes decent notes.
It also needs numbers and doesn’t have many. The obvious experiment: a stream whose key distribution shifts halfway through, a linear-attention memory and a TTT memory at matched state size. Don’t score final accuracy. Score two curves. How fast does each pick up the new pattern, and how much of the old one can it still answer? The shapes should differ more than the endpoints do. That’s a guess, not a result.
Outside the network, forgetting becomes a retrieval problem
One family refuses the question. Why compress the past into the network at all?
kNN-LM, the Memorizing Transformer, RETRO, RAG, most agent memory systems: they write events to a store outside the model,
\[m_i = (k_i, v_i), \qquad i^* = \arg\max_i \; \mathrm{sim}(q, k_i),\]and look them up at read time. There’s no real limit on capacity. Nothing gets overwritten by accident. RETRO took it to the limit and retrieved from a database of trillions of tokens. The Memorizing Transformer showed you can bolt a kNN lookup onto an attention layer and let the model learn to use it.
The hard problems don’t go away. They move to the edge of the store. What’s worth writing? How do you index it so the right thing comes back? Should the model retrieve at all, or trust its weights? What happens when a memory is out of date, or when two memories disagree? Over long horizons those questions matter more than capacity. A store with a bad write policy is a growing pile of noise. A store with a bad read policy is a library with no index.
In the three-axis language: unlimited storage, a write rule nobody has specified, and a forgetting rule handed to a similarity function. That’s a strange place to park, and it won’t hold.
It’s the stability–plasticity dilemma in an architecture costume
Line the families up and the same shape shows up every time. Each one is good on one side of a single tradeoff and pays for it on the other. One word needs pinning down first, because plasticity gets used two ways. Here it means how much one new token is allowed to change what the memory already holds. Retention means how much of what was stored can still be read back. Both are properties of the memory, not of the model around it.
That’s what separates the first two rows. Neither ever erases anything. But KV attention keeps every pair separate, and linear attention sums them into one matrix, so its reads blur. Mamba and RWKV often get filed under linear attention too. Here they’re the gated row, because the decay is what moves them along the curve.
| Substrate | Plasticity | Retention | Forgetting curve | Failure mode |
|---|---|---|---|---|
| KV attention | None in the store. A token adds a pair and changes nothing; the read does the adapting | Perfect within the window, zero past it | Flat, then a cliff | Context is finite; nothing survives past it |
| Linear attention | Low. Additive, so a reused key is out-voted, not replaced | High but blurred; every read mixes values together | Not a function of time; blur grows with the number of distinct keys | Interference |
| DeltaNet | High. A reused key is overwritten | Moderate | Not a function of time; drops at each reassignment | Overwrite destroys the old value |
| SSM / gated RNN (Mamba, RWKV, RetNet) | Gate-controlled | Gate-controlled; below KV attention at every lag within the window | Smooth decay. Exponential for a fixed rate, closer to a power law when rates vary by channel | Compression loss; a wrong gate can’t be undone |
| Test-time training | Very high | Fragile | Steep | Catastrophic forgetting in the fast weights |
| Titans-style | Selective | Improved | Decay, except for surprising tokens | Only as good as the surprise signal |
| External memory | None in the store; it only grows | Strong, if retrieved | Flat; loss happens at retrieval, not over time | Retrieval is the bottleneck; out-of-date or conflicting entries |
Two rows don’t play the game. KV attention and the external store never change what they hold. They grow instead. That’s not a resolution of the tradeoff. It’s a way of buying out of it, one slot per token. The bill comes due at the window, or at the retriever. Every fixed-size memory has to pay on one side or the other.
That tradeoff has a name. Grossberg called it the stability–plasticity dilemma back in 1980, and continual learning still hasn’t found a way around it. Put the two sides on axes,
\[\begin{aligned} \text{Plasticity} &= \text{how much one new token can change what is already stored}, \\ \text{Retention} &= \text{how much of what was stored can still be read back}, \end{aligned}\]and every architecture above is a point on that plane. Change its hyperparameters and it traces a curve.
The retention axis hides a shape. Plot how much of a token the memory can still return against how long ago it was written. The families separate again, this time by the form of the curve rather than its height.
KV attention’s curve is a step. Inside the window, everything is there, verbatim, at lag one and at lag ten thousand. Past the window, nothing is. The gated row forgets on a slope instead. Every step multiplies the state by a number a little below one. So whatever a token wrote fades a bit with each token that follows. In RetNet that number is fixed, and the fade is a clean exponential. In Mamba each channel of the state has its own rate, and the input sets that rate token by token. Recent versions of RWKV do the same. Add up many exponentials with different rates and the sum no longer looks like an exponential. It looks like a power law: steep early loss, then a long tail. That’s the shape Ebbinghaus drew in 1885 and the one Wixted and Ebbesen fit to human recall a century later. So the slope is the human-looking one, and the step isn’t. Nobody keeps a conversation verbatim for eight thousand words and then loses all of it at once.
But look at where the curves sit, not only how they bend. Inside the window the gated state’s curve is under KV attention’s everywhere. A gated state is lossy from the moment a token is written, and it keeps losing. The copying results from Jelassi and colleagues and the recall results from the Zoology paper both measure this gap. Past the window the order flips, because KV attention has nothing left and the state still has its tail. Mamba and RWKV forget more like a person does, and they remember less than a Transformer does. Both things are true, and which one matters depends on whether your task fits inside the window.
DeltaNet isn’t on that plot, and that’s informative. Its forgetting isn’t a function of lag at all. An association survives until the same key is written again, then it’s gone, however old or new it was. Linear attention is missing for a related reason. What blurs it is how many different keys have been written since, not how much time has passed. Lag is only the right axis for the memories that decay.
Curves like these are a better comparison than another Mamba-versus-Transformer-versus-TTT benchmark, because you can control what produces them. Build an online task whose statistics shift partway through. Keys get reassigned. Popularity follows a heavy tail. A new key family shows up in a burst. Junk arrives that should be ignored. Then ask each substrate two things. How fast does it pick up the new pattern? How much of the old one does it still get right? The answers aren’t scalars. They’re curves, and the curve is the architecture’s signature.
A few predictions fall out, and all of them are cheap to test:
- Linear attention will be slow to pick up a reassigned key. The new value has to out-vote the old one, and that takes repeated writes. It will also get blurrier on every key, old and new, as the number of distinct keys grows. What hurts it is interference, not forgetting.
- DeltaNet cares less about how many keys there are. What hurts it is how often a key changes meaning. Each reassignment wipes out the old value, so retention should drop as the reassignment rate goes up.
- A TTT memory with an MLP should adapt fastest when the stream shifts, and forget the most. Regularize it and the gap closes. The fixes that work should be the ones continual learning already knows: decay toward the starting weights, an EWC-style penalty, replay.
- Surprise-gated writes should help when most tokens are routine, and hurt when the surprise signal is off. The obvious trap is junk that’s novel but useless. Real data is full of that.
Maybe worth studying more
Going in, the differences between these architectures looked like efficiency differences. That’s how the papers frame them. Coming out, efficiency looks more like the constraint and memory policy like the actual content. A few questions kept showing up no matter which family I read, and none of them have clean answers yet.
How much can a fixed state hold? For SSMs this is the whole question. We have real capacity results for the linear case and mostly intuition for the selective one.
What deserves to be written? Surprise is one signal. Reward, downstream utility and predicted future relevance are others. None of them is obviously right. It’s the same question as what an agent should put in its scratchpad, and it’s just as open there. An earlier post on agent memory ran into the same wall from the other side.
Is forgetting a bug or a feature? In classical continual learning it’s the bug with a name. In a memory that has to survive a shifting stream, decay is doing real work. It encodes a prior: recent information matters more. Architectures that can’t forget, plain linear attention and ever-growing stores among them, aren’t obviously better than the ones that can. Over long horizons they’re probably worse.
Can you compose substrates? The most interesting recent systems mix several kinds. Attention for sharp short-term recall. A recurrent or fast-weight state for compressed medium-term context. An external index for the long tail. That’s roughly the human split between working, semantic and episodic memory, which is either encouraging or a sign of seeing patterns that aren’t there. Either way the question stops being which substrate wins. It becomes how to coordinate the write and forgetting rules across several at once. Nobody has a good answer to that yet.
A sequence architecture is an online learning algorithm for its own memory. The axis that matters most isn’t FLOPs or context length. It’s where the design sits on the plasticity–retention frontier. Most papers report the first two. Almost none report the third.
Notes
References:
- Katharopoulos et al., Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, ICML 2020 — arXiv:2006.16236
- Ba et al., Using Fast Weights to Attend to the Recent Past, NIPS 2016 — arXiv:1610.06258
- Schlag, Irie & Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers, ICML 2021 — arXiv:2102.11174
- Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length (DeltaNet), NeurIPS 2024 — arXiv:2406.06484
- Yang, Kautz & Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule, ICLR 2025 — arXiv:2412.06464
- Yang et al., Gated Linear Attention Transformers with Hardware-Efficient Training, ICML 2024 — arXiv:2312.06635
- Gu, Goel & Ré, Efficiently Modeling Long Sequences with Structured State Spaces (S4), ICLR 2022 — arXiv:2111.00396
- Gu & Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, COLM 2024 — arXiv:2312.00752
- Dao & Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2), ICML 2024 — arXiv:2405.21060
- Sun et al., Retentive Network: A Successor to Transformer for Large Language Models, 2023 — arXiv:2307.08621
- Peng et al., RWKV: Reinventing RNNs for the Transformer Era, Findings of EMNLP 2023 — arXiv:2305.13048
- Peng et al., Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence (RWKV-5 and RWKV-6), 2024 — arXiv:2404.05892
- Peng et al., RWKV-7 “Goose” with Expressive Dynamic State Evolution, 2025 — arXiv:2503.14456
- Qin, Yang & Zhong, Hierarchically Gated Recurrent Neural Network for Sequence Modeling (HGRN), NeurIPS 2023 — arXiv:2311.04823
- Sun et al., Learning to (Learn at Test Time): RNNs with Expressive Hidden States (TTT), 2024 — arXiv:2407.04620
- Behrouz, Zhong & Mirrokni, Titans: Learning to Memorize at Test Time, NeurIPS 2025 — arXiv:2501.00663
- Behrouz et al., It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization, 2025 — arXiv:2504.13173
- Behrouz et al., ATLAS: Learning to Optimally Memorize the Context at Test Time, 2025 — arXiv:2505.23735
- Wang, Nguyen & Duvenaud, Test-Time Regression: A Unifying Framework for Designing Sequence Models with Associative Memory, 2025 — arXiv:2501.12352
- Dai et al., Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, ACL 2019 — arXiv:1901.02860
- Rae et al., Compressive Transformers for Long-Range Sequence Modelling, ICLR 2020 — arXiv:1911.05507
- Mohtashami & Jaggi, Landmark Attention: Random-Access Infinite Context Length for Transformers, NeurIPS 2023 — arXiv:2305.16300
- Munkhdalai, Faruqui & Gopal, Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention, 2024 — arXiv:2404.07143
- Khandelwal et al., Generalization through Memorization: Nearest Neighbor Language Models (kNN-LM), ICLR 2020 — arXiv:1911.00172
- Wu et al., Memorizing Transformers, ICLR 2022 — arXiv:2203.08913
- Borgeaud et al., Improving Language Models by Retrieving from Trillions of Tokens (RETRO), ICML 2022 — arXiv:2112.04426
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020 — arXiv:2005.11401
- Arora et al., Zoology: Measuring and Improving Recall in Efficient Language Models, ICLR 2024 — arXiv:2312.04927
- Arora et al., Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff (Based), ICML 2024 — arXiv:2402.18668
- Jelassi et al., Repeat After Me: Transformers are Better than State Space Models at Copying, ICML 2024 — arXiv:2402.01032
- Kirkpatrick et al., Overcoming Catastrophic Forgetting in Neural Networks (EWC), PNAS 2017 — arXiv:1612.00796
The stability–plasticity dilemma predates all of this; the standard reference is Grossberg’s adaptive resonance work from the 1970s and 1980s, later summarized in Carpenter and Grossberg’s ART papers. The human forgetting curve goes back to Ebbinghaus, Über das Gedächtnis, 1885; Wixted and Ebbesen, On the Form of Forgetting, Psychological Science 1991, is the standard case for the power-law fit. The claim that a gated state forgets along something like that shape is an argument from the spread of decay rates, not a measurement. Positions in the plasticity–retention figure and heights in the forgetting-curve figure are illustrative rather than measured, and the reused-key figure assumes $\beta = 1$ and unit-norm keys so that the update is a clean replacement.
@article{reneejia2026memoryalgorithm,
title = {Every Sequence Model Is a Memory Algorithm},
author = {Renee Jia},
journal = {renee-jia.github.io},
year = {2026},
url = {https://renee-jia.github.io/research%20blog/ai%20research/every-sequence-model-is-a-memory-algorithm/}
}