The article begins by noting that large language models can recall obscure facts on demand, treating this ability as a form of memory. It then asks how models store and retrieve so much information, proposing to view linear maps, multilayer perceptrons (MLPs) and self‑attention as different kinds of memory stores.
A memory store is defined as a function that holds M memories, each a key‑value pair where the key is used for similarity‑based lookup and the value is what is retrieved. Querying the store with an input returns a weighted sum of values, weighted by the similarity between the query and each stored key. This makes the store an associative memory.
The simplest store is an outer‑product linear layer, which accumulates memories by adding outer products of keys and values to a weight matrix. When querying, recall consists of the target value plus interference from all other memories whose keys are similar to the query. Assuming keys are random unit vectors, the interference‑to‑signal ratio grows as the square root of M divided by the key dimension d_k. Setting this ratio to one gives a capacity of approximately M ≈ d_k; beyond that, interference drowns the signal. For a 64‑dimensional key space, this yields a capacity of about 64 memories.
To improve capacity, the post describes an optimal linear memory store obtained by solving a least‑squares problem that fits all stored memories simultaneously. This store whitens the keys before comparison, reducing interference. When the number of memories does not exceed the key dimension, recall can be exact even with non‑orthogonal keys. When M > d_k, the interference‑to‑signal ratio becomes sqrt(M/d_k − 1), reaching unity at M ≈ 2d_k. Hence the optimal linear store doubles the capacity to roughly 2 × d_k—about 128 memories for d_k = 64.
The article then explains how algorithmic changes can further raise capacity. Storing memories more carefully (rather than blindly adding outer products) yields improved linear stores, while recalling selectively via nonlinearities leads to MLPs. These nonlinearities suppress contributions from keys with low similarity, thereby lowering interference and increasing both capacity and recall fidelity.
Finally, self‑attention is presented as the most capable memory store. With 64‑dimensional keys, its capacity jumps from the linear store’s ~64, to the optimal linear store’s ~2 700, and can exceed a million stored associations. The post attributes this leap to self‑attention’s ability to weigh keys dynamically, allowing it to store a potentially unbounded number of memories while maintaining good recall.
Throughout, the discussion centers on the interference‑to‑signal ratio as the key metric for memory capacity, offering intuition for why successive architectural layers can hold more information before interference overwhelms the signal.