Fight ClubBlog

Anatomy of a memory for LLMs

Which facts reach the model

The project behind this series: ns-brain.fightclub.pro

At a party you can talk to one person while twenty other people talk around you and you hear almost none of what they say: ask yourself afterwards what the group behind you was discussing and you will not know. Then someone across the room says your name and you turn your head. Not everybody hears it (when psychologists tested it, about one listener in three did), but most of us have been in that situation at least once. Your brain ignores almost everything you are not listening to and still lets a few words through (like your own name), because those words matter to you.

An agent's context window (the text a model can read at one time) is that noisy room. Everything stored is the noise around the question and you pay for every line of it that enters the context. The job of recall is to let the right few facts reach the model and to keep the rest out. Recall in ns-brain works in two steps. First it finds the candidates: any word of the question can find a memory but the brain returns a memory only if it holds most of the question's words (both words of a two-word question, about three in five of a longer one, rounded up). When no memory holds that many, the brain returns the partial matches instead and marks the answer as loose. Then recall orders what it returns.

This is a real transcript from a test brain for an invented online bookshop with seven facts in it: refunds, shipping, payments, a delay in order emails and how customer emails are signed. The question has three words, so the brain returns a memory only if it holds two of them:

$ ns-brain recall "refund expired card"
#3  [shop:refunds/gotcha] (score 1.120) The payment provider rejects a refund to an expired card
      Issue store credit instead and tell the customer by email.
#1  [shop:refunds/decision] (score 0.535) Card refunds are paid within five working days of the return arriving
      Promised on the returns page.

Four of the seven facts hold at least one of the words. The rule that a refund over 500 euros needs a second approval holds only "refund" and the fact about the payment provider's EU endpoint holds only "card", so both were dropped before any ranking began: they share a word with the question but answer a different one. The fact that holds all three words ranks first. The fact about card refunds holds two of the three, so its match counts two thirds.

Figure 1Two steps before anything is ranked

7 facts hold any word hold 2 of 3 #1 0.535 #1 card refund #1 Card refunds are paidwithin five working days ofthe return arriving dropped #2 refund #2 A refund over 500 eurosneeds a second approval fromfinance #3 1.120 #3 refund expired card #3 The payment providerrejects a refund to anexpired card #4 Order emails are delayedthis week while the mailprovider limits our sending #5 Parcels to Norway go bypost, never by courier dropped #6 card #6 Card payments go throughthe payment provider’sEU endpoint #7 Customer emails are signedby the team, never by oneperson
The question "refund expired card" against the seven-fact bookshop brain. Four facts hold at least one of its words. Two of those hold only one word and are dropped, because a three-word question needs two. The two the brain returns are then scored.

The order comes from one formula, with every term a number in the code:

score = word match x share of the question x freshness x stakes x scope

The limit on stakes came from a test on this project's own brain (571 memories, ten real questions). Without the limit, an important memory could rank above a memory that actually answered the question: in the worst cases the right answer was pushed several places down. Now importance and type only decide between memories that match equally well. Freshness has no limit, so an old fact that has stopped being true can sink to the bottom.

A search that matches nothing has to be handled with care. When an agent or a script asks for results as data, ns-brain returns nothing at all on a miss and never the whole brain for the LLM to pick what feels right. Instead the answer names what the project does hold near those words (categories with counts), so the agent can try again with better words. This rule comes from a real accident: a script searched the brain for memories to delete and, when nothing matched, an older version of ns-brain answered with every memory in the project instead of nothing. The warning that said so was written only for people to read, so the script never saw it and deleted everything it was given, real memories included. The same bookshop brain has nothing about gift vouchers, so recall says so (the first four lines of the answer):

$ ns-brain recall "gift vouchers" --json | head -4
{
  "count": 0,
  "identity": null,
  "matched": false,

The other half of recall happens without anyone asking. On every message the person types, a hook searches the brain for the message's words. The hook leaves out the words this brain uses everywhere, which are the chatter in the room. When the match is strong the hook adds up to three facts to the agent's context, each as one short line. Those facts never count as a use, so a guess by the hook cannot promote a memory over time. In the bookshop brain a message saying that a customer wants a refund on a card that has expired brings the store credit rule into the agent's context before the agent has read a word of the message. This is the name in the noisy room: the agent never asked for the fact and the fact arrives because it matters to this message. An agent can search the brain itself but only when it thinks of it and an agent busy with a task often does not think of it: it works with what it already has. So the hook does the looking on every message instead of waiting for the agent to ask.

This has a real cost: ns-brain searches words, so a question asked in very different words from the memory ("outage" when the memory says "downtime") can miss the memory. The strong-match rule also means the hook sometimes stays silent when a fact would have helped, which I prefer to a hook that fills every message with noise. The hook is not perfectly quiet either: for the expired-card message it also brought the fact about the payment provider's EU endpoint, which matched "card" and "payment" but answers nothing about refunds.

The project behind this series ns-brain.fightclub.pro