Building a memory that knows what I mean

Why I built it

I have ADHD, and executive functioning has never been my strong side. Since ChatGPT came out I kept coming back to the same thought. If an LLM had my memory and my context, it could actually help me.

In December 2025 I was using OpenClaw as my personal agent, and its memory kept missing what I meant. Search finds text that looks similar to the question. It doesn’t know that for me “Apple” is a project, or which memories I actually rely on.

Around then I read Google’s Titans paper, about memory that keeps learning after deployment, and that set the direction. I wanted a memory that learns from my past and from how I use it, and keeps changing as I do.

What I built

A learning layer on top of normal vector search. The embeddings (Voyage, 1,024 dimensions) stay frozen, and the layer on top learns.

Three small adapters sit on the frozen embeddings, each a standard residual MLP (1,024 → 256 → 1,024). f_Q reshapes the question, and f_K and f_V reshape the stored memories. f_K learns relevance, and f_V learns how useful a memory is to me. A memory’s score is:

Each adapter’s output layer starts at zero, so on day one the system behaves exactly like plain search. If the learned path fails or times out, it falls back to plain search.

f_Q learns online, after every question. f_K and f_V retrain nightly, and then every stored memory is re-projected and saved with a version number. I called this the sleep cycle.

Besides one summary per conversation, the system extracts small standalone facts, referred to below as fact anchors, each pointing back to its conversation. A search can hit a fact and return the whole conversation.

Feedback comes from two places. Implicitly, from which memories the agent used or ignored. Explicitly, from training sessions where the LLM asks me questions and I give a thumbs up or down.

How I tested it

The test data is 8 years of my own WhatsApp. That’s 5,239 conversations, each summarized, plus 36,752 extracted facts.

An LLM read each conversation and wrote questions it answers, with a target memory and a relevance label for each. To make the questions sound like me, the prompt included up to 40 of my real past questions, and a few non-standard patterns like Arabizi (“shou sar ma3ak bi X?”) and my personal style. Near-miss memories were mined automatically as negatives.

10% of the question–memory pairs were held out at random, about 490 unique test questions. The rest trained the adapters in 10 updates of 1,000 pairs. The baseline is summary search plus fact search, merged with reciprocal rank fusion, with no learning. I measured Recall@20 and MRR (1.0 means the right memory always ranks first).

Results

The learned layer lifted MRR by 59%, from 0.22 to 0.35 over the baseline. MRR measures how high the right memory ranks, so the MLP put the right memory much closer to the top. With about 490 test questions the noise is roughly ±0.04, so the gap holds for this setup. Almost all of it appeared after the first of the 10 updates, then stayed flat.

The ranges cover my last clean runs.

Fact anchors more than doubled recall, from 0.24 to 0.55. Searching small facts and returning their parent conversation is proposition-level retrieval, and on my data it made the biggest difference to what gets found in the top 20.

Re-projecting the stored memories didn’t change recall. It moved between −0.006 and +0.004.

What the layer learned

When I looked inside the trained adapters, most of the change was in the output bias b₂. The adapters had learned one direction per user, added to every question, that lifts memories from the areas I use most. Most of the 59% comes from that direction.

Each adapter computes:

W₂ and b₂ start at zero, which is what makes day one safe. From there, under plain gradient descent, the bias learns fastest. W₂’s gradient is scaled by the hidden activations and b₂’s isn’t, and W₁ only starts receiving gradient once W₂ grows. With initScale = 0.01, W₂’s gradients were about 500× smaller than b₂’s. After training on my data (106K samples), $\lVert W_2 \rVert$ was 1.24e-5 and $\lVert b_2 \rVert$ was 3.12e-3.

An A/B run confirmed it:

b₂ learned the same amount at every init scale, while W₂ scaled with the init. This is the same dynamic studied for LoRA’s zero-initialized matrix (Hayou et al., 2024, and LoRA+).

After switching to Adam and initScale = 5.0, W₂ grew to 0.198, so the weight matrices learn too once the optimizer lets them. The WhatsApp numbers above are from before that change. Ablating W₂ and b₂ would show how much the weights add on top of the bias.

K and V

K and V were the two parts I was most excited about, and neither showed a measurable gain in this test. Madeleine (Shi, October 2026) later reported the same two results.

K is the memory side, the nightly re-projection of every stored memory. It didn’t move recall, for a few reasons. f_K mostly learned a shared shift, and adding the same shift to every memory doesn’t change their order. A drift-check bug froze the stored vectors at version 21 for part of the work. Each update learned from 1,000 pairs but moved all ~37,000 memories. And near initialization, reshaping a memory and reshaping the question affect the score almost the same way, so f_Q can learn what f_K would without touching the index. Madeleine found the same thing on their benchmark. Adapting both sides lowered Recall@10 from 71.1 to 69.6, while adapting only the query side raised it to 78.8. Online learning pays off on the question side.

V is the usefulness score. K-only, V-only and the 0.7/0.3 blend landed within 1% of each other, and the order flipped between runs. At about 490 questions, that’s no difference. But this test couldn’t really judge V.

Where V goes next

V is the part I care about most. Search answers “is this on topic?”. V should answer “is this useful to me right now?”, shaped by what I talk and ask about. And it should change with me, so a memory that used to be useful decays as my focus moves on.

The WhatsApp test couldn’t show that. V’s labels came from the LLM’s relevance ratings, not from what I actually used. Trained on its own, it relearned plain similarity, since useful memories are usually on topic too. The linear blend also folded V into K, as one combined key per memory. And a random split of one batch of questions has no notion of time.

Part of the idea already works. The per-user direction f_Q learned boosts a memory the same way whatever I ask, which is a first, rough “how useful is this kind of memory to me” score. So usefulness is learnable from my data. It just landed in f_Q instead of in V.

Next for V:

1. Train it on real signal, thumbs and used-versus-ignored from daily use, over weeks.
2. Train it on the summed score, so it learns what similarity misses.
3. Start with one usefulness number per memory, from how often and how recently I use it, added to the search score.
4. Use two speeds. A slow vector for long-term interests, updated nightly and never reset, and a fast one for current focus, updated after each question and fading over days. VARS uses a similar long-term plus short-term design.

Bugs that moved the numbers

These are bugs and mistakes I made during the project. They changed my results more than I expected.

1. “10-epoch” runs were really 1 epoch, because pairs were marked as used after the first. Fixing it doubled the MRR lift on my controlled test (+4.7% to +9.1%).
2. Questions were embedded in document mode. Fixing that alone raised the untrained baseline’s MRR by 0.030. The WhatsApp pipeline still uses document mode, the same way in training and test.
3. The drift check compared each step with the previous one. Each step changed less than 5% while total drift grew from 0.032 to 0.187, so stored vectors stopped at version 21. It now compares against the last reindex.
4. The answer generator re-sorted retrieved facts by date, so better ranking couldn’t produce better answers.
5. A public benchmark’s adversarial questions, which have no gold answers, were averaged in instead of reported separately.
6. A user could be marked as trained with zero training pairs.
7. Changing the learning rate or loss could resume an old trained state, because those settings weren’t in the resume key.
8. The fact cache was keyed by sample ID only, so switching extraction modes reused the wrong facts.

None of them crashed anything, and all of them produced numbers that looked normal.

Related work

• Adapters on frozen embeddings: Search-Adaptor (Yoon et al., 2023), which found one shared adapter for questions and documents beats separate ones, and Adapted Dense Retrieval (Khatry et al., 2023).
• Fact anchors: Dense X Retrieval (Chen et al., 2024).
• Learning to rank from feedback: Joachims (2002), and Joachims et al. (2017) on position bias.
• Re-indexing during training: ANCE (Xiong et al., 2020).
• Zero-initialized branches: ReZero, ControlNet’s zero convolutions, LoRA, and LoRA+ on the uneven learning that follows.
• Test-time memory: Titans (Behrouz et al., 2025) and TTT layers update a memory network’s weights at test time with a self-supervised loss. This system learns online from feedback.
• Since then: VARS (March 2026) learns per-user vectors that bias memory retrieval, EGA (May 2026) studies zero-init adapters on frozen encoders, and Madeleine (October 2026) learns associative recall for conversational memory and explains both K and V results above.

Why I’m sharing it now

I stopped working on this in February. Since then others have reached the same conclusion, that memory needs to learn online, per user. So here’s what I found. The part I care about most, a memory that changes with me over time, is the one this test never measured, and it’s what I’d test next.

Possible applications

This idea isn’t limited to personal memory. Security video analytics is a good example, seen from the operator’s seat.

A control room has the same problem I had. The system finds events that match a rule or look like the search, but it doesn’t know what this operator, on this site, actually cares about. So operators get flooded with alerts and spend the shift dismissing the same false alarms.

Here’s how it would work.

• Clips and detections are embedded by a frozen vision model and stored in a vector DB, like my memories.
• A detection works like a fact anchor. A search hits “person at gate 3” and returns the whole incident clip.
• f_Q learns from each operator’s searches and actions, on top of a shared layer for the site. Opening, escalating or exporting a clip is a positive signal. Dismissing an alert is a negative one.
• V learns what’s useful on this site. The camera where wind moves the trees every evening drops, and the loading bay that matters after hours rises.
• Two speeds fit naturally. A slow vector for the site’s long-term patterns, and a fast one for the current shift or a live incident, which fades as the situation changes.