Lesson 03. Retrieval and Curation

Six months into running the memory files from lesson 2, episodic.md has 1,400 lines. That was the design goal: it's allowed to grow forever, because lesson 2 promised the search step would make its size a non-issue. But if the search step is grep, "no issue" was a lie. grep "standup" finds the line that says "standup" and misses the line that says "daily sync" about the exact same topic. Keyword matching is honest about what it does (find these literal characters), and that honesty is exactly why it misses paraphrase, synonyms, and anything the agent phrased differently than you did the first time.

This lesson builds the retrieval step lesson 2 deferred, and adds the maintenance pass that keeps 1,400 lines from becoming 1,400 lines of noise.

Keyword, embedding, hybrid: and when each wins

Keyword search matches literal terms. It wins when the query uses the same vocabulary as the stored text: searching for a person's name, a ticket number, an exact tool name. It is fast, needs no model, and is fully explainable: you can always say exactly why a result matched. It loses on paraphrase: "meeting moved" won't find "rescheduled the call."

Embedding (vector) search matches meaning. Both text and query get converted into vectors by a model, and similarity is measured geometrically: vectors that point in similar directions represent similar meaning, even with zero words in common. It wins on paraphrase and vague recall ("that thing about deployment timing" finding an entry about a specific rollout delay). It loses on precision: it can rank a topically-similar-but-wrong entry above the exact keyword match you actually wanted, and it can't explain a match the way "this word is present" can.

Hybrid blends both scores. It wins in the general case, which is why it's the default this lesson builds: most real queries benefit from both catching exact terms (a name, an ID) and catching paraphrase (a description of the same event in different words). The cost is complexity: two scoring paths to maintain instead of one, and a blend weight to tune. For a small personal memory store, that cost is a few dozen lines of code, which is a reasonable trade.

The choice isn't really three competing options; it's one question asked of every query, which the picture below makes concrete.

Neither branch alone covers both cases; the blend is what stops a query from silently depending on which vocabulary you happened to use.

Running embeddings locally

The embedding step needs a model that turns text into a vector. The naive approach calls a hosted embedding API, but every one of your memory entries would then leave your machine on every index build and every search, which directly violates the read-access ceiling from lesson 1: your memory files are exactly the kind of thing you decided the agent (and by extension, anything the agent calls) should not leak.

The fix is a model that runs on your own machine. Transformers.js (@huggingface/transformers) runs small embedding models in Node/Bun via ONNX Runtime, entirely on-device. The first call to pipeline(...) downloads the model weights once from Hugging Face's model hub (a one-time network request for the model, not your data), and every embedding computed after that happens locally, with no text leaving the process. That distinction matters: "the model download touched the network" is not the same claim as "your memory content touched the network," and only the second one is the thing this course is protecting against.

Source: Transformers.js documentation

A simple, honest scoring blend

The blend in this lesson's code is deliberately simple: score = VECTOR_WEIGHT * cosineSimilarity + (1 - VECTOR_WEIGHT) * keywordOverlap, with VECTOR_WEIGHT a single tunable constant. This is not a tuned ranking model — it is an honest heuristic you can read in one line and adjust by feel. Start at 0.6 (favoring meaning slightly over literal wording) and move it if search results feel wrong: raise it if paraphrased queries keep missing, lower it if exact terms (names, IDs) keep losing to topically-similar-but-wrong results.

Resist the urge to make this fancier before you've used it. A learned re-ranker or a more elaborate scoring function is real engineering effort for a personal memory store that, per lesson 2's sizing guidance, holds a few hundred to a few thousand entries, small enough that a simple linear blend, re-scored on every query, is fast and good enough. Add sophistication only after you've watched the simple version fail on a specific query you can name.

Chunking memory entries

Chunking is usually the hard part of building retrieval over documents — how much text goes into one retrievable unit? Lesson 2's format already solved this by accident: every episodic.md line is YYYY-MM-DD (source): fact, one dated, sourced claim per line. That is already the right chunk size. It's small enough to be specific (a search result points at exactly one fact, not a paragraph you have to re-read to find the relevant sentence) and large enough to be meaningful on its own (the date and source travel with the fact, so a retrieved chunk is never missing the context needed to judge it).

The one adjustment worth making: if a single event genuinely needs more than one line (a longer note, a multi-step outcome), keep it as one chunk with the date/source prefix on the first line and continuation lines indented under it, rather than splitting it into several dateless fragments that lose their shared context when retrieved independently. The indexer below treats each top-level dated line as the start of a chunk and folds any indented continuation lines into it.

The curation pass

Retrieval alone doesn't stop memory from degrading. It just changes the failure mode from "can't find anything" to "finds too much, some of it stale or duplicated." A curation pass is a scheduled, separate run (weekly is reasonable for a personal store) that does three things, and does them by flagging for review, never by silently deleting. The same "ask before an irreversible write" discipline from lesson 1 applies to memory maintenance, not just to external actions:

• Merge duplicates. Two entries with near-identical embeddings are the same fact recorded twice, often because the agent re-observed something it had already logged. Flag pairs above a similarity threshold for a human to collapse.
• Mark contradictions. An entry containing language like "supersedes," "no longer," or "correction:" is explicitly claiming to override an earlier one. Flag it so a human can confirm the older entry should be treated as historical, not current.
• Archive what stopped being referenced. An entry that's old and has never once surfaced in a search result's top results is a candidate for moving to a separate archive file, out of the active index, but not deleted, in case it's needed later.

Why curation is necessary and not optional: memory that only grows accumulates noise strictly faster than it accumulates value, because every new entry adds one unit of value but the search space it has to be found within grows for every other entry too. Without curation, retrieval quality degrades slowly and invisibly. Searches keep "working" in the sense of returning results, but the top results get progressively less relevant as duplicates and abandoned threads dilute the ranking. Curation is what keeps the store's signal-to-noise ratio from decaying as it scales.

Key Takeaways

• Keyword search wins on exact terms; embedding search wins on paraphrase; hybrid blends both and is the right default for a personal memory store.
• Run embeddings locally (e.g. Transformers.js) so memory content never leaves your machine. Only the one-time model download touches the network.
• Keep the scoring blend simple and readable: a single tunable weight, not a learned ranker, until a specific failure justifies more.
• Lesson 2's one-entry-per-line format already solved chunking: each dated, sourced line is the right retrieval unit.
• Curation is not optional: memory that only grows accumulates noise faster than value. Run a scheduled pass that merges duplicates, marks contradictions, and archives what's stale.
• Curation flags, it never deletes unattended. The same human-confirms-irreversible-writes discipline from lesson 1 and 2 applies here too.

What You Have Now

Local hybrid search over your memory, plus a reflection pass that prunes what went stale.

Example Files

agent/search/build-index.ts

Reads episodic.md, chunks it into one entry per dated line, embeds each entry locally, and writes the searchable index. Run once after setup and again whenever episodic.md grows significantly.

/**
 * build-index.ts
 * Reads agent/memory/episodic.md, chunks it into one entry per dated
 * line, embeds each entry locally, and writes agent/memory/index.json.
 *
 * Install: npm install @huggingface/transformers
 * Run:     npx tsx agent/search/build-index.ts
 */
import { readFileSync, writeFileSync } from "node:fs";
import { pipeline } from "@huggingface/transformers";

interface MemoryEntry {
  id: string;
  date: string;
  source: string;
  text: string;
  embedding: number[];
}

const ENTRY_START = /^(\d{4}-\d{2}-\d{2}) \(([^)]+)\): (.+)$/;

function parseEpisodic(raw: string): { date: string; source: string; text: string }[] {
  const lines = raw.split("\n");
  const chunks: { date: string; source: string; text: string }[] = [];

  for (const line of lines) {
    const match = line.match(ENTRY_START);
    if (match) {
      chunks.push({ date: match[1], source: match[2], text: match[3] });
    } else if (line.startsWith("  ") && chunks.length > 0) {
      // Indented continuation line -- folds into the current chunk.
      chunks[chunks.length - 1].text += " " + line.trim();
    }
  }
  return chunks;
}

async function main() {
  const raw = readFileSync("agent/memory/episodic.md", "utf-8");
  const parsed = parseEpisodic(raw);

  const embed = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");

  const entries: MemoryEntry[] = [];
  for (let i = 0; i < parsed.length; i++) {
    const { date, source, text } = parsed[i];
    const output = await embed(text, { pooling: "mean", normalize: true });
    entries.push({
      id: `episodic-${i}`,
      date,
      source,
      text,
      embedding: Array.from(output.data as Float32Array),
    });
  }

  writeFileSync("agent/memory/index.json", JSON.stringify(entries, null, 2));
  console.log(`Indexed ${entries.length} entries -> agent/memory/index.json`);
}

main();

agent/search/search.ts

Hybrid keyword + vector search over the index built above. Exports search() for the agent to call, and doubles as a CLI for manual testing.

/**
 * search.ts
 * Hybrid keyword + embedding search over agent/memory/index.json.
 *
 * Run: npx tsx agent/search/search.ts "what did alice review"
 */
import { readFileSync } from "node:fs";
import { pipeline } from "@huggingface/transformers";

interface MemoryEntry {
  id: string;
  date: string;
  source: string;
  text: string;
  embedding: number[];
}

interface ScoredEntry extends MemoryEntry {
  keywordScore: number;
  vectorScore: number;
  score: number;
}

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB) || 1);
}

function keywordScore(query: string, text: string): number {
  const queryTerms = query.toLowerCase().split(/\W+/).filter(Boolean);
  if (queryTerms.length === 0) return 0;
  const textLower = text.toLowerCase();
  const hits = queryTerms.filter((term) => textLower.includes(term)).length;
  return hits / queryTerms.length; // fraction of query terms present, 0..1
}

// How much to trust semantic similarity vs literal keyword overlap.
// 0.6 favors meaning over exact wording. A named constant, not a tuned
// model -- adjust it if results feel wrong. See the lesson text.
const VECTOR_WEIGHT = 0.6;

export async function search(query: string, topK = 5): Promise<ScoredEntry[]> {
  const entries: MemoryEntry[] = JSON.parse(
    readFileSync("agent/memory/index.json", "utf-8")
  );

  const embed = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
  const queryOutput = await embed(query, { pooling: "mean", normalize: true });
  const queryEmbedding = Array.from(queryOutput.data as Float32Array);

  const scored: ScoredEntry[] = entries.map((entry) => {
    const vectorScore = cosineSimilarity(queryEmbedding, entry.embedding);
    const kwScore = keywordScore(query, entry.text);
    const score = VECTOR_WEIGHT * vectorScore + (1 - VECTOR_WEIGHT) * kwScore;
    return { ...entry, keywordScore: kwScore, vectorScore, score };
  });

  return scored.sort((a, b) => b.score - a.score).slice(0, topK);
}

// CLI entry point: `npx tsx agent/search/search.ts <query>`
if (import.meta.url === `file://${process.argv[1]}`) {
  const query = process.argv.slice(2).join(" ");
  if (!query) {
    console.error("Usage: npx tsx agent/search/search.ts <query>");
    process.exit(1);
  }
  search(query).then((results) => {
    for (const r of results) {
      console.log(`[${r.score.toFixed(3)}] ${r.date} (${r.source}): ${r.text}`);
    }
  });
}

agent/search/reflect.ts

The curation pass. Run on a schedule (weekly is reasonable) alongside the heartbeat built in lesson 6. Flags issues into a report file, never deletes anything on its own.

/**
 * reflect.ts
 * Scheduled curation pass over agent/memory/index.json:
 *   - flags near-duplicate entries (cosine similarity above threshold)
 *   - flags entries whose text signals they supersede an earlier one
 *   - flags entries old enough, and never once surfaced in search
 *     results, to be archive candidates
 *
 * Run: npx tsx agent/search/reflect.ts
 */
import { readFileSync, writeFileSync, existsSync } from "node:fs";

interface MemoryEntry {
  id: string;
  date: string;
  source: string;
  text: string;
  embedding: number[];
}

const DUPLICATE_THRESHOLD = 0.92;
const ARCHIVE_AFTER_DAYS = 90;
const SUPERSEDE_SIGNALS = ["supersedes", "no longer", "instead of", "correction:"];

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0, normA = 0, normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB) || 1);
}

function daysSince(dateStr: string): number {
  return (Date.now() - new Date(dateStr).getTime()) / (1000 * 60 * 60 * 24);
}

function loadHitCounts(): Record<string, number> {
  if (!existsSync("agent/memory/search-hits.json")) return {};
  return JSON.parse(readFileSync("agent/memory/search-hits.json", "utf-8"));
}

function main() {
  const entries: MemoryEntry[] = JSON.parse(
    readFileSync("agent/memory/index.json", "utf-8")
  );
  const hitCounts = loadHitCounts();

  const duplicates: { a: string; b: string; similarity: number }[] = [];
  const supersedeCandidates: string[] = [];
  const archiveCandidates: string[] = [];

  for (let i = 0; i < entries.length; i++) {
    const a = entries[i];

    for (let j = i + 1; j < entries.length; j++) {
      const b = entries[j];
      const sim = cosineSimilarity(a.embedding, b.embedding);
      if (sim >= DUPLICATE_THRESHOLD) {
        duplicates.push({ a: a.id, b: b.id, similarity: Number(sim.toFixed(3)) });
      }
    }

    const lower = a.text.toLowerCase();
    if (SUPERSEDE_SIGNALS.some((signal) => lower.includes(signal))) {
      supersedeCandidates.push(a.id);
    }

    const hits = hitCounts[a.id] ?? 0;
    if (daysSince(a.date) > ARCHIVE_AFTER_DAYS && hits === 0) {
      archiveCandidates.push(a.id);
    }
  }

  const report = {
    generatedAt: new Date().toISOString(),
    duplicates,
    supersedeCandidates,
    archiveCandidates,
  };

  writeFileSync("agent/memory/reflection-report.json", JSON.stringify(report, null, 2));

  console.log(
    `Reflection pass: ${duplicates.length} likely duplicates, ` +
    `${supersedeCandidates.length} supersede signals, ` +
    `${archiveCandidates.length} archive candidates.`
  );
  console.log(
    "Review agent/memory/reflection-report.json before archiving anything -- " +
    "this script flags, it does not delete."
  );
}

main();

Exercises

1. Run build-index.ts against your own episodic.md from lesson 2, then run search.ts with three real queries: one using exact vocabulary from an entry, one paraphrasing an entry in different words, and one that shouldn't match anything. Confirm the ranking makes sense for each.
2. Change VECTOR_WEIGHT to 0.2 and rerun the same three queries from exercise 1. Note which results change and explain, in one sentence per query, why lowering the vector weight shifted the ranking the way it did.
3. Run reflect.ts against your index. If it reports zero duplicates and zero archive candidates (likely, on a fresh store), manually add one deliberately duplicate entry and one deliberately old, unreferenced entry to episodic.md, rebuild the index, and confirm the reflection report catches both.

Check your understanding