Skip to content

Create an AI Assistant With Local Models · Part 8 of 13 in the series

Part 8: A Memory That Finds Things by Meaning

Give the assistant a memory that survives a restart. Learn what embeddings are, store facts with them, find the right ones by meaning instead of by words, and put them in front of the model.

Quest objectives

  • Understand what an embedding is
  • Turn sentences into vectors with a second local model
  • Compare two vectors with the cosine similarity
  • Store, find and forget facts
  • Put what the memory knows into every request

Tell Pixel that Anna is allergic to nuts, close the chat, open it again, and Pixel has never heard of Anna. You know why: the model keeps nothing, and the history holds three exchanges and lives in memory. An assistant for a home should know the people in it.

Storing facts is easy: a file will do. The hard part is finding the right ones again. When someone asks "can I put walnuts in the cake for Anna?", the fact that helps is "Anna is allergic to nuts", and the two sentences share exactly one word. A search for words would look for "walnuts" and "cake" and find nothing.

Embeddings

An embedding model is a small cousin of a language model. It does not write anything. It reads a text and answers with a long list of numbers, a vector, that stands for what the text means. Think of it as coordinates on a map with hundreds of directions instead of two. The trick is how the map is laid out: texts that mean similar things end up close together, whatever words they use. "Walnuts in a cake for Anna" lands near "Anna is allergic to nuts" and far from "the hallway lamp is orange".

So a memory needs three steps:

  1. When a fact is stored, compute its vector and store both.

  2. When a question comes in, compute the question's vector.

  3. Pick the facts whose vectors are closest to it.

Ollama runs embedding models too. Download one; it is small:

~/projects/pixel-assistant
$ ollama pull embeddinggemma

embeddinggemma understands over a hundred languages and needs about 600 MB. Name it in src/config.ts:

src/config.ts
/** The model that turns sentences into vectors for the memory. */
export const EMBED_MODEL = process.env.EMBED_MODEL ?? 'embeddinggemma';

And add a function for it at the end of src/ollama.ts, with EMBED_MODEL added to the import from ./config.ts at the top:

src/ollama.ts
/** Turns each text into a vector: a list of numbers that stands for its meaning. */
export async function embed(texts: string[]): Promise<number[][]> {
  const response = await post('/api/embed', { model: EMBED_MODEL, input: texts });
  const data = (await response.json()) as { embeddings: number[][] };
  return data.embeddings;
}

You send texts and get one vector per text back, each a list of 768 numbers.

The memory

Create src/memory.ts:

src/memory.ts
import { existsSync, readFileSync, writeFileSync } from 'node:fs';
import { embed } from './ollama.ts';

interface Memory {
  text: string;
  /** the text as a vector, from the embedding model */
  vector: number[];
  at: string;
}

const FILE = 'memory.json';
/** How alike two texts must be to count as a hit. Print the scores and tune this for your model. */
const ALIKE = 0.5;
/** Above this, two texts say the same thing. */
const SAME = 0.9;

const memories: Memory[] = existsSync(FILE) ? JSON.parse(readFileSync(FILE, 'utf8')) : [];

const save = () => writeFileSync(FILE, JSON.stringify(memories));

/** How alike two vectors are: 1 means the same direction, 0 nothing in common. */
export function cosine(a: number[], b: number[]): number {
  let dot = 0;
  let lengthA = 0;
  let lengthB = 0;
  for (let i = 0; i < a.length; i += 1) {
    dot += a[i]! * b[i]!;
    lengthA += a[i]! ** 2;
    lengthB += b[i]! ** 2;
  }
  return dot / (Math.sqrt(lengthA * lengthB) || 1);
}

/** The memories with their likeness to the text, the most alike first. */
async function ranked(text: string): Promise<Array<{ memory: Memory; score: number }>> {
  if (memories.length === 0) return [];
  const [vector] = await embed([text]);
  return memories
    .map((memory) => ({ memory, score: cosine(vector!, memory.vector) }))
    .sort((a, b) => b.score - a.score);
}

export async function remember(text: string): Promise<void> {
  const [best] = await ranked(text);
  if (best && best.score >= SAME) return; // known already
  const [vector] = await embed([text]);
  memories.push({ text, vector: vector!, at: new Date().toISOString() });
  save();
}

/** What the memory holds about the text, at most `limit` entries. */
export async function recall(text: string, limit = 3): Promise<string[]> {
  const hits = await ranked(text);
  return hits.filter((hit) => hit.score >= ALIKE).slice(0, limit).map((hit) => hit.memory.text);
}

/** Removes the memory most like the text and returns it, or null when nothing is alike. */
export async function forget(text: string): Promise<string | null> {
  const [best] = await ranked(text);
  if (!best || best.score < ALIKE) return null;
  memories.splice(memories.indexOf(best.memory), 1);
  save();
  return best.memory.text;
}

The memory is a list of facts, each with its vector, in a file called memory.json. Open it after you have stored something: there is no secret in there, only your sentences and a lot of numbers.

cosine measures how close two vectors are. It is called the cosine similarity: it compares the directions two vectors point in. The result is 1 when they point the same way, and near 0 when the texts have nothing to do with each other. The loop multiplies the vectors number by number and adds everything up; the division at the end makes the result independent of how long the vectors are.

ranked compares a text with every memory and sorts by likeness. Comparing with everything sounds wasteful, but for the few hundred facts a household collects it takes well under a millisecond. Databases for vectors exist for the day you have a hundred thousand.

  • remember stores a fact, unless one that says the same is already there.

  • recall returns the facts that are alike enough.

  • forget removes the one that fits best.

Two ways into the prompt

How does the model get to the memory? There are two ways, and Pixel uses both.

As tools. The model decides to store, search or forget. Create src/tools/memory.ts:

src/tools/memory.ts
import { say } from '../language.ts';
import { forget, recall, remember } from '../memory.ts';
import type { Tool } from '../tools.ts';

export const memoryTools: Tool[] = [
  {
    name: 'remember',
    description:
      'Stores a lasting fact about the household: names, birthdays, allergies, likes and dislikes, plans.',
    parameters: {
      type: 'object',
      properties: {
        fact: { type: 'string', description: 'the fact as one full sentence, such as "Anna is allergic to nuts."' },
      },
      required: ['fact'],
    },
    async run(args) {
      if (typeof args.fact !== 'string' || !args.fact.trim()) return { info: 'fact is missing.' };
      await remember(args.fact.trim());
      return { say: say.noted };
    },
  },
  {
    name: 'recall',
    description: 'Searches the memory for what is known about a person or a topic.',
    parameters: {
      type: 'object',
      properties: { about: { type: 'string', description: 'what to look for' } },
      required: ['about'],
    },
    async run(args) {
      const hits = await recall(String(args.about ?? ''), 5);
      return { info: hits.length ? hits.join('\n') : 'The memory holds nothing about this.' };
    },
  },
  {
    name: 'forget',
    description: 'Removes a fact from the memory.',
    parameters: {
      type: 'object',
      properties: { fact: { type: 'string', description: 'the fact to remove, roughly as it was stored' } },
      required: ['fact'],
    },
    async run(args) {
      const gone = await forget(String(args.fact ?? ''));
      return { say: gone === null ? say.nothingToForget : say.forgotten(gone) };
    },
  },
];

The description of remember asks for "one full sentence", with an example. That matters: a stored "nuts" is useless next week, while "Anna is allergic to nuts." still says everything. Add the tools to the list in src/tools.ts:

src/tools.ts
import { lightTools } from './tools/lights.ts';
import { memoryTools } from './tools/memory.ts';
import { timerTools } from './tools/timers.ts';
import { weatherTools } from './tools/weather.ts';

// ...

export const tools: Tool[] = [...lightTools, ...timerTools, ...weatherTools, ...memoryTools];

Without being asked. A model that has to decide to search its memory often does not, because it does not know what it could know. So Pixel looks things up itself, before every request, and writes what it found into the note you know from part 3. In src/prompt.ts, replace contextBlock:

src/prompt.ts
/**
 * What changes from one request to the next: the date and time, a small calendar and what
 * the memory knows. It goes in front of the user's words, not into the system prompt.
 */
export function contextBlock(memories: string[] = [], now = new Date()): string {
  const lines = [`${LABELS.now}: ${day.format(now)}, ${time.format(now)}`];
  // small models are bad at counting weekdays, so the next days come ready-made
  const next = Array.from({ length: 6 }, (_, i) => day.format(new Date(now.getTime() + (i + 1) * 86_400_000)));
  lines.push(`${LABELS.nextDays}: ${next.join('; ')}`);
  if (memories.length) lines.push(`${LABELS.known}:`, ...memories.map((memory) => `- ${memory}`));
  return `[${lines.join('\n')}]`;
}

The labels and sentences, in src/language.ts:

src/language.ts
/** The labels of the note that goes in front of every request. */
export const LABELS = {
  // ...
  known: 'What you know about the household',
};

/** The sentences Pixel says without the model. */
export const say = {
  // ...
  noted: 'Noted.',
  forgotten: (text: string) => `Forgotten: ${text}`,
  nothingToForget: 'I have nothing like that in my memory.',
};

And in src/model.ts, add import { recall } from './memory.ts'; and replace the start of askModel, down to the end of the list of messages:

src/model.ts
export async function askModel(text: string): Promise<Reply | null> {
  if (!(await available())) return null;
  const signal = AbortSignal.timeout(BUDGET_MS);
  const memories = await recall(text).catch(() => []);

  const messages: Message[] = [
    // first what never changes, so Ollama can reuse its work from the last request ...
    { role: 'system', content: PERSONA },
    ...recent(),
    // ... and last what is new
    { role: 'user', content: `${contextBlock(memories)}\n${text}` },
  ];

The memories travel in the note at the end of the prompt, with the date and the time, because they change from question to question. The .catch makes the memory optional: if the embedding model is missing, Pixel works as before and simply knows nothing about you.

This pattern, fetching what fits the question and putting it in front of the model, has a name you will meet everywhere: retrieval-augmented generation, RAG for short. You have just built one.

Try it

~/projects/pixel-assistant
$ npm run chatTalk to Pixel. /bye ends the chat.you> Anna is allergic to nuts.pixel> Noted.  (model, 1480 ms)you> My sister Maria has her birthday on March 3.pixel> Noted.  (model, 1390 ms)you> /bye$ npm run chatTalk to Pixel. /bye ends the chat.you> Can I put walnuts in the cake for Anna?pixel> Better not: Anna is allergic to nuts.  (model, 1720 ms)you> Forget the thing about Maria's birthday.pixel> Forgotten: My sister Maria has her birthday on March 3.  (model, 1510 ms)you> /bye

The first chat stored two facts with the remember tool. The second chat is a new program that knows nothing, and still the question about walnuts was answered correctly: before the model saw it, recall had found the fact about Anna and put it into the note. The words "nuts" and "walnuts" are different. Their meanings are close.

That completes the core of an assistant: rules, a model with tools, checks and a memory, in under a thousand lines, and you have read every one of them. The remaining parts give it a body. In the next part, Pixel leaves the terminal and moves into a page in your browser, where you can finally see the lamps.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.