Skip to content

Create an AI Assistant With Local Models · Part 6 of 13 in the series

Part 6: Fast Rules Before the Model

Answer the simple commands with plain rules in under a millisecond, and build the router that tries the fastest way first and passes everything else on to the model.

Quest objectives

  • See why not every request should go to the model
  • Write rules that only answer what they fully understand
  • Compute what should be computed: the time and arithmetic
  • Build a router with tiers that pass requests on
  • See which tier answered and how long it took

"Kitchen light on." There is nothing to understand in this sentence. A room, a thing, a state. And still it goes to a language model, which reads a few hundred tokens of prompt and tools, thinks about it and writes a tool call, for a second or more. On the real assistant behind this series, a lamp command through the model took about 1.4 seconds. The same command through a rule takes a few milliseconds: the lamp is on before you have finished the sentence.

A second sounds short. But the commands you use most are exactly these simple ones, twenty times a day, and an assistant that feels slow is one you stop using. So Pixel gets tiers: the fastest and simplest way is tried first, and what it cannot answer goes on to the next.

Rules that know their limits

The danger of rules is that they understand too much. A rule that looks for "kitchen" and "on" will also fire on "turn the kitchen light on when I come home at six", and do the wrong thing with great confidence. The rules you write here follow one principle: a rule only answers a sentence it understands completely. It takes out every word it knows, and if anything is left over, it steps aside.

First the words, in src/language.ts:

src/language.ts
/** The words the rules know. Anything else in a sentence makes them step aside. */
export const WORDS = {
  all: ['all', 'every', 'everything', 'everywhere'],
  on: ['on'],
  off: ['off', 'out'],
  percent: ['percent', '%'],
  filler: [
    'hey', 'pixel', 'please', 'turn', 'switch', 'set', 'make', 'put', 'the', 'a', 'light',
    'lights', 'lamp', 'lamps', 'in', 'to', 'at', 'and', 'of', 'it',
  ],
};

/** Questions the rules answer themselves. They see the sentence in lower case, without punctuation. */
export const ASKS = {
  time: /^(what time is it|what's the time|tell me the time)$/,
  date: /^(what day is it|what is the date|what's the date)( today)?$/,
  sum: /^(?:what is|what's|how much is) (\d+(?:\.\d+)?) (plus|minus|times|divided by) (\d+(?:\.\d+)?)$/,
};
export const OPERATORS: Record<string, '+' | '-' | '*' | '/'> = {
  plus: '+',
  minus: '-',
  times: '*',
  'divided by': '/',
};

const clock = new Intl.DateTimeFormat(LOCALE, { hour: 'numeric', minute: '2-digit' });
const calendar = new Intl.DateTimeFormat(LOCALE, { weekday: 'long', month: 'long', day: 'numeric' });
const number = new Intl.NumberFormat(LOCALE, { maximumFractionDigits: 3 });

/** The sentences Pixel says without the model. */
export const say = {
  // ...
  time: (now: Date) => `It is ${clock.format(now)}.`,
  date: (now: Date) => `Today is ${calendar.format(now)}.`,
  sum: (result: number) => `That makes ${number.format(result)}.`,
  noZero: 'Dividing by zero is beyond even me.',
};

WORDS.filler holds the words that carry no meaning in a lamp command: "please", "the", "turn", "light". ASKS holds whole questions as patterns. Now the rules, in src/rules.ts:

src/rules.ts
import { type Change, setLamps } from './home.ts';
import { ASKS, COLORS, OPERATORS, ROOMS, say, WORDS } from './language.ts';
import type { Reply } from './model.ts';

/** A pattern for any of the words, as whole words: "on" is not found in "onion". */
function anyOf(words: string[]): RegExp {
  const escaped = words.map((word) => word.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'));
  return new RegExp(`(?<![\\p{L}\\p{N}])(?:${escaped.join('|')})(?![\\p{L}\\p{N}])`, 'gu');
}

function lightRule(sentence: string): Reply | null {
  let rest = sentence;
  /** Whether the pattern is in the sentence. What it matched is taken out. */
  const take = (pattern: RegExp): boolean => {
    const before = rest;
    rest = rest.replace(pattern, ' ');
    return rest !== before;
  };

  const rooms = ROOMS.filter((room) => take(anyOf(room.words))).map((room) => room.name);
  const everywhere = take(anyOf(WORDS.all));
  const percent = new RegExp(`(\\d{1,3}) ?(?:${WORDS.percent.join('|')})`).exec(rest);
  if (percent) rest = rest.replace(percent[0], ' ');
  // the longest name first, so "warm white" is not taken for "white"
  const color = [...COLORS]
    .sort((a, b) => b.name.length - a.name.length)
    .find((candidate) => take(anyOf(candidate.words)));
  const off = take(anyOf(WORDS.off));
  const on = take(anyOf(WORDS.on));
  take(anyOf(WORDS.filler));

  // a word is left over that the rule does not know: the model should read this sentence
  if (rest.trim()) return null;
  if (rooms.length === 0 && !everywhere) return null;
  if (on && off) return null;

  const change: Change = {};
  if (off) change.on = false;
  if (on) change.on = true;
  if (percent) change.brightness = Number(percent[1]);
  if (color) change.color = color.name;
  if (Object.keys(change).length === 0) return null;

  return { speech: say.lights(setLamps(rooms, change), change) };
}

function askRule(sentence: string): Reply | null {
  if (ASKS.time.test(sentence)) return { speech: say.time(new Date()) };
  if (ASKS.date.test(sentence)) return { speech: say.date(new Date()) };
  const sum = ASKS.sum.exec(sentence);
  if (sum) {
    const [a, b] = [Number(sum[1]!.replace(',', '.')), Number(sum[3]!.replace(',', '.'))];
    const operator = OPERATORS[sum[2]!];
    if (operator === '/' && b === 0) return { speech: say.noZero };
    const result = operator === '+' ? a + b : operator === '-' ? a - b : operator === '*' ? a * b : a / b;
    return { speech: say.sum(result) };
  }
  return null;
}

/** Answers what needs no model, in well under a millisecond. Null means: not for me. */
export function rules(text: string): Reply | null {
  const sentence = text
    .toLowerCase()
    .replace(/[.,!?]+(?=\s|$)/g, ' ')
    .replace(/\s+/g, ' ')
    .trim();
  return askRule(sentence) ?? lightRule(sentence);
}

lightRule works like someone crossing words off a list. Take "Living room warm white at 40 percent, please":

  1. Cross off the rooms: "living room". One room found.

  2. No word for "everywhere". Cross off "40 percent": a brightness.

  3. Cross off a color, the longest names first: "warm white".

  4. No "on", no "off". Cross off the filler words: "at", "please".

  5. Nothing is left. The rule understood everything, and switches.

Now take "turn the kitchen light on when I come home at six". After the crossing off, "when i come home six" is left over. The rule returns null, which means: not for me. So does a sentence with no room in it, or with both "on" and "off".

anyOf builds a pattern that finds whole words only. The odd-looking (?<![\p{L}\p{N}]) says "no letter or digit before this", so that "on" is not found inside "onion". It works for letters of every language, which the usual \b does not.

askRule answers what should never go to a language model in the first place: the time and the date, which the computer simply knows, and arithmetic, which it simply computes. Remember that the model was unsure about 17 times 23.

The router

Now the piece that holds the tiers together. Create src/assistant.ts:

src/assistant.ts
import { keep } from './history.ts';
import { say } from './language.ts';
import { askModel, type Reply } from './model.ts';
import { rules } from './rules.ts';

export interface Answer extends Reply {
  /** who answered */
  tier: string;
  ms: number;
}

interface Tier {
  name: string;
  /** an answer, or null to pass the request on to the next tier */
  ask(text: string): Reply | null | Promise<Reply | null>;
}

/** From the fastest and simplest to the slowest and smartest. */
const tiers: Tier[] = [
  { name: 'rules', ask: rules },
  { name: 'model', ask: askModel },
];

export async function handle(text: string): Promise<Answer> {
  const started = performance.now();
  let reply: Reply | null = null;
  let tier = 'nobody';
  for (const candidate of tiers) {
    try {
      reply = await candidate.ask(text);
    } catch (error) {
      console.error(`[${candidate.name}] ${(error as Error).message}`);
    }
    if (reply) {
      tier = candidate.name;
      break;
    }
  }
  const speech = reply?.speech ?? say.beyondMe;
  keep(text, speech);
  return { speech, tier, ms: Math.round(performance.now() - started) };
}

A tier is a name and a function that returns an answer or null. handle asks them in order and takes the first answer. If a tier throws an error, the error is logged and the next tier gets its chance: a model that has crashed must not take the light switch down with it. And if nobody answers, Pixel says so.

This small file is the heart of the assistant. Everything you add from here on is either a new tier or makes a tier better at saying "not for me".

handle also keeps the exchange in the history, whoever answered. So after "kitchen light on", answered by a rule, the model still understands "and the hallway too". The model tier therefore no longer keeps anything itself. Replace src/model.ts:

src/model.ts
import { recent } from './history.ts';
import { PERSONA, say } from './language.ts';
import { chat, type Message } from './ollama.ts';
import { contextBlock, spoken } from './prompt.ts';
import { runTool, toolDefinitions } from './tools.ts';

export interface Reply {
  speech: string;
}

/** Asks the local model, which may call tools on the way to its answer. */
export async function askModel(text: string): Promise<Reply> {
  const messages: Message[] = [
    // first what never changes, so Ollama can reuse its work from the last request ...
    { role: 'system', content: PERSONA },
    ...recent(),
    // ... and last what is new
    { role: 'user', content: `${contextBlock()}\n${text}` },
  ];
  // a call, its result, another call ...: after four rounds it is enough
  for (let round = 0; round < 4; round += 1) {
    const answer = await chat(messages, { tools: toolDefinitions });
    messages.push(answer);
    if (!answer.tool_calls?.length) return { speech: spoken(answer.content) };

    const said: string[] = [];
    for (const call of answer.tool_calls) {
      const result = await runTool(call.function.name, call.function.arguments);
      if ('say' in result) said.push(result.say);
      messages.push({
        role: 'tool',
        tool_name: call.function.name,
        content: 'say' in result ? result.say : result.info,
      });
    }
    // every tool had its answer ready: no need to ask the model again
    if (said.length === answer.tool_calls.length) return { speech: said.join(' ') };
  }
  return { speech: say.beyondMe };
}

The chat now talks to handle and shows who answered. Replace src/chat.ts:

src/chat.ts
import { createInterface } from 'node:readline/promises';
import { onAnnounce } from './announce.ts';
import { handle } from './assistant.ts';
import { NAME } from './config.ts';
import { drawHome } from './home.ts';

const terminal = createInterface({ input: process.stdin, output: process.stdout });
// Ctrl+D closes the input: leave without an error
terminal.on('close', () => process.exit());
const name = NAME.toLowerCase();

// a timer that is up, for example
onAnnounce((speech) => console.log(`\n${name}> ${speech}`));

console.log(`Talk to ${NAME}. /bye ends the chat.\n`);
let home = drawHome();

while (true) {
  const text = (await terminal.question('you> ')).trim();
  if (text === '/bye') break;
  if (!text) continue;

  const answer = await handle(text);
  console.log(`${name}> ${answer.speech}  (${answer.tier}, ${answer.ms} ms)`);
  if (drawHome() !== home) {
    home = drawHome();
    console.log(home);
  }
}
terminal.close();
~/projects/pixel-assistant
$ npm run chatTalk to Pixel. /bye ends the chat.you> Kitchen light onpixel> The light in the kitchen is on.  (rules, 1 ms)  living room  ○  kitchen      ● 100% white  bedroom      ○  bathroom     ○  hallway      ○you> Living room warm white at 40 percent, pleasepixel> The light in the living room is warm white at 40 percent.  (rules, 0 ms)  living room  ● 40% warm white  kitchen      ● 100% white  bedroom      ○  bathroom     ○  hallway      ○you> What is 17 times 23?pixel> That makes 391.  (rules, 0 ms)you> It's so bright in here, I can't sleep.pixel> The light in the bedroom is off.  (model, 1630 ms)you> /bye

Three answers from the rules, before you could blink, and one from the model, which understood that someone who cannot sleep wants the bedroom dark. That is the division of labor: rules for what is said the same way every time, the model for everything said in free words. Your times for the model will differ, the rules will be this fast on any machine.

The first tier knows exactly what it cannot do. The model does not: it answers everything, with the same confidence, whether it is right or not. In the next part you teach the model tier to say "not for me", too.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.