Skip to content

Create an AI Assistant With Local Models · Part 7 of 13 in the series

Part 7: Checking the Model's Answers

A small model claims things it has not done, answers when it should not, and sometimes does not answer at all. Catch all three in code: check every answer before anyone hears it.

Quest objectives

  • Know the typical mistakes of a small model
  • Catch an answer that claims an action no tool has done
  • Recognize an answer that gives up, and pass the request on
  • Give every turn a time limit
  • Keep the assistant working when the model is not running

So far Pixel believes everything the model says. That is a mistake, and every assistant built on a language model has to deal with it. A model does not know what it knows. It produces the likeliest next words, and "Done, the light is on!" is a very likely thing for an assistant to say, whether or not anything happened.

When the real assistant behind this series first got its local model, it handled 66 percent of its test sentences correctly. After the checks of this part, the note with the calendar from part 3 and a few rules like those of part 6, it was 98 percent, with the same model. None of it made the model smarter. It is ordinary code that checks the model's work, the way you would check the work of a new colleague who is eager and a bit too sure of themselves.

Three ways to go wrong

  • The empty promise. "Done!" or "I've set a timer for you", without a tool call. Nothing happened, and the user finds out when the tea is cold.

  • The polite surrender. "I'm sorry, I don't have access to current sports results." True, but useless, and no reason to end the turn: maybe another tier can help.

  • The silence. The model takes forever, or Ollama is not even running.

For each of them the answer is the same as in the last part: the tier returns null, "not for me", and the router moves on.

Recognizing the first two

How do you know that an answer claims an action? You look for the words such answers use. It is cruder than it sounds like it should be, and it works. Add to src/language.ts:

src/language.ts
/** An answer that claims something was done. Without a tool call, nothing was. */
export const CLAIM =
  /\b(done|i('ve| have) (set|switched|turned|started|saved|noted|stored|cancelled|canceled|dimmed)|i('ll| will) remember|noted|is now (on|off|set)|are now (on|off|set)|has been (set|switched|turned|saved))\b/i;

/** An answer that gives up. The next tier may know more. */
export const REFUSAL =
  /\b(i (don't|do not|can't|cannot|am not able to|'m not able to) (know|have|help|access|answer|tell|do|browse|look)|i'm not sure|i am not sure|no idea|as an ai|i have no (information|access|way))\b/i;

export const NUDGE =
  'You did not call a tool, so nothing has happened. Call the right tool now.';

CLAIM finds answers that say something was done. REFUSAL finds answers that give up. NUDGE is what Pixel says to the model when it catches an empty promise.

Time limits and a missing model

The client needs two additions: a way to give up on a request, and a way to ask whether Ollama is there at all. Replace src/ollama.ts:

src/ollama.ts
import { MODEL, OLLAMA_URL } from './config.ts';

export interface ToolCall {
  function: { name: string; arguments: Record<string, unknown> };
}

export interface Message {
  role: 'system' | 'user' | 'assistant' | 'tool';
  content: string;
  /** on an assistant message: the tools the model wants to call */
  tool_calls?: ToolCall[];
  /** on a tool message: the tool whose result this is */
  tool_name?: string;
}

export interface ChatOptions {
  /** the tools the model may call, as Ollama expects them */
  tools?: object[];
  /** lets the caller give up, for example after a time limit */
  signal?: AbortSignal;
}

async function post(path: string, body: object, signal?: AbortSignal): Promise<Response> {
  const response = await fetch(`${OLLAMA_URL}${path}`, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify(body),
    signal,
  });
  if (!response.ok) throw new Error(`Ollama answered ${response.status}: ${await response.text()}`);
  return response;
}

/** Sends the conversation to the model and returns its whole answer. */
export async function chat(messages: Message[], options: ChatOptions = {}): Promise<Message> {
  const response = await post(
    '/api/chat',
    {
      model: MODEL,
      messages,
      tools: options.tools,
      stream: false,
      think: false,
      options: { temperature: 0.2, num_predict: 300 },
    },
    options.signal,
  );
  const data = (await response.json()) as { message: Message };
  return data.message;
}

/** Sends the conversation to the model and hands out its answer piece by piece. */
export async function* chatStream(messages: Message[]): AsyncGenerator<string> {
  const response = await post('/api/chat', { model: MODEL, messages, stream: true, think: false });
  const decoder = new TextDecoder();
  let rest = '';
  for await (const chunk of response.body!) {
    rest += decoder.decode(chunk, { stream: true });
    // one JSON object per line; the last line may still be incomplete
    const lines = rest.split('\n');
    rest = lines.pop()!;
    for (const line of lines) {
      if (!line.trim()) continue;
      const part = JSON.parse(line) as { message: Message };
      yield part.message.content;
    }
  }
}

/** Whether Ollama is running and answers. */
export async function available(): Promise<boolean> {
  try {
    const response = await fetch(`${OLLAMA_URL}/api/version`, { signal: AbortSignal.timeout(1000) });
    return response.ok;
  } catch {
    return false;
  }
}

A signal is JavaScript's way of cancelling something that is under way. fetch accepts one, and when the signal fires, the request ends with an error. available asks Ollama for its version and waits one second at most; any answer means it is running.

The model tier, with checks

Replace src/model.ts:

src/model.ts
import { recent } from './history.ts';
import { CLAIM, NUDGE, PERSONA, REFUSAL } from './language.ts';
import { available, chat, type Message } from './ollama.ts';
import { contextBlock, spoken } from './prompt.ts';
import { runTool, toolDefinitions } from './tools.ts';

export interface Reply {
  speech: string;
}

/** How long a whole turn may take before Pixel gives up on the model. */
const BUDGET_MS = 30_000;

/**
 * Asks the local model, which may call tools. Returns null when the model is not the one
 * to answer: it is not running, it took too long, or it gave up.
 */
export async function askModel(text: string): Promise<Reply | null> {
  if (!(await available())) return null;
  const signal = AbortSignal.timeout(BUDGET_MS);

  const messages: Message[] = [
    // first what never changes, so Ollama can reuse its work from the last request ...
    { role: 'system', content: PERSONA },
    ...recent(),
    // ... and last what is new
    { role: 'user', content: `${contextBlock()}\n${text}` },
  ];
  let toolsUsed = 0;
  let nudged = false;

  for (let round = 0; round < 4; round += 1) {
    const answer = await chat(messages, { tools: toolDefinitions, signal });
    messages.push(answer);

    if (answer.tool_calls?.length) {
      const said: string[] = [];
      for (const call of answer.tool_calls) {
        const result = await runTool(call.function.name, call.function.arguments);
        toolsUsed += 1;
        if ('say' in result) said.push(result.say);
        messages.push({
          role: 'tool',
          tool_name: call.function.name,
          content: 'say' in result ? result.say : result.info,
        });
      }
      // every tool had its answer ready: no need to ask the model again
      if (said.length === answer.tool_calls.length) return { speech: said.join(' ') };
      continue;
    }

    const speech = spoken(answer.content);
    if (!speech) return null;
    // "Done!" without a tool call: nothing was done. Ask once more, firmly.
    if (toolsUsed === 0 && CLAIM.test(speech)) {
      if (nudged) return null;
      nudged = true;
      messages.push({ role: 'user', content: NUDGE });
      continue;
    }
    if (REFUSAL.test(speech)) return null;
    return { speech };
  }
  return null;
}

The function can now return null, and it does so whenever the model is not the one to answer. From the top:

  • Ollama is not running: not for me. No error, no waiting.

  • AbortSignal.timeout(BUDGET_MS) starts a clock for the whole turn, all rounds together. When it runs out, the request that is under way ends with an error, which the router from the last part catches. Thirty seconds are generous, because the very first request also has to load the model into memory. Lower the number once you know how fast your machine is.

  • An empty answer: not for me.

  • The empty promise: the answer claims an action, but no tool has run in this turn. Pixel tells the model so, in plain words, and asks once more. Most of the time the model then calls the tool it forgot. If it claims again, the tier gives up.

  • The surrender: the answer matches REFUSAL. Not for me.

Notice what the empty promise check does not do: it does not fire when a tool has run. "Done, the timer is set" after a real tool call is simply true.

Try it

Ask for something the model cannot know:

~/projects/pixel-assistant
$ npm run chatTalk to Pixel. /bye ends the chat.you> Who won the football match last night?pixel> That's beyond me, I'm afraid.  (nobody, 1210 ms)you> /bye

The model said something like "I don't have access to current sports results", the check recognized the surrender, and since there is no further tier yet, the router answered itself. That looks like a step back, but it is the honest answer, and in the last part of this series a third tier will take over exactly here.

Now stop the model and see what is left of Pixel:

~/projects/pixel-assistant
# quit Ollama: close the app, or on Linux:$ sudo systemctl stop ollama$ npm run chatTalk to Pixel. /bye ends the chat.you> Kitchen light onpixel> The light in the kitchen is on.  (rules, 1 ms)you> Make it cosy in here.pixel> That's beyond me, I'm afraid.  (nobody, 4 ms)

The lamps still work, in a millisecond. This is the second gift of the tiers, after speed: when the clever part fails, the simple part keeps the house running.

Pixel is now fast, careful and honest about its limits. But it forgets everything the moment you close the chat. In the next part it gets a memory that finds things by what they mean.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.