Create an AI Assistant With Local Models · Part 4 of 13 in the series
Part 4: Tools, or How the Model Switches the Lights
Build a small virtual home and give the model its first tool. Learn how tool calling really works: the model only asks, your code acts, and the result goes back into the conversation.
Create an AI Assistant With Local ModelsPart 4 of 13
On this page
Quest objectives
- Build a virtual home with a lamp in every room
- Describe a tool so that a model understands when to use it
- Understand the loop of tool calling
- Check what the model sends before acting on it
- Switch lamps by saying what you want in your own words
A language model can only produce text. It cannot switch a lamp, start a timer or look anything up. And yet assistants do all that. The trick is called tool calling, and it is simpler than it sounds:
With the conversation, you send the model a list of tools: a name, a description and the arguments each one takes.
Instead of answering in words, the model may answer "call this tool with these arguments". That is still just text, in a fixed form.
Your code runs the tool and adds what it reports to the conversation.
You send the conversation again. Now the model answers in words, knowing what the tool did.
The model never does anything itself. It asks, and your program decides whether and how to do it. Keep that in mind: it is what makes an assistant safe.
A home
The real assistant behind this series switches real lamps over the network. To need no hardware, Pixel gets a virtual home: five rooms with a lamp each, kept in memory. First the rooms and colors, which have names, and names are language. Add this to src/language.ts, the import at the very top and the rest at the end:
import type { Change, Lamp } from './home.ts';
export interface Room {
name: string;
/** how a sentence says where: "in the kitchen" */
where: string;
/** what people call the room */
words: string[];
}
export const ROOMS: Room[] = [
{ name: 'living room', where: 'in the living room', words: ['living room', 'lounge'] },
{ name: 'kitchen', where: 'in the kitchen', words: ['kitchen'] },
{ name: 'bedroom', where: 'in the bedroom', words: ['bedroom'] },
{ name: 'bathroom', where: 'in the bathroom', words: ['bathroom', 'bath'] },
{ name: 'hallway', where: 'in the hallway', words: ['hallway', 'hall', 'corridor'] },
];
export interface Color {
name: string;
words: string[];
/** how the page paints it */
css: string;
}
export const COLORS: Color[] = [
{ name: 'white', words: ['white'], css: '#fff4e0' },
{ name: 'warm white', words: ['warm white', 'warm'], css: '#ffd08a' },
{ name: 'red', words: ['red'], css: '#ff5a4d' },
{ name: 'orange', words: ['orange'], css: '#ff9a3c' },
{ name: 'yellow', words: ['yellow'], css: '#ffe14d' },
{ name: 'green', words: ['green'], css: '#5fe07a' },
{ name: 'blue', words: ['blue'], css: '#5aa2ff' },
{ name: 'purple', words: ['purple', 'violet'], css: '#b06cff' },
{ name: 'pink', words: ['pink'], css: '#ff7ac8' },
];
const list = new Intl.ListFormat(LOCALE, { type: 'conjunction' });
/** The sentences Pixel says without the model. */
export const say = {
lights(lamps: Lamp[], change: Change): string {
const everywhere = lamps.length === ROOMS.length;
const where = list.format(lamps.map((lamp) => ROOMS.find((room) => room.name === lamp.room)!.where));
const subject = everywhere ? 'All lights are' : lamps.length > 1 ? `The lights ${where} are` : `The light ${where} is`;
if (change.on === false) return `${subject} off.`;
const now = [change.color, change.brightness && `at ${change.brightness} percent`].filter(Boolean);
return `${subject} ${now.length ? now.join(' ') : 'on'}.`;
},
beyondMe: "That's beyond me, I'm afraid.",
};A room has a name, the words a sentence uses to say where ("in the kitchen"), and the words people use for it. A color has a css value that the page in part 9 will paint the lamp with. say.lights builds the sentence that reports a change, such as "The lights in the kitchen and in the hallway are off." Intl.ListFormat joins the rooms with commas and "and" the way the language does.
Now the home itself, src/home.ts:
import { COLORS, ROOMS } from './language.ts';
export interface Lamp {
room: string;
on: boolean;
/** 1 to 100 */
brightness: number;
color: string;
}
export type Change = Partial<Pick<Lamp, 'on' | 'brightness' | 'color'>>;
/** The home: one lamp per room. It lives in memory and starts dark. */
const lamps: Lamp[] = ROOMS.map((room) => ({
room: room.name,
on: false,
brightness: 100,
color: COLORS[0]!.name,
}));
const listeners: Array<() => void> = [];
export function getLamps(): Lamp[] {
return lamps;
}
/** Changes the lamps of the given rooms, or of every room when none is given. */
export function setLamps(rooms: string[], change: Change): Lamp[] {
if (change.brightness !== undefined) {
change.brightness = Math.min(100, Math.max(1, Math.round(change.brightness)));
}
const hit = lamps.filter((lamp) => rooms.length === 0 || rooms.includes(lamp.room));
// a new brightness or color switches the lamp on, unless the change says off
for (const lamp of hit) Object.assign(lamp, { on: true }, change);
for (const listener of listeners) listener();
return hit;
}
/** Calls `listener` after every change. */
export function onHomeChange(listener: () => void): void {
listeners.push(listener);
}
/** The home as a few lines of text, for the terminal. */
export function drawHome(): string {
return lamps
.map((lamp) => {
const state = lamp.on ? `● ${lamp.brightness}% ${lamp.color}` : '○';
return ` ${lamp.room.padEnd(12)} ${state}`;
})
.join('\n');
}setLamps is the one place where lamps change. It keeps the brightness between 1 and 100, whatever it is handed. The listeners are for part 9, where the page wants to know about every change; for now nobody listens. drawHome draws the home in the terminal.
If you had real lamps, setLamps is where you would talk to them. The rest of the assistant would not notice the difference.
Describing a tool
A tool has two sides: a description for the model and a function for your program. Create src/tools.ts:
import { lightTools } from './tools/lights.ts';
export interface Tool {
name: string;
/** tells the model when to use the tool */
description: string;
/** the arguments, as a JSON schema */
parameters: Record<string, unknown>;
/** does the work and reports what happened, in words the model can read */
run(args: Record<string, unknown>): string | Promise<string>;
}
export const tools: Tool[] = [...lightTools];
/** The tools in the form Ollama expects. */
export const toolDefinitions = tools.map((tool) => ({
type: 'function',
function: { name: tool.name, description: tool.description, parameters: tool.parameters },
}));
export async function runTool(name: string, args: Record<string, unknown>): Promise<string> {
const tool = tools.find((candidate) => candidate.name === name);
if (!tool) return `There is no tool called ${name}.`;
try {
return await tool.run(args);
} catch (error) {
return `The tool failed: ${(error as Error).message}`;
}
}toolDefinitions is the list that travels to Ollama with every request. runTool looks a tool up by the name the model asked for and runs it. Two things can go wrong there, and both are answered in words, not with a crash: a model sometimes asks for a tool that does not exist, and a tool can fail. The model reads the message and can try something else.
The lamp tools go in src/tools/lights.ts:
import { type Change, getLamps, setLamps } from '../home.ts';
import { COLORS, ROOMS, say } from '../language.ts';
import type { Tool } from '../tools.ts';
const roomNames = ROOMS.map((room) => room.name);
const colorNames = COLORS.map((color) => color.name);
export const lightTools: Tool[] = [
{
name: 'set_lights',
description: 'Switches lamps on or off, dims them or changes their color.',
parameters: {
type: 'object',
properties: {
rooms: {
type: 'array',
items: { type: 'string', enum: roomNames },
description: 'The rooms to change. An empty list means the whole home.',
},
on: { type: 'boolean', description: 'true switches on, false switches off' },
brightness: { type: 'number', description: 'from 1 (dark) to 100 (bright)' },
color: { type: 'string', enum: colorNames },
},
required: ['rooms'],
},
run(args) {
// the model wrote these arguments, so check every one of them
const wanted = Array.isArray(args.rooms) ? args.rooms : [];
const rooms = wanted.filter((room) => roomNames.includes(room));
// a room that does not exist must not become "the whole home"
if (rooms.length < wanted.length) return `Unknown room. The rooms are: ${roomNames.join(', ')}.`;
const change: Change = {};
if (typeof args.on === 'boolean') change.on = args.on;
if (typeof args.brightness === 'number') change.brightness = args.brightness;
if (typeof args.color === 'string' && colorNames.includes(args.color)) change.color = args.color;
const changed = setLamps(rooms, change);
return say.lights(changed, change);
},
},
{
name: 'get_lights',
description: 'Looks up which lamps are on, how bright they are and which color they have.',
parameters: { type: 'object', properties: {} },
run: () => JSON.stringify(getLamps()),
},
];The parameters are a JSON schema, a common way to describe the shape of data. The model reads it to learn which arguments exist. A few things help a small model get them right:
An
enumwherever there is a fixed set. With the room names listed, the model picks one of them and does not invent a "dining room".Few arguments, few tools. One tool that switches, dims and colors is easier for the model than three.
Descriptions that say when, not how. "Looks up which lamps are on" tells the model that this is the tool for questions.
And then the most important lines of this part, at the top of run: the model wrote these arguments, so check every one of them. A schema is a request to the model, not a promise. It can send a number as a string, a color that is not in the list, or a room that does not exist. Without the check, a made-up room would be dropped, the list would be empty, and "empty" means the whole home: one hallucinated word, and every lamp in the house changes. So an unknown room is reported back, with the list of rooms that do exist.
Tool calls in the client
Ollama needs to receive the tools, and the messages get two new fields. In src/ollama.ts, replace everything between the import line and chatStream with this:
export interface ToolCall {
function: { name: string; arguments: Record<string, unknown> };
}
export interface Message {
role: 'system' | 'user' | 'assistant' | 'tool';
content: string;
/** on an assistant message: the tools the model wants to call */
tool_calls?: ToolCall[];
/** on a tool message: the tool whose result this is */
tool_name?: string;
}
export interface ChatOptions {
/** the tools the model may call, as Ollama expects them */
tools?: object[];
}
async function post(path: string, body: object): Promise<Response> {
const response = await fetch(`${OLLAMA_URL}${path}`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(body),
});
if (!response.ok) throw new Error(`Ollama answered ${response.status}: ${await response.text()}`);
return response;
}
/** Sends the conversation to the model and returns its whole answer. */
export async function chat(messages: Message[], options: ChatOptions = {}): Promise<Message> {
const response = await post('/api/chat', {
model: MODEL,
messages,
tools: options.tools,
stream: false,
think: false,
options: { temperature: 0.2, num_predict: 300 },
});
const data = (await response.json()) as { message: Message };
return data.message;
}When the model wants a tool, its message has no text but a list of tool_calls:
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"function": {
"name": "set_lights",
"arguments": { "rooms": ["kitchen"], "on": true }
}
}
]
}And the answer to it is a message with a new role, tool, that carries what the tool reported.
The loop
Replace src/model.ts:
import { keep, recent } from './history.ts';
import { PERSONA, say } from './language.ts';
import { chat, type Message } from './ollama.ts';
import { contextBlock, spoken } from './prompt.ts';
import { runTool, toolDefinitions } from './tools.ts';
export interface Reply {
speech: string;
}
/** Asks the local model, which may call tools on the way to its answer. */
export async function askModel(text: string): Promise<Reply> {
const messages: Message[] = [
// first what never changes, so Ollama can reuse its work from the last request ...
{ role: 'system', content: PERSONA },
...recent(),
// ... and last what is new
{ role: 'user', content: `${contextBlock()}\n${text}` },
];
const done = (speech: string): Reply => {
keep(text, speech);
return { speech };
};
// a call, its result, another call ...: after four rounds it is enough
for (let round = 0; round < 4; round += 1) {
const answer = await chat(messages, { tools: toolDefinitions });
messages.push(answer);
if (!answer.tool_calls?.length) return done(spoken(answer.content));
// the model asked for tools: run each one and add what it reports to the conversation
for (const call of answer.tool_calls) {
const result = await runTool(call.function.name, call.function.arguments);
messages.push({ role: 'tool', tool_name: call.function.name, content: result });
}
}
return done(say.beyondMe);
}This is the loop from the start of the part, in code. Ask the model. If it answers in words, that is the answer. If it asks for tools, run each one, add a tool message with the result, and ask again. One request can need several rounds: to make the kitchen "a bit darker", the model first has to look up how bright it is. Four rounds are the limit, so a confused model cannot go around in circles forever.
The persona needs to know about the tools as well. In src/language.ts, replace the last part of PERSONA:
What you do:
- You have tools for everything you can do in this home. To do something, call the tool.
- Never say that something is done unless a tool did it.
- If you don't know something, say so in one sentence. Don't guess.And the chat should show the home whenever it has changed. Replace src/chat.ts:
import { createInterface } from 'node:readline/promises';
import { NAME } from './config.ts';
import { drawHome } from './home.ts';
import { askModel } from './model.ts';
const terminal = createInterface({ input: process.stdin, output: process.stdout });
// Ctrl+D closes the input: leave without an error
terminal.on('close', () => process.exit());
const name = NAME.toLowerCase();
console.log(`Talk to ${NAME}. /bye ends the chat.\n`);
let home = drawHome();
while (true) {
const text = (await terminal.question('you> ')).trim();
if (text === '/bye') break;
if (!text) continue;
const answer = await askModel(text);
console.log(`${name}> ${answer.speech}`);
// show the home whenever it has changed
if (drawHome() !== home) {
home = drawHome();
console.log(home);
}
}
terminal.close();$ npm run chatTalk to Pixel. /bye ends the chat.you> It's a bit dark in the kitchen.pixel> Let there be light: the kitchen is on now. living room ○ kitchen ● 100% white bedroom ○ bathroom ○ hallway ○you> Make the bedroom cosy.pixel> The bedroom is now warm white at 30 percent. Very cosy. living room ○ kitchen ● 100% white bedroom ● 30% warm white bathroom ○ hallway ○you> Which lights are on?pixel> The kitchen at full brightness and the bedroom at 30 percent in warm white.you> /bye
Nobody said "switch on" or named a brightness. The model worked out that a dark kitchen wants light and that cosy means warm and dim, picked the tool and filled in the arguments. That is what a language model is good at, and no list of commands could do it.
Each of these answers took two requests to the model: one that led to the tool call and one to put the result into words. In the next part you add timers and the weather, and cut that in half.
Comments
The comments are GitHub Discussions: sign in with GitHub to write one.
The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.