Skip to content

Create an AI Assistant With Local Models · Part 13 of 13 in the series

Part 13: Calling In a Bigger Brain: Claude Code in tmux

Give the assistant a third tier for the questions a small model should not answer: Claude Code, running in a tmux session that your program types into and that reports its answers back through a hook.

Quest objectives

  • Decide which requests should not go to the small model
  • Run Claude Code in a tmux session in the background
  • Type a question into it from your program
  • Get the answer back through a hook
  • Limit what the bigger brain may do

Ask Pixel why the sky is blue, who won last night or how to get a red wine stain out of a carpet, and since part 7 it answers honestly that this is beyond it. For a model of 4 billion parameters that is the right answer: it would otherwise tell you something that only sounds right. But it is not a satisfying one.

The real assistant behind this series solves it with a third tier. Above the rules and the local model sits a large model with access to the web, and it gets exactly the requests the small one should not touch. This last part builds that tier. It is optional, and it is the only part of Pixel that sends your words to someone else's computer.

Why Claude Code, and why tmux

Claude Code is Anthropic's assistant for the terminal. It is made for programming, but underneath it is a large model that can search the web, read pages and keep a conversation, which is just what Pixel needs. And it runs on a Claude subscription, so you pay nothing per question.

Claude Code is a program you type into. So Pixel will do just that: start it in a terminal that nobody looks at, type the question, and wait for the answer. tmux makes this possible. It is a program that keeps terminals running in the background and lets other programs send keystrokes to them. The session stays alive between questions, so Claude Code does not have to start each time and remembers what was just talked about. And you can look into the session at any time and watch your assistant's brain at work.

You need both installed. tmux comes with the package manager of Linux and macOS (apt install tmux, brew install tmux); on Windows, this part works inside WSL. Claude Code needs version 2.1.248 or newer.

~
$ tmux -V$ claude --version

A folder for the brain

Claude Code gets a folder of its own, with two files that tell it who it is and how to report back. Create the folder brain in the project, start Claude Code in it once by hand, and log in:

~/projects/pixel-assistant
$ mkdir brain$ cd brain$ claude# log in, say yes when it asks whether you trust this folder, then leave with /exit$ cd ..

The first file is the brain's persona, brain/PERSONA.md:

brain/PERSONA.md
You are the research brain of Pixel, a small home assistant. Pixel passes you the questions
it cannot answer itself: general knowledge, current events, anything that needs the web.

- Every request is one line that starts with [Pixel].
- Search the web when the answer may have changed since your training, or when you are not sure.
- Your answer is read aloud. Reply with at most three short sentences of plain spoken English:
  no markdown, no lists, no links.
- You cannot switch lamps, set timers or change anything in the home. If you are asked to, say so.

It is the same idea as Pixel's own persona in part 3: who you are, what arrives, what the answer should look like. The answer will be spoken, so it has to be short here too.

How the answer comes back

Typing into a terminal is easy. Reading the answer off the screen is not: it arrives bit by bit, between frames and status lines. Claude Code offers something better. A hook is a command that Claude Code runs by itself at certain moments, and it hands that command a description of what just happened. Two moments matter here:

  • SessionStart: Claude Code has started, or has cleared its conversation, and is ready for input.

  • Stop: it has finished an answer. The description contains the answer's text.

The second file sets them up, brain/settings.json:

brain/settings.json
{
  "permissions": {
    "allow": ["WebSearch", "WebFetch"]
  },
  "hooks": {
    "SessionStart": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "curl -s -m 5 -X POST --data-binary @- http://127.0.0.1:3211/ || true",
            "timeout": 10
          }
        ]
      }
    ],
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "curl -s -m 5 -X POST --data-binary @- http://127.0.0.1:3211/ || true",
            "timeout": 10
          }
        ]
      }
    ]
  }
}

Both hooks run the same command: curl takes the description, which Claude Code hands it as JSON, and posts it to an address on your own computer, where Pixel will be listening. The || true makes sure that Claude Code is not bothered when Pixel is not running. permissions allows the two tools the brain may use, searching the web and reading a page, without asking.

The port for that address goes in src/config.ts:

src/config.ts
/** Where Claude Code's hooks report to. The address is also in brain/settings.json. */
export const BRAIN_PORT = Number(process.env.BRAIN_PORT ?? 3211);

The bridge

Create src/brain.ts:

src/brain.ts
import { execFile } from 'node:child_process';
import { createServer } from 'node:http';
import { join } from 'node:path';
import { setTimeout as sleep } from 'node:timers/promises';
import { promisify } from 'node:util';
import { BRAIN_PORT } from './config.ts';
import type { Reply } from './model.ts';
import { spoken } from './prompt.ts';

const FOLDER = join(import.meta.dirname, '..', 'brain');
const SESSION = 'brain';
/** How long Claude may take for one answer. */
const ANSWER_MS = 120_000;
/** After this long without a question, the next one starts with an empty conversation. */
const FORGET_MS = 20 * 60_000;

const run = promisify(execFile);
/** tmux with a server of its own (-L), apart from any tmux you use yourself. */
const tmux = (...args: string[]) => run('tmux', ['-L', 'pixel', ...args]);

/** Who is waiting for Claude Code to start (or to clear its conversation), and for its answer. */
let started: (() => void) | undefined;
let answered: ((text: string) => void) | undefined;

/**
 * Claude Code's hooks post here (see brain/settings.json): SessionStart when it is ready
 * for input, Stop when it has finished an answer, with the answer's text.
 */
const hooks = createServer(async (request, response) => {
  let body = '';
  for await (const chunk of request) body += chunk;
  response.end();
  try {
    const event = JSON.parse(body) as { hook_event_name?: string; last_assistant_message?: string };
    if (event.hook_event_name === 'SessionStart') started?.();
    if (event.hook_event_name === 'Stop') answered?.(event.last_assistant_message ?? '');
  } catch {
    // not from a hook
  }
});
hooks.listen(BRAIN_PORT, '127.0.0.1');
hooks.unref(); // does not keep the program alive on its own

/** Waits until `arm` is handed a value, but no longer than `ms`. Null means: it took too long. */
function waitFor<T>(arm: (done: (value: T) => void) => void, ms: number): Promise<T | null> {
  return new Promise((resolve) => {
    const timer = setTimeout(() => resolve(null), ms);
    arm((value) => {
      clearTimeout(timer);
      resolve(value);
    });
  });
}

async function running(): Promise<boolean> {
  try {
    await tmux('has-session', '-t', SESSION);
    return true;
  } catch {
    return false;
  }
}

/** Starts Claude Code in a tmux session nobody needs to look at. */
async function start(): Promise<boolean> {
  const ready = waitFor<void>((done) => (started = done), 30_000);
  await tmux(
    'new-session', '-d', '-s', SESSION, '-x', '120', '-y', '36', '-c', FOLDER,
    'claude',
    '--restricted', '--tools', 'WebSearch,WebFetch', // it may search and read the web, nothing else
    '--settings', 'settings.json', // the hooks, and the permission to use those two tools
    '--permission-mode', 'dontAsk', // never stop to ask: what is not allowed is refused
    '--append-system-prompt-file', 'PERSONA.md',
    '--model', 'sonnet',
  );
  await sleep(500);
  if (!(await running())) return false; // it ended at once: is claude installed?
  return (await ready) !== null;
}

/** Types a line into Claude Code, as if on its keyboard, and presses Enter. */
async function type(line: string): Promise<void> {
  await tmux('send-keys', '-t', SESSION, '-l', '--', line);
  await sleep(400); // the input box needs a moment before Enter means "send"
  await tmux('send-keys', '-t', SESSION, 'Enter');
}

let lastAsked = 0;

async function ask(text: string): Promise<Reply | null> {
  if (!(await running()) && !(await start())) return null;

  if (lastAsked && Date.now() - lastAsked > FORGET_MS) {
    const cleared = waitFor<void>((done) => (started = done), 10_000);
    await type('/clear');
    await cleared;
  }
  lastAsked = Date.now();

  const answer = waitFor<string>((done) => (answered = done), ANSWER_MS);
  // one line, and not a long one: the input box would take more for pasted text
  await type(`[Pixel] ${text.replace(/\s+/g, ' ').slice(0, 400)}`);
  const speech = await answer;
  if (speech === null) {
    await tmux('send-keys', '-t', SESSION, 'Escape'); // took too long: interrupt it
    return null;
  }
  return speech ? { speech: spoken(speech, 3) } : null;
}

let line: Promise<unknown> = Promise.resolve();

/** Hands the request to Claude Code. One at a time: there is only one keyboard. */
export function askBrain(text: string): Promise<Reply | null> {
  const turn = line.then(() => ask(text));
  line = turn.catch(() => {});
  return turn;
}

It is the longest file of the series, and it reads from top to bottom like the story of one question.

The hooks' address. A tiny web server, like the one of part 9, receives what the hooks post. It looks at the name of the event and wakes whoever is waiting: started for a session that is ready, answered with the text of an answer.

Waiting, but not forever. waitFor wraps this in a promise with a time limit. Everything here that waits for Claude Code waits through it, because a program you drive by keystrokes can always be somewhere you did not expect.

Starting. start opens a tmux session called brain and runs Claude Code in it. The -L pixel gives Pixel a tmux server of its own, so it never touches terminals you use yourself. The options are worth reading one by one:

  • --restricted --tools WebSearch,WebFetch: Claude Code can normally run commands and change files. This brain can search the web and read pages, and that is all.

  • --settings settings.json: the file with the hooks.

  • --permission-mode dontAsk: never stop and wait for a yes. What is not allowed is refused. Nobody is sitting at this terminal to answer.

  • --append-system-prompt-file PERSONA.md: the persona.

  • --model sonnet: a fast, capable model. Spoken answers do not need the largest one.

Typing. type sends a line to the session, letter for letter, waits a moment and presses Enter. The -l tells tmux to take the text literally, whatever is in it.

Asking. ask makes sure a session is running, arms the wait for the answer, and types the question as one line that starts with [Pixel]. The tag is not decoration: a line that starts with / or ! would be a command to Claude Code itself, and with the tag in front, nothing a user says can ever start that way. If the answer does not come within two minutes, Pixel presses Escape, which interrupts Claude Code, and gives up.

Forgetting. After twenty minutes without a question, the next one is preceded by /clear, Claude Code's command for an empty conversation. A question about dinner should not be answered in the light of this morning's football results.

One at a time. There is one keyboard, so askBrain queues the questions with the promise trick from part 11.

Who goes to the brain?

The third tier goes into the list in src/assistant.ts:

src/assistant.ts
import { askBrain } from './brain.ts';

// ...

/** From the fastest and simplest to the slowest and smartest. */
const tiers: Tier[] = [
  { name: 'rules', ask: rules },
  { name: 'model', ask: askModel },
  { name: 'brain', ask: askBrain },
];

That alone already works: when the model gives up, part 7 turns that into "not for me", and the router moves on to the brain. But it wastes a second or two on a request the small model was never going to answer, and worse, the small model does not always give up when it should. So some requests skip it. In src/language.ts:

src/language.ts
/** Requests the small model should not even try: they need the web or a longer answer. */
export const FOR_BRAIN =
  /\b(search|look up|google|news|latest|price|prices|cost|costs|recipe|who won|score|explain|why|how do i|how does|how to|recommend|compare|summari[sz]e)\b/i;

And at the start of askModel in src/model.ts:

src/model.ts
import { CLAIM, FOR_BRAIN, NUDGE, PERSONA, REFUSAL } from './language.ts';

// ...

export async function askModel(text: string): Promise<Reply | null> {
  if (FOR_BRAIN.test(text)) return null;
  if (!(await available())) return null;

The words stand for the kinds of request that need the web or a longer answer: searching, news, prices, explanations, recommendations. The real assistant goes further and turns the question around: it sends everything to the big model except what clearly belongs to the home. Which way is right for you depends on how much you trust your small model. Start like this, and move the line when you see it answer what it should not.

Try it

~/projects/pixel-assistant
$ npm run chatTalk to Pixel. /bye ends the chat.you> Kitchen light onpixel> The light in the kitchen is on.  (rules, 1 ms)you> Make it cosy in the living room.pixel> The light in the living room is warm white at 30 percent.  (model, 1580 ms)you> Why is the sky blue?pixel> Sunlight is a mix of all colors, and the air scatters the blue part much more strongly than the red part. So blue light reaches your eyes from every direction of the sky.  (brain, 9400 ms)you> /bye

Three sentences, three tiers: a rule in a millisecond, the local model in a second or two, and the brain in a few more, because it thinks longer and may search the web. The first question to the brain takes longest, since Claude Code has to start. In the page, with the voice on, it is the same: you ask your kitchen why the sky is blue, and it tells you.

You can watch the brain work:

~/projects/pixel-assistant
$ tmux -L pixel attach -t brain# Ctrl+B, then D, lets go of it again without stopping it$ tmux -L pixel kill-server# ends the session for good; Pixel starts a new one when it needs it

What you have built

Look back at what stands on your machine now:

  • a language model that runs locally, behind a small client you wrote yourself;

  • a persona, and a prompt arranged so that the model's work is reused;

  • tools for lamps, timers, the weather and a memory, with every argument checked;

  • three tiers, each of which knows how to say "not for me";

  • checks that catch the model's typical mistakes;

  • a memory that finds facts by their meaning;

  • a page with a microphone, a voice and a wake word.

None of it is a framework, and all of it is yours to change. Some ideas for where to go from here:

  • Real lamps. setLamps in home.ts is the only place that would need to talk to them. Many smart lamps can be reached over the home network.

  • More tools. A shopping list, the departures at your bus stop, your calendar. Each is a description and a function.

  • More rules, for the commands you find yourself saying every day.

  • A collection of test sentences that you run after every change. It is how the real assistant went from 66 to 98 percent.

  • A small computer of its own, with a microphone and a loudspeaker, so Pixel is there when your laptop is not.

The most important thing you take with you is not a piece of code, though. It is the way of thinking about a language model as one part of a program: a part that understands language remarkably well and should be trusted with exactly that. Do with code what code can do, check what the model hands you, and let every part say when a request is not for it. That holds for a model of any size.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.