Series
Create an AI Assistant With Local Models
Build your own assistant on a language model that runs on your computer: it switches the lamps of a small home, sets timers, knows the weather, remembers things, listens, speaks and knows when to ask a bigger brain.
Thirteen parts, from installing a language model on your own computer to an assistant you can talk to. You meet the model in the terminal, then talk to it from a few lines of TypeScript, give it a persona and teach it to use tools: it switches the lamps of a small virtual home, sets timers and looks up the weather. Fast rules answer the simple commands before the model has even woken up, checks catch the model when it claims things it has not done, and a memory lets it find what you told it last week. Then it moves into a page in your browser, learns to listen and to speak, waits for its name, and hands the questions that are too big for it to Claude Code. Everything runs on your machine, and no line of it is magic.
Quest line
- A Model on Your Own MachineWhat a language model is, what it takes to run one on your own computer, and your first conversation with it in the terminal, using Ollama.15 min
- Talking to the Model From CodeSend your first request to the local model from TypeScript, learn what a conversation looks like to a model, and build a small chat in the terminal that shows the answer while it is written.20 min
- A Persona and a Prompt That Stays FastTurn the model into Pixel with a system prompt, tell it the date and time with every request, keep the conversation short, and arrange the prompt so that the model can reuse its work.20 min
- Tools, or How the Model Switches the LightsBuild a small virtual home and give the model its first tool. Learn how tool calling really works: the model only asks, your code acts, and the result goes back into the conversation.25 min
- Timers, the Weather and Tools That Answer ThemselvesAdd timers that ring by themselves and a weather tool that asks a free service on the internet, and make the assistant twice as fast by letting tools that act speak for themselves.25 min
- Fast Rules Before the ModelAnswer the simple commands with plain rules in under a millisecond, and build the router that tries the fastest way first and passes everything else on to the model.20 min
- Checking the Model's AnswersA small model claims things it has not done, answers when it should not, and sometimes does not answer at all. Catch all three in code: check every answer before anyone hears it.20 min
- A Memory That Finds Things by MeaningGive the assistant a memory that survives a restart. Learn what embeddings are, store facts with them, find the right ones by meaning instead of by words, and put them in front of the model.25 min
- A Chat Page in the BrowserMove the assistant out of the terminal: a small web server with Node.js alone, a page with the conversation and the lamps of the home, and a channel through which the server tells the page what has changed.25 min
- Voice In: Speech to TextGive the assistant ears. Record your voice in the browser while you hold a button, send it to your own machine and let Whisper, a speech recognizer that runs locally, turn it into text.30 min
- Voice Out: Text to SpeechGive the assistant a voice. Run the Piper speech synthesizer on your own machine, pack its sound into a WAV file by hand, and play the answer in the browser sentence by sentence, so the first words come quickly.20 min
- A Wake WordTake your hand off the button. The page sends the microphone to your machine all the time, a small model finds the moments in which someone speaks, and the assistant answers the sentences that start with its name.25 min
- Calling In a Bigger Brain: Claude Code in tmuxGive the assistant a third tier for the questions a small model should not answer: Claude Code, running in a tmux session that your program types into and that reports its answers back through a hook.30 min