Create an AI Assistant With Local Models · Part 1 of 13 in the series
Part 1: A Model on Your Own Machine
What a language model is, what it takes to run one on your own computer, and your first conversation with it in the terminal, using Ollama.
Create an AI Assistant With Local ModelsPart 1 of 13
On this page
Quest objectives
- Know what you build in this series
- Understand what a language model is and what its size means
- Install Ollama and download a model
- Talk to the model in the terminal
- Know why a small model needs help, and where it will come from
In this series you build an assistant called Pixel. You tell it "it's a bit dark in the kitchen", and the kitchen lamp goes on. It sets a timer for your tea, tells you whether you need an umbrella, and remembers that Anna is allergic to nuts. In the later parts it gets a page in your browser, a voice and ears. The language model behind it runs on your own computer: nothing you say leaves the house, it costs nothing per question, and it works without the internet.
The series follows a real assistant that runs in a living room on a small PC, day in, day out. Its lamps are real. Yours will be a small virtual home, so you need no hardware, but the logic is the same, and so are the lessons it took to get there.
What a language model is
A language model is a very large table of numbers, called weights, plus a small program that uses them for one thing: given a text, work out which piece of text probably comes next. Add that piece, ask again, and again. That is all a chatbot does when it "writes".
The pieces are tokens: short words or parts of words. An English word is about one and a third tokens.
A model's size is counted in parameters, the numbers in that table. The model you will use has 4 billion of them, written "4b". The large models behind the well-known chat services have hundreds of billions and need a data center.
To make the file smaller, the numbers are stored coarsely, with about 4 bits each. This is called quantization, and it is why 4 billion parameters fit into a download of 2.5 GB with little loss.
The context window is how much text the model can look at in one go: your question, everything said before, and its own answer. What does not fit is simply not seen.
A model learns nothing while you use it and remembers nothing between two requests. It is a file that is read, never written. Everything that looks like memory or personality is text that a program puts in front of your question. Writing that program is what this series is about.
What a small model can and cannot do
A 4b model understands everyday language well, in many languages, and is good at turning "it's a bit dark in the kitchen" into "switch the kitchen lamp on". It is fast enough to talk to. It is not an encyclopedia: ask it for facts and it will often answer with something that only sounds right. It cannot count weekdays reliably, it is bad at arithmetic, and it knows nothing that happened after it was made.
So Pixel will not rely on the model alone. It gets three levels:
Rules answer the simple, clear commands in under a millisecond, without any model.
The local model handles what is said in free words, and uses tools to act.
A bigger brain gets the questions the small model should not answer. That last step is optional and the only one that leaves your machine.
The series
A model on your own machine (this part)
Talking to the model from code
A persona and a prompt that stays fast
Tools, or how the model switches the lights
Timers, the weather and tools that answer themselves
Fast rules before the model
Checking the model's answers
A memory that finds things by meaning
A chat page in the browser
Voice in: speech to text
Voice out: text to speech
A wake word
Calling in a bigger brain: Claude Code in tmux
What you need
A computer with Windows, macOS or Linux and at least 8 GB of memory; 16 GB is more comfortable. A graphics card makes the model faster, and Ollama uses one if it finds it (NVIDIA, AMD or the chip of an Apple Silicon Mac), but it is not needed: on the processor alone the answers just take a little longer. You also need about 5 GB of free disk space, and from the next part on Node.js.
You do not need to know anything about machine learning. If you have written a little JavaScript before, you can follow all of the code. If you have not, you can still read along and understand how such an assistant works.
Install Ollama
Ollama is a free program that downloads language models and runs them. It hides the hard parts: finding your graphics card, loading the model into memory and unloading it again. And it gives other programs an address on your computer where they can send text and get the model's answer. On Linux, one line installs it:
$ curl -fsSL https://ollama.com/install.sh | sh# downloads Ollama, installs it and starts it in the background$ ollama --versionollama version is 0.40.16
On macOS and Windows, download the installer from ollama.com/download and run it. Afterwards Ollama runs in the background, and the ollama command works in a new terminal window. Your version number will be a newer one.
Ollama now listens on port 11434 of your computer, and only there: nobody else on your network can reach it. Check that it is up:
$ curl http://localhost:11434/Ollama is running
Download a model
Ollama has a library of models. This series uses Qwen3 with 4 billion parameters, in its "instruct" variant:
$ ollama pull qwen3:4b-instructpulling manifest# ... a progress bar for every file, 2.5 GB in all ...verifying sha256 digestwriting manifestsuccess
Talk to it
ollama run loads the model and opens a chat right in the terminal. The first start takes a few seconds, because 2.5 GB have to be read into memory:
$ ollama run qwen3:4b-instruct>>> Why is the sky blue? One sentence, please.Because the air scatters the blue part of sunlight much more than the red part, so blue reaches your eyes from every direction.>>> And at sunset?Then the light travels through far more air, most of the blue is scattered away on the way, and what is left is orange and red.>>> /bye
That answer came from your own machine. Watch how it appears word by word: you are seeing the model produce one token after the other. Try a few things: ask it something in another language, ask it what day it is today (it cannot know), ask it for 17 times 23 (the right answer is 391; do not be surprised by a wrong one). /bye ends the chat.
Where the model lives
After the chat, the model stays in memory for five minutes in case you come back, then Ollama unloads it and your memory is free again. Three commands show and control that:
$ ollama list# the models on your disk$ ollama ps# the models in memory right now, and whether they run on the GPU or the CPU$ ollama stop qwen3:4b-instruct# takes a model out of memory
If ollama ps shows 100% GPU in the PROCESSOR column, the model runs on your graphics card; 100% CPU means it runs on the processor. Both work.
The chat in the terminal is Ollama's own little program. In the next part you write yours: a few lines of TypeScript that send a question to the model and print its answer.
Comments
The comments are GitHub Discussions: sign in with GitHub to write one.
The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.