Skip to content

Create an AI Assistant With Local Models · Part 12 of 13 in the series

Part 12: A Wake Word

Take your hand off the button. The page sends the microphone to your machine all the time, a small model finds the moments in which someone speaks, and the assistant answers the sentences that start with its name.

Quest objectives

  • Send sound to the server continuously over a WebSocket
  • Find speech in a stream of sound with voice activity detection
  • Recognize the assistant's name at the start of a sentence
  • Allow a follow-up question without the name
  • Keep the assistant from listening to itself

"Pixel, switch on the kitchen light." No button, no screen. To get there, the microphone has to stay open, and that raises two questions. How does Pixel know when somebody is speaking, without running the speech recognizer on hours of silence? And how does it know that a sentence is meant for it and not for the cat?

The plan

  1. The page sends the microphone's packets to your machine all the time.

  2. A tiny model watches the stream and cuts out the stretches of speech. This is called voice activity detection, VAD for short.

  3. Each stretch goes to Whisper, as in part 10.

  4. If the text starts with "Pixel", the rest is a request. If not, it is thrown away.

Commercial assistants use a special small model that listens for the one wake word and nothing else. The real assistant behind this series tried that too, and then found that reading the name in the transcript worked better: in its tests the special model caught 15 to 24 of 30 calls, the transcript 25 of 30, and neither fired on any of 45 sentences that only sounded similar. And it needs no extra model and no training for your assistant's name.

What you need

A package for WebSockets and the VAD model, which is only 2 MB:

~/projects/pixel-assistant
$ npm install ws$ npm install --save-dev @types/ws$ curl -L -o models/silero_vad_v5.onnx https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/silero_vad_v5.onnx

A WebSocket is a connection between a page and a server that stays open and lets both sides send whenever they like. The server-sent events of part 9 only go one way and carry text; a stream of sound needs the real thing.

Finding speech

Create src/voice/vad.ts:

src/voice/vad.ts
import { MODELS, sherpa } from './sherpa.ts';

/** Silero looks at the sound in windows of 512 samples: 32 milliseconds. */
const WINDOW = 512;

/**
 * Finds the stretches of speech in a stream of sound. Feed it whatever arrives; it hands
 * back every utterance that has ended since, as samples.
 */
export function createVad(): { feed(samples: Float32Array): Float32Array[] } {
  const vad = new sherpa.Vad(
    {
      sileroVad: {
        model: `${MODELS}/silero_vad_v5.onnx`,
        threshold: 0.5,
        minSilenceDuration: 0.6, // this much silence ends an utterance
        minSpeechDuration: 0.25,
        maxSpeechDuration: 15,
        windowSize: WINDOW,
      },
      sampleRate: 16000,
      numThreads: 1,
      provider: 'cpu',
      debug: 0,
    },
    60,
  );
  let waiting = new Float32Array(0);

  return {
    feed(samples) {
      // what did not fill a window last time comes first
      const all = new Float32Array(waiting.length + samples.length);
      all.set(waiting);
      all.set(samples, waiting.length);

      const utterances: Float32Array[] = [];
      let at = 0;
      for (; at + WINDOW <= all.length; at += WINDOW) {
        vad.acceptWaveform(all.subarray(at, at + WINDOW));
        while (!vad.isEmpty()) {
          utterances.push(vad.front(false).samples);
          vad.pop();
        }
      }
      waiting = all.slice(at);
      return utterances;
    },
  };
}

Silero is a model that answers one question about 32 milliseconds of sound: is this a voice? It is so small that it uses about a third of a percent of one processor core, so it can run all day. The library around it does the bookkeeping: when it has heard a voice and then 0.6 seconds of silence, it hands out the whole utterance.

The model wants its sound in windows of exactly 512 samples, but the page sends packets of 320. So feed puts what arrives behind what was left over from last time, feeds as many whole windows as there are, and keeps the rest.

Is it for me?

The name, in src/language.ts:

src/language.ts
/**
 * The start of a sentence that is meant for Pixel: an optional greeting, then the name as
 * the speech recognizer writes it. Print what it hears and add the spellings you see.
 */
export const WAKE = /^(?:(?:hey|hi|hello|okay|ok)[,!]? +)?(?:pixel|pixels|pixl|piksel|picsel|pixar)(?![\p{L}\p{N}])[,.!?:]* */iu;

The pattern matches the start of a sentence: an optional greeting, then the name in every spelling the recognizer comes up with. Whisper has never read your assistant's name in this role, and will sometimes write "Pixels" or "Pixar". You find these spellings by watching what it writes, and you add them.

Add a function to src/voice/text.ts that uses the pattern, and extend the import at the top:

src/voice/text.ts
import { NOT_SAID, WAKE } from '../language.ts';

// ...

/**
 * Whether the sentence is addressed to Pixel. Returns what comes after the name ("" when
 * the name is all there is), or null when the sentence does not start with it.
 */
export function afterWake(text: string): string | null {
  const match = WAKE.exec(text);
  return match ? text.slice(match[0].length).trim() : null;
}

The function has three possible results, and the difference matters: the words after the name, an empty text when only the name was said, or null when the sentence was not addressed to Pixel at all.

The open microphone

Create src/voice/listen.ts:

src/voice/listen.ts
import type { Server } from 'node:http';
import { WebSocketServer } from 'ws';
import { handle } from '../assistant.ts';
import { transcribe } from './stt.ts';
import { afterWake } from './text.ts';
import { createVad } from './vad.ts';

/** After an answer, Pixel keeps listening this long without hearing its name again. */
const FOLLOW_UP_MS = 6000;

/**
 * The open microphone: the page sends sound all the time, and Pixel answers the sentences
 * that start with its name.
 */
export function listen(server: Server): void {
  const sockets = new WebSocketServer({ server, path: '/api/listen' });

  sockets.on('connection', (socket) => {
    const vad = createVad();
    let awakeUntil = 0;
    let work: Promise<void> = Promise.resolve();
    const send = (message: object) => socket.send(JSON.stringify(message));

    async function heard(samples: Float32Array, began: number): Promise<void> {
      const text = await transcribe(samples);
      if (!text) return;
      const command = afterWake(text);
      // not addressed to Pixel, and not a follow-up to its last answer: none of its business
      if (command === null && began > awakeUntil) return;
      if (command === '') {
        // only the name: the next sentence counts
        awakeUntil = Date.now() + FOLLOW_UP_MS;
        return send({ wake: true });
      }
      const said = command ?? text;
      send({ heard: said });
      send({ answer: await handle(said) });
    }

    socket.on('message', (data: Buffer, isBinary: boolean) => {
      if (!isBinary) {
        // the page has finished speaking the answer
        if (data.toString() === 'played') awakeUntil = Date.now() + FOLLOW_UP_MS;
        return;
      }
      const samples = new Int16Array(data.buffer, data.byteOffset, Math.floor(data.byteLength / 2));
      for (const utterance of vad.feed(Float32Array.from(samples, (sample) => sample / 32768))) {
        const began = Date.now() - utterance.length / 16; // 16 samples are a millisecond
        // one after the other, in the order they were said
        work = work.then(() => heard(utterance, began)).catch((error) => console.error(error));
      }
    });
  });
}

Every page that connects gets its own VAD and its own state. Sound arrives as binary messages; each one is fed to the VAD, and every utterance that comes out of it goes to heard, strictly one after the other.

heard turns the utterance into text and then decides:

  • Not addressed to Pixel: thrown away, and that is the end of it. Nothing is stored and nothing is logged.

  • Only the name: Pixel wakes up, and whatever is said in the next six seconds counts.

  • Name and request: the request goes to handle, the same function as ever, and the page gets what was heard and the answer.

The six seconds are also for follow-up questions. Having to say the name before every sentence of a conversation is tiresome. When the page reports that it has finished speaking an answer, a window of six seconds opens, in which a sentence counts without the name: "Pixel, what's the weather tomorrow?", answer, "And on Thursday?". What counts is when the sentence began, which is why began is worked out from the length of the utterance.

Start the listener in src/server.ts:

src/server.ts
import { listen } from './voice/listen.ts';

// ...

listen(server);

// 127.0.0.1: only this computer can reach the page
server.listen(PORT, '127.0.0.1', () => console.log(`${NAME} is at http://localhost:${PORT}`));

The page

Create public/listen.js:

public/listen.js
import { onSound } from './mic.js';

/**
 * The open microphone: while the box is checked, the sound goes to the server all the time,
 * and the server says what it heard and what Pixel answers.
 */
export function setupListen(box, show, answered) {
  let socket;
  let stop;
  /** While Pixel speaks, nothing is sent: it should not listen to itself. */
  let speaking = false;

  box.addEventListener('change', async () => {
    if (!box.checked) {
      stop?.();
      socket?.close();
      return;
    }
    try {
      stop = await onSound((packet) => {
        if (!speaking && socket?.readyState === WebSocket.OPEN) socket.send(packet);
      });
    } catch (error) {
      // no microphone, or no permission to use it
      box.checked = false;
      return alert(error.message);
    }
    socket = new WebSocket(`ws://${location.host}/api/listen`);
    socket.onmessage = async (event) => {
      const message = JSON.parse(event.data);
      if (message.wake) show('pixel', '…');
      if (message.heard) show('you', message.heard);
      if (message.answer) {
        speaking = true;
        await answered(message.answer);
        speaking = false;
        socket.send('played');
      }
    };
  });
}

While the box is ticked, every packet from the microphone goes straight into the WebSocket. The messages that come back are shown, and an answer is spoken.

The variable speaking solves a problem you would otherwise notice within a minute: Pixel's voice comes out of the loudspeaker and into the microphone. If it says "Pixel is here", it would hear its own name and answer itself. So while an answer is being played, nothing is sent. When it is over, the page tells the server, which opens the window for a follow-up.

Add the switch in public/index.html:

public/index.html
<p class="switches">
        <label><input type="checkbox" id="voice" /> Speak the answers</label>
        <label><input type="checkbox" id="listen" /> Listen for "Pixel"</label>
      </p>

And connect it in public/app.js:

public/app.js
import { setupListen } from './listen.js';

// ... and at the end of the file:

setupListen(document.querySelector('#listen'), show, answered);

Restart the server, reload the page, tick both boxes and say: "Pixel, switch on the kitchen light." Then, right after the answer: "And the hallway too." The first sentence is answered by a rule and the second by the model, which knows from the history what "too" means.

You can now walk around the room and talk to your assistant. It switches lamps in a blink, understands what you mean, remembers your family and tells you about the weather, all of it on your own machine. What it cannot do is answer a real question. In the last part it learns to ask someone who can.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.