Skip to content

Create an AI Assistant With Local Models · Part 10 of 13 in the series

Part 10: Voice In: Speech to Text

Give the assistant ears. Record your voice in the browser while you hold a button, send it to your own machine and let Whisper, a speech recognizer that runs locally, turn it into text.

Quest objectives

  • Understand how sound becomes numbers
  • Run the Whisper speech recognizer on your own machine
  • Record from the microphone in the browser
  • Send a recording to the server and get text back
  • Clean up what a speech recognizer invents

Typing to an assistant is fine at a desk. In a kitchen, with dough on your hands, you want to talk. Over this part and the next two, Pixel learns to listen, to speak and to wait for its name. The page in the browser is the microphone and the loudspeaker; the listening and speaking themselves happen on your machine, with models that run there, just like the language model.

This part is about speech to text: sound goes in, a sentence comes out, and that sentence takes the same way as one you typed.

Sound as numbers

A microphone measures the air pressure many thousand times a second. Each measurement is a number, a sample, and a recording is nothing but a long list of them. Two things describe such a list:

  • the sample rate: how many samples there are per second. Music uses 44,100 or 48,000. Speech recognizers want 16,000, which is plenty for a voice.

  • the format of a sample: either a whole number from -32,768 to 32,767 (16 bits, compact, good for sending) or a decimal number from -1 to 1 (what audio software computes with).

One second of speech is therefore 16,000 numbers, or 32,000 bytes. That is all the "audio" you will send around.

Whisper on your machine

Whisper is a speech recognition model that OpenAI published openly. It understands about a hundred languages and copes with accents and background noise. To run it, you use sherpa-onnx, a free library for speech models that works on Windows, macOS and Linux and needs no graphics card. Install it:

~/projects/pixel-assistant
$ npm install sherpa-onnx-node@1.13.8

Then download the model, about 640 MB, into a folder models:

~/projects/pixel-assistant
$ mkdir models$ cd models$ curl -LO https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-small.tar.bz2$ tar xjf sherpa-onnx-whisper-small.tar.bz2$ rm sherpa-onnx-whisper-small.tar.bz2$ cd ..

This is Whisper "small". It needs about 1 GB of memory, so it fits next to the language model on a machine with 8 GB. There are larger Whispers that hear more accurately, and if you have memory to spare, sherpa-onnx-whisper-turbo from the same place is a good next step: on the real assistant it made clearly fewer mistakes, for about twice the memory.

The library is a package without types, so describe the parts Pixel uses once, in src/voice/sherpa.ts:

src/voice/sherpa.ts
/**
 * sherpa-onnx-node is a CommonJS package with native code and without types, so it is loaded
 * with require, and the few parts Pixel uses are described here.
 */
import { createRequire } from 'node:module';

interface Recognizer {
  createStream(): { acceptWaveform(audio: { samples: Float32Array; sampleRate: number }): void };
  decodeAsync(stream: object): Promise<{ text: string }>;
}

interface Tts {
  generateAsync(request: { text: string; sid: number; speed: number }): Promise<{ samples: Float32Array; sampleRate: number }>;
}

export interface Vad {
  acceptWaveform(samples: Float32Array): void;
  isEmpty(): boolean;
  front(enableExternalBuffer: boolean): { start: number; samples: Float32Array };
  pop(): void;
}

interface Sherpa {
  OfflineRecognizer: { createAsync(config: object): Promise<Recognizer> };
  OfflineTts: { createAsync(config: object): Promise<Tts> };
  Vad: new (config: object, bufferSizeInSeconds: number) => Vad;
}

export const sherpa = createRequire(import.meta.url)('sherpa-onnx-node') as Sherpa;

/** Where the downloaded models are. */
export const MODELS = 'models';

The interfaces for speaking (Tts) and for finding speech (Vad) are for the next two parts. Now the recognizer, src/voice/stt.ts ("speech to text"):

src/voice/stt.ts
import { LOCALE } from '../language.ts';
import { MODELS, sherpa } from './sherpa.ts';
import { clean } from './text.ts';

const folder = `${MODELS}/sherpa-onnx-whisper-small`;

/** Whisper, loaded once when the server starts. */
const recognizer = await sherpa.OfflineRecognizer.createAsync({
  featConfig: { sampleRate: 16000, featureDim: 80 },
  modelConfig: {
    whisper: {
      encoder: `${folder}/small-encoder.int8.onnx`,
      decoder: `${folder}/small-decoder.int8.onnx`,
      // without a language Whisper guesses, and short commands come out in the wrong one
      language: LOCALE.slice(0, 2),
      task: 'transcribe',
      tailPaddings: 1000,
    },
    tokens: `${folder}/small-tokens.txt`,
    numThreads: 4,
    provider: 'cpu',
    debug: 0,
  },
  decodingMethod: 'greedy_search',
});

/** One recording at a time: the recognizer already uses several processor cores. */
let queue: Promise<unknown> = Promise.resolve();

/** Turns a recording (16 000 samples a second, one channel) into text. */
export function transcribe(samples: Float32Array): Promise<string> {
  const result = queue.then(async () => {
    // under a third of a second there is no word in it
    if (samples.length < 16000 * 0.3) return '';
    const stream = recognizer.createStream();
    stream.acceptWaveform({ samples, sampleRate: 16000 });
    const { text } = await recognizer.decodeAsync(stream);
    return clean(text);
  });
  queue = result.catch(() => {});
  return result;
}

Loading the model takes a second or two and happens once, when the server starts. transcribe takes samples and returns text. Three details in it were learned the hard way on the real assistant:

  • The language is fixed. A recognizer that guesses the language has little to go on in a command of three words. An earlier recognizer on the real assistant turned short German commands into English nonsense. Telling Whisper the language removes that whole class of errors.

  • One recording at a time. The recognizer already uses four processor cores for one recording. Two at once would only get in each other's way, so they queue.

  • Nothing under a third of a second. There is no word in it, and what Whisper makes of non-words is the next topic.

What nobody said

Whisper learned from a huge number of videos with subtitles. Give it silence or noise, and it writes what it saw most often at the quiet end of a video: "Thank you for watching." This is not a bug you can fix. It is what the model is, and every program that uses Whisper has to clean up after it. Create src/voice/text.ts:

src/voice/text.ts
import { NOT_SAID } from '../language.ts';

/**
 * Tidies what Whisper wrote. On silence and noise it likes to invent sentences it saw at the
 * end of many videos, and now and then it repeats itself.
 */
export function clean(text: string): string {
  const sentences = text
    .replace(/\[[^\]]*\]|\([^)]*\)|[♪♫]+/g, ' ') // [Music], (applause), ♪
    .split(/(?<=[.!?])\s+/)
    .map((sentence) => sentence.trim())
    .filter((sentence) => sentence && !NOT_SAID.test(sentence.toLowerCase().replace(/[.!?,]/g, '').trim()));
  // "What time is it? What time is it?" is said once
  const once = sentences.filter((sentence, i) => sentence !== sentences[i - 1]);
  return once.join(' ');
}

And the list of sentences nobody said, in src/language.ts:

src/language.ts
/** Sentences nobody said: Whisper invents them when it hears silence or noise. */
export const NOT_SAID =
  /^(thank you|thanks|thanks for watching|thank you for watching|thank you very much|please subscribe|subtitles by .*|you|bye|okay|so)$/;

The list will grow. Whenever Pixel reacts to something you did not say, look at what the recognizer wrote, and add it.

The server takes recordings

In src/server.ts, add the import and one more address, above the one for /api/events:

src/server.ts
import { transcribe } from './voice/stt.ts';

// ...

    if (request.method === 'POST' && pathname === '/api/hear') {
      // the recording arrives as 16-bit samples; the recognizer wants numbers from -1 to 1
      const bytes = await body(request);
      const samples = new Int16Array(bytes.buffer, bytes.byteOffset, Math.floor(bytes.byteLength / 2));
      return json(response, { text: await transcribe(Float32Array.from(samples, (sample) => sample / 32768)) });
    }

The page will send the recording as raw 16-bit samples, nothing else: no file format, no header. The server reads the bytes as numbers, turns them into decimal numbers from -1 to 1 for the recognizer, and answers with the text.

The microphone in the browser

Three small files do the recording. The first runs on the browser's audio thread and makes the sound small enough to send. Create public/capture.js:

public/capture.js
/**
 * Runs on the browser's audio thread. The microphone delivers 44 100 or 48 000 samples a
 * second as numbers from -1 to 1; speech recognition wants 16 000 a second. So this takes
 * the average of every three or so samples and posts them in packets of 20 milliseconds,
 * as 16-bit numbers.
 */
const RATE = 16000;
const PACKET = 320;

class Capture extends AudioWorkletProcessor {
  step = sampleRate / RATE; // `sampleRate` is the rate of the audio context
  position = 0;
  sum = 0;
  count = 0;
  packet = new Int16Array(PACKET);
  filled = 0;

  process(inputs) {
    const channel = inputs[0]?.[0];
    if (!channel) return true;
    for (const sample of channel) {
      this.sum += sample;
      this.count += 1;
      this.position += 1;
      if (this.position < this.step) continue;
      this.position -= this.step;
      const average = this.sum / this.count;
      this.sum = 0;
      this.count = 0;
      this.packet[this.filled] = Math.max(-32768, Math.min(32767, Math.round(average * 32767)));
      this.filled += 1;
      if (this.filled === PACKET) {
        this.port.postMessage(this.packet.buffer, [this.packet.buffer]);
        this.packet = new Int16Array(PACKET);
        this.filled = 0;
      }
    }
    return true;
  }
}

registerProcessor('capture', Capture);

The browser hands this code the microphone's sound in small portions, at its own rate of 44,100 or 48,000 samples a second. The loop averages every three or so samples into one, which brings the rate down to 16,000, and collects the results in packets of 320: 20 milliseconds each. A full packet is posted to the page.

The second file opens the microphone, public/mic.js:

public/mic.js
/** Everyone who wants the microphone's sound, packet by packet. */
const listeners = new Set();
let opening;

async function open() {
  if (!navigator.mediaDevices) throw new Error('The microphone needs https or http://localhost.');
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true, channelCount: 1 },
  });
  const context = new AudioContext();
  await context.audioWorklet.addModule('capture.js');
  const capture = new AudioWorkletNode(context, 'capture');
  capture.port.onmessage = (event) => {
    for (const listener of listeners) listener(event.data);
  };
  // the browser only runs what leads to the speakers, so lead there, silently
  const silence = context.createGain();
  silence.gain.value = 0;
  context.createMediaStreamSource(stream).connect(capture).connect(silence).connect(context.destination);
}

/**
 * Calls `listener` with every packet of sound: 20 milliseconds as an ArrayBuffer of 16-bit
 * samples. The first call asks for the microphone. Returns a function that stops it.
 */
export async function onSound(listener) {
  // if opening fails, the next call may try again
  opening ??= open().catch((error) => {
    opening = undefined;
    throw error;
  });
  await opening;
  listeners.add(listener);
  return () => listeners.delete(listener);
}

getUserMedia asks for the microphone, and the browser asks you. The three options switch on what phone calls use as well: echo cancellation, noise suppression and automatic volume. The microphone is opened once, on first use, and then stays open; whoever wants its sound adds a listener.

The third file is the button, public/talk.js:

public/talk.js
import { onSound } from './mic.js';

/** Records while the button is held, has the server turn it into text and hands that on. */
export function setupTalk(button, onText) {
  let packets = [];
  let held = false;
  let stop;

  button.addEventListener('pointerdown', async () => {
    held = true;
    packets = [];
    button.classList.add('active');
    try {
      const off = await onSound((packet) => packets.push(packet));
      // the very first press waits for the permission; the button may be up again by then
      if (held) stop = off;
      else off();
    } catch (error) {
      // no microphone, or no permission to use it
      button.classList.remove('active');
      alert(error.message);
    }
  });

  button.addEventListener('pointerup', async () => {
    held = false;
    button.classList.remove('active');
    stop?.();
    stop = undefined;
    if (packets.length === 0) return;
    const response = await fetch('/api/hear', { method: 'POST', body: new Blob(packets) });
    const { text } = await response.json();
    if (text) onText(text);
  });
}

While the button is down, the packets are collected. When it comes up, they are sent to /api/hear as one recording, and the text that comes back is handed on. Add the button to the form in public/index.html:

public/index.html
<form id="form">
        <input id="text" placeholder="Say something to Pixel" autocomplete="off" autofocus />
        <button>Send</button>
        <button type="button" id="talk">Hold to talk</button>
      </form>

And connect it in public/app.js:

public/app.js
import { setupTalk } from './talk.js';

// ... and at the end of the file:

setupTalk(document.querySelector('#talk'), send);

The button hands the text to send, the same function the text field uses. From there on, a spoken sentence and a typed one are the same thing.

Try it

~/projects/pixel-assistant
$ npm run servePixel is at http://localhost:3210

Reload the page, hold the button, say "kitchen light on", let go. The very first press makes the browser ask for the microphone, so press again after you have allowed it. Your sentence appears in the conversation as if you had typed it, and the lamp goes on.

Pixel hears you. In the next part it answers out loud.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.