Skip to content

Create an AI Assistant With Local Models · Part 11 of 13 in the series

Part 11: Voice Out: Text to Speech

Give the assistant a voice. Run the Piper speech synthesizer on your own machine, pack its sound into a WAV file by hand, and play the answer in the browser sentence by sentence, so the first words come quickly.

Quest objectives

  • Run a Piper voice on your own machine
  • Understand what is inside a WAV file
  • Play sound in the browser
  • Speak an answer sentence by sentence
  • Know why the answers were kept short all along

Pixel hears you, and answers in writing. In this part it gets a voice. The way is the one from the last part, backwards: text goes to your machine, a model turns it into samples, and the page plays them.

A voice

Piper is a free, fast speech synthesizer with voices for dozens of languages. It runs with the library you installed in the last part. Download a voice, about 65 MB:

~/projects/pixel-assistant
$ cd models$ curl -LO https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-lessac-medium.tar.bz2$ tar xjf vits-piper-en_US-lessac-medium.tar.bz2$ rm vits-piper-en_US-lessac-medium.tar.bz2$ cd ..

A voice is a folder: the model, a list of the sounds it knows, and data for turning written words into those sounds. Name it in src/language.ts, because a voice speaks one language:

src/language.ts
/** The Piper voice, a folder in models/. */
export const VOICE = 'vits-piper-en_US-lessac-medium';

You find more voices on the release page; everything that starts with vits-piper- works here. Now the synthesizer, src/voice/tts.ts ("text to speech"):

src/voice/tts.ts
import { readdirSync } from 'node:fs';
import { VOICE } from '../language.ts';
import { MODELS, sherpa } from './sherpa.ts';

const folder = `${MODELS}/${VOICE}`;

/** The voice, loaded once when the server starts. */
const voice = await sherpa.OfflineTts.createAsync({
  model: {
    vits: {
      model: `${folder}/${readdirSync(folder).find((file) => file.endsWith('.onnx'))}`,
      tokens: `${folder}/tokens.txt`,
      dataDir: `${folder}/espeak-ng-data`,
    },
    numThreads: 4,
    provider: 'cpu',
    debug: 0,
  },
  maxNumSentences: 1,
});

let queue: Promise<unknown> = Promise.resolve();

/** Speaks the text and returns the sound as a WAV file. */
export function speak(text: string): Promise<Buffer> {
  const result = queue.then(async () => {
    const audio = await voice.generateAsync({ text, sid: 0, speed: 1 });
    return wav(audio.samples, audio.sampleRate);
  });
  queue = result.catch(() => {});
  return result;
}

/** Packs samples into a WAV file: a header of 44 bytes, then every sample as a 16-bit number. */
function wav(samples: Float32Array, sampleRate: number): Buffer {
  const file = Buffer.alloc(44 + samples.length * 2);
  file.write('RIFF', 0);
  file.writeUInt32LE(36 + samples.length * 2, 4);
  file.write('WAVEfmt ', 8);
  file.writeUInt32LE(16, 16); // length of the format part
  file.writeUInt16LE(1, 20); // 1 = plain samples (PCM)
  file.writeUInt16LE(1, 22); // one channel
  file.writeUInt32LE(sampleRate, 24);
  file.writeUInt32LE(sampleRate * 2, 28); // bytes a second
  file.writeUInt16LE(2, 32); // bytes per sample
  file.writeUInt16LE(16, 34); // bits per sample
  file.write('data', 36);
  file.writeUInt32LE(samples.length * 2, 40);
  samples.forEach((sample, i) => {
    file.writeInt16LE(Math.round(Math.max(-1, Math.min(1, sample)) * 32767), 44 + i * 2);
  });
  return file;
}

speak hands a text to the voice and gets samples back, as decimal numbers from -1 to 1, together with the voice's sample rate. As with listening, one text is spoken at a time.

A WAV file by hand

A browser cannot play bare samples: it wants to know the sample rate and the format. The simplest file that says so is a WAV, and wav writes one by hand. It is a good moment to see that such a file format is no mystery: 44 bytes that say "this is sound, one channel, so many samples a second, 16 bits each", followed by the samples. Every line of the function writes one of those facts to its fixed place.

The server gets an address for it. In src/server.ts:

src/server.ts
import { speak } from './voice/tts.ts';

// ...

    if (request.method === 'POST' && pathname === '/api/speak') {
      response.writeHead(200, { 'content-type': 'audio/wav' });
      return response.end(await speak((await body(request)).toString()));
    }

Playing it

Create public/speak.js:

public/speak.js
let context;
/** The end of what is being said now, so the next sentence waits its turn. */
let line = Promise.resolve();

function play(sound) {
  return new Promise((done) => {
    const source = context.createBufferSource();
    source.buffer = sound;
    source.connect(context.destination);
    source.onended = done;
    source.start();
  });
}

/** Speaks the text, sentence by sentence. The promise ends when the last one has been played. */
export function speak(text) {
  // a browser lets a page make sound only after a click or a key press, which there has been by now
  context ??= new AudioContext();
  context.resume();
  const sentences = text.match(/[^.!?]+[.!?]+(?=\s|$)|[^.!?]+$/g) ?? [];
  // ask for every sentence at once, play them in order: the first one starts while the rest is still being made
  const sounds = sentences.map(async (sentence) => {
    const response = await fetch('/api/speak', { method: 'POST', body: sentence.trim() });
    return context.decodeAudioData(await response.arrayBuffer());
  });
  for (const sound of sounds) {
    line = line.then(async () => play(await sound)).catch((error) => console.error(error));
  }
  return line;
}

The function does not send the whole answer to be spoken at once. It cuts it into sentences and asks for all of them at the same moment. The server makes them one after the other, and the page plays each as soon as it is there and its turn has come. The effect: you hear the first sentence while the second is still being made. On the real assistant, the first sentence of an answer is ready after about 0.4 seconds, however long the answer is.

line is a small trick that you have now seen three times in this series: a promise that stands for "everything that was started before". Whatever comes next waits for it and then takes its place. That is all a queue needs.

Add a switch to public/index.html, below the form:

public/index.html
<p class="switches">
        <label><input type="checkbox" id="voice" /> Speak the answers</label>
      </p>

And in public/app.js, let answered speak: add the import and the constant at the top, and replace the function.

public/app.js
import { speak } from './speak.js';

// ...

const voice = document.querySelector('#voice');

// ...

/** Shows Pixel's answer and speaks it, if that is switched on. Ends when it has been said. */
async function answered(answer) {
  show('pixel', answer.speech, answer.tier ? `${answer.tier}, ${answer.ms} ms` : '');
  if (voice.checked) await speak(answer.speech);
}

Restart the server, reload the page, tick "Speak the answers" and say or type something. Pixel answers out loud. Set a timer: when it rings, Pixel says so by itself, because announcements go through answered as well.

You can now hold a button, ask for the weather and hear the answer. Only the button is still in the way. In the next part you take your hand off it: Pixel listens all the time and reacts when it hears its name.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.