Speech recognition involves receiving speech through a device's microphone, which is then checked by a speech recognition service against a list of grammar (basically, the vocabulary you want to have recognized in a particular app.) When a word or phrase is successfully recognized, it is returned as a result (or list of results) as a text string, and further actions can be initiated as a result.

The Web Speech API has a main controller interface for this —SpeechRecognition — plus a number of closely-related interfaces for representing grammar, results, etc. Generally, the default speech recognition system available on the device will be used for the speech recognition — most modern OSes have a speech recognition system for issuing voice commands. Think about Dictation on macOS, Siri on iOS, Cortana on Windows 10, Android Speech, etc.

To show simple usage of Web speech recognition, we've written a demo calledSpeech color changer. When the screen is tapped/clicked, you can say an HTML color keyword, and the app's background color will change to that color.

To run the demo, navigate to thelive demo URL in a supporting mobile browser (such as Chrome).

The HTML and CSS for the app is really trivial. We have a title, instructions paragraph, and a div into which we output diagnostic messages.

<h1>Speech color changer</h1><p></p><div>  <p><em>…diagnostic messages</em></p></div>

The CSS provides a very simple responsive styling so that it looks OK across devices.

Let's look at the JavaScript in a bit more detail.

Prefixed properties

Browsers currently support speech recognition with prefixed properties.Therefore at the start of our code we include these lines to allow for both prefixed properties and unprefixed versions that may be supported in future:

const SpeechRecognition =  window.SpeechRecognition || window.webkitSpeechRecognition;const SpeechGrammarList =  window.SpeechGrammarList || window.webkitSpeechGrammarList;const SpeechRecognitionEvent =  window.SpeechRecognitionEvent || window.webkitSpeechRecognitionEvent;

The grammar

The next part of our code defines the grammar we want our app to recognize. The following variable is defined to hold our grammar:

const colors = [  "aqua",  "azure",  "beige",  "bisque",  "black",  "blue",  "brown",  "chocolate",  "coral",  // …];const grammar = `#JSGF V1.0; grammar colors; public <color> = ${colors.join(  " | ",)};`;

The grammar format used isJSpeech Grammar Format (JSGF) — you can find a lot more about it at the previous link to its spec. However, for now let's just run through it quickly:

• The lines are separated by semicolons, just like in JavaScript.
• The first line —#JSGF V1.0; — states the format and version used. This always needs to be included first.
• The second line indicates a type of term that we want to recognize.public declares that it is a public rule, the string in angle brackets defines the recognized name for this term (color), and the list of items that follow the equals sign are the alternative values that will be recognized and accepted as appropriate values for the term. Note how each is separated by a pipe character.
• You can have as many terms defined as you want on separate lines following the above structure, and include fairly complex grammar definitions. For this basic demo, we are just keeping things simple.

Plugging the grammar into our speech recognition

The next thing to do is define a speech recognition instance to control the recognition for our application. This is done using theSpeechRecognition() constructor. We also create a new speech grammar list to contain our grammar, using theSpeechGrammarList() constructor.

const recognition = new SpeechRecognition();const speechRecognitionList = new SpeechGrammarList();

We add ourgrammar to the list using theSpeechGrammarList.addFromString() method. This accepts as parameters the string we want to add, plus optionally a weight value that specifies the importance of this grammar in relation of other grammars available in the list (can be from 0 to 1 inclusive.) The added grammar is available in the list as aSpeechGrammar object instance.

speechRecognitionList.addFromString(grammar, 1);

We then add theSpeechGrammarList to the speech recognition instance by setting it to the value of theSpeechRecognition.grammars property. We also set a few other properties of the recognition instance before we move on:

• SpeechRecognition.continuous: Controls whether continuous results are captured (true), or just a single result each time recognition is started (false).
• SpeechRecognition.lang: Sets the language of the recognition. Setting this is good practice, and therefore recommended.
• SpeechRecognition.interimResults: Defines whether the speech recognition system should return interim results, or just final results. Final results are good enough for this simple demo.
• SpeechRecognition.maxAlternatives: Sets the number of alternative potential matches that should be returned per result. This can sometimes be useful, say if a result is not completely clear and you want to display a list if alternatives for the user to choose the correct one from. But it is not needed for this simple demo, so we are just specifying one (which is actually the default anyway.)

recognition.grammars = speechRecognitionList;recognition.continuous = false;recognition.lang = "en-US";recognition.interimResults = false;recognition.maxAlternatives = 1;

Starting the speech recognition

After grabbing references to the output<div> and the HTML element (so we can output diagnostic messages and update the app background color later on), we implement an onclick handler so that when the screen is tapped/clicked, the speech recognition service will start. This is achieved by callingSpeechRecognition.start(). TheforEach() method is used to output colored indicators showing what colors to try saying.

const diagnostic = document.querySelector(".output");const bg = document.querySelector("html");const hints = document.querySelector(".hints");let colorHTML = "";colors.forEach((color, i) => {  console.log(color, i);  colorHTML += `<span> ${color} </span>`;});hints.innerHTML = `Tap or click then say a color to change the background color of the app. Try ${colorHTML}.`;document.body.onclick = () => {  recognition.start();  console.log("Ready to receive a color command.");};

Receiving and handling results

Once the speech recognition is started, there are many event handlers that can be used to retrieve results, and other pieces of surrounding information (see theSpeechRecognition events.) The most common one you'll probably use is theresult event, which is fired once a successful result is received:

recognition.onresult = (event) => {  const color = event.results[0][0].transcript;  diagnostic.textContent = `Result received: ${color}.`;  bg.style.backgroundColor = color;  console.log(`Confidence: ${event.results[0][0].confidence}`);};

The second line here is a bit complex-looking, so let's explain it step by step. TheSpeechRecognitionEvent.results property returns aSpeechRecognitionResultList object containingSpeechRecognitionResult objects. It has a getter so it can be accessed like an array — so the first[0] returns theSpeechRecognitionResult at position 0. EachSpeechRecognitionResult object containsSpeechRecognitionAlternative objects that contain individual recognized words. These also have getters so they can be accessed like arrays — the second[0] therefore returns theSpeechRecognitionAlternative at position 0. We then return itstranscript property to get a string containing the individual recognized result as a string, set the background color to that color, and report the color recognized as a diagnostic message in the UI.

We also use thespeechend event to stop the speech recognition service from running (usingSpeechRecognition.stop()) once a single word has been recognized and it has finished being spoken:

recognition.onspeechend = () => {  recognition.stop();};

Handling errors and unrecognized speech

The last two handlers are there to handle cases where speech was recognized that wasn't in the defined grammar, or an error occurred. Thenomatch event seems to be supposed to handle the first case mentioned, although note that at the moment it doesn't seem to fire correctly; it just returns whatever was recognized anyway:

recognition.onnomatch = (event) => {  diagnostic.textContent = "I didn't recognize that color.";};

Theerror event handles cases where there is an actual error with the recognition successfully — theSpeechRecognitionErrorEvent.error property contains the actual error returned:

recognition.onerror = (event) => {  diagnostic.textContent = `Error occurred in recognition: ${event.error}`;};

Speech synthesis (aka text-to-speech, or TTS) involves receiving synthesizing text contained within an app to speech, and playing it out of a device's speaker or audio output connection.

The Web Speech API has a main controller interface for this —SpeechSynthesis — plus a number of closely-related interfaces for representing text to be synthesized (known as utterances), voices to be used for the utterance, etc. Again, most OSes have some kind of speech synthesis system, which will be used by the API for this task as available.

To show simple usage of Web speech synthesis, we've provided a demo calledSpeak easy synthesis. This includes a set of form controls for entering text to be synthesized, and setting the pitch, rate, and voice to use when the text is uttered. After you have entered your text, you can pressEnter/Return to hear it spoken.

To run the demo, navigate to thelive demo URL in a supporting mobile browser.

The HTML and CSS are again pretty trivial, containing a title, some instructions for use, and a form with some simple controls. The<select> element is initially empty, but is populated with<option>s via JavaScript (see later on.)

<h1>Speech synthesizer</h1><p>  Enter some text in the input below and press return to hear it. Change voices  using the dropdown menu.</p><form>  <input type="text" />  <div>    <label for="rate">Rate</label    ><input type="range" min="0.5" max="2" value="1" step="0.1" />    <div>1</div>    <div></div>  </div>  <div>    <label for="pitch">Pitch</label    ><input type="range" min="0" max="2" value="1" step="0.1" />    <div>1</div>    <div></div>  </div>  <select></select></form>

Let's investigate the JavaScript that powers this app.

Setting variables

First of all, we capture references to all the DOM elements involved in the UI, but more interestingly, we capture a reference toWindow.speechSynthesis. This is API's entry point — it returns an instance ofSpeechSynthesis, the controller interface for web speech synthesis.

const synth = window.speechSynthesis;const inputForm = document.querySelector("form");const inputTxt = document.querySelector(".txt");const voiceSelect = document.querySelector("select");const pitch = document.querySelector("#pitch");const pitchValue = document.querySelector(".pitch-value");const rate = document.querySelector("#rate");const rateValue = document.querySelector(".rate-value");const voices = [];

Populating the select element

To populate the<select> element with the different voice options the device has available, we've written apopulateVoiceList() function. We first invokeSpeechSynthesis.getVoices(), which returns a list of all the available voices, represented bySpeechSynthesisVoice objects. We then loop through this list — for each voice we create an<option> element, set its text content to display the name of the voice (grabbed fromSpeechSynthesisVoice.name), the language of the voice (grabbed fromSpeechSynthesisVoice.lang), and-- DEFAULT if the voice is the default voice for the synthesis engine (checked by seeing ifSpeechSynthesisVoice.default returnstrue.)

We also createdata- attributes for each option, containing the name and language of the associated voice, so we can grab them easily later on, and then append the options as children of the select.

function populateVoiceList() {  voices = synth.getVoices();  for (const voice of voices) {    const option = document.createElement("option");    option.textContent = `${voice.name} (${voice.lang})`;    if (voice.default) {      option.textContent += " — DEFAULT";    }    option.setAttribute("data-lang", voice.lang);    option.setAttribute("data-name", voice.name);    voiceSelect.appendChild(option);  }}

Older browser don't support thevoiceschanged event, and just return a list of voices whenSpeechSynthesis.getVoices() is fired.While on others, such as Chrome, you have to wait for the event to fire before populating the list.To allow for both cases, we run the function as shown below:

populateVoiceList();if (speechSynthesis.onvoiceschanged !== undefined) {  speechSynthesis.onvoiceschanged = populateVoiceList;}

Speaking the entered text

Next, we create an event handler to start speaking the text entered into the text field. We are using anonsubmit handler on the form so that the action happens whenEnter/Return is pressed. We first create a newSpeechSynthesisUtterance() instance using its constructor — this is passed the text input's value as a parameter.

Next, we need to figure out which voice to use. We use theHTMLSelectElementselectedOptions property to return the currently selected<option> element. We then use this element'sdata-name attribute, finding theSpeechSynthesisVoice object whose name matches this attribute's value. We set the matching voice object to be the value of theSpeechSynthesisUtterance.voice property.

Finally, we set theSpeechSynthesisUtterance.pitch andSpeechSynthesisUtterance.rate to the values of the relevant range form elements. Then, with all necessary preparations made, we start the utterance being spoken by invokingSpeechSynthesis.speak(), passing it theSpeechSynthesisUtterance instance as a parameter.

inputForm.onsubmit = (event) => {  event.preventDefault();  const utterThis = new SpeechSynthesisUtterance(inputTxt.value);  const selectedOption =    voiceSelect.selectedOptions[0].getAttribute("data-name");  for (const voice of voices) {    if (voice.name === selectedOption) {      utterThis.voice = voice;    }  }  utterThis.pitch = pitch.value;  utterThis.rate = rate.value;  synth.speak(utterThis);  utterThis.onpause = (event) => {    const char = event.utterance.text.charAt(event.charIndex);    console.log(      `Speech paused at character ${event.charIndex} of "${event.utterance.text}", which is "${char}".`,    );  };  inputTxt.blur();};

In the final part of the handler, we include apause event to demonstrate howSpeechSynthesisEvent can be put to good use. WhenSpeechSynthesis.pause() is invoked, this returns a message reporting the character number and name that the speech was paused at.

Finally, we callblur() on the text input. This is mainly to hide the keyboard on Firefox OS.

Updating the displayed pitch and rate values

The last part of the code updates thepitch/rate values displayed in the UI, each time the slider positions are moved.

pitch.onchange = () => {  pitchValue.textContent = pitch.value;};rate.onchange = () => {  rateValue.textContent = rate.value;};