Chrome 64 comes with a highly anticipated new feature in Web Audio API - AudioWorklet. Here you'll learn concepts and usage to create a custom audio processor with JavaScript code. Take a look at the live demos. The next article in series, Audio Worklet Design Pattern, might be an interesting read for building an advanced audio app.
Background: ScriptProcessorNode
Audio processing in Web Audio API runs in a separate thread from the main UI thread, so it runs smoothly. To enable custom audio processing in JavaScript, the Web Audio API proposed a ScriptProcessorNode which used event handlers to invoke user script in the main UI thread.
There are two problems in this design: the event handling is asynchronous
by design, and the code execution happens on the main thread. The former
induces the latency, and the latter pressures the main thread that is
commonly crowded with various UI and DOM-related tasks causing either UI
to "jank" or audio to "glitch". Because of this fundamental design flaw,
ScriptProcessorNode is deprecated from the specification and
replaced with AudioWorklet.
Concepts
Audio Worklet keeps the user-supplied JavaScript code all within the
audio processing thread. That means it doesn't have to jump over to the main
thread to process audio. This means the user-supplied script code gets to run
on the audio rendering thread (AudioWorkletGlobalScope) along with other
built-in AudioNodes, which ensures zero additional latency and synchronous
rendering.
Registration and instantiation
Using Audio Worklet consists of two parts: AudioWorkletProcessor and
AudioWorkletNode. This is more involved than using ScriptProcessorNode,
but it is needed to give developers the low-level capability for custom audio
processing. AudioWorkletProcessor represents the actual audio processor
written in JavaScript code, and it lives in the AudioWorkletGlobalScope.
AudioWorkletNode is the counterpart of AudioWorkletProcessor and takes
care of the connection to and from other AudioNodes in the main thread. It
is exposed in the main global scope and functions like a regular AudioNode.
Here's a pair of code snippets that demonstrate the registration and the instantiation.
// The code in the main global scope.
class MyWorkletNode extends AudioWorkletNode {
constructor(context) {
super(context, 'my-worklet-processor');
}
}
let context = new AudioContext();
context.audioWorklet.addModule('processors.js').then(() => {
let node = new MyWorkletNode(context);
});
To create an AudioWorkletNode, you must add an AudioContext
object and the processor name as a string. A processor definition can be
loaded and registered by the new Audio Worklet object's addModule() call.
Worklet APIs including Audio Worklet are only available in a
secure context, thus a
page using them must be served over HTTPS, although http://localhost is
considered a secure for local testing.
You can subclass AudioWorkletNode to define
a custom node backed by the processor running on the worklet.
// This is the "processors.js" file, evaluated in AudioWorkletGlobalScope
// upon audioWorklet.addModule() call in the main global scope.
class MyWorkletProcessor extends AudioWorkletProcessor {
constructor() {
super();
}
process(inputs, outputs, parameters) {
// audio processing code here.
}
}
registerProcessor('my-worklet-processor', MyWorkletProcessor);
The registerProcessor() method in the AudioWorkletGlobalScope takes a
string for the name of processor to be registered and the class definition.
After the completion of script code evaluation in the global scope, the
promise from AudioWorklet.addModule() will be resolved notifying users
that the class definition is ready to be used in the main global scope.
Custom audio parameters
One of the useful things about AudioNodes is schedulable parameter
automation with AudioParam. AudioWorkletNodes can use these to get
exposed parameters that can be controlled at the audio rate automatically.
User-defined audio parameters can be declared in an AudioWorkletProcessor
class definition by setting up a set of AudioParamDescriptor. The
underlying WebAudio engine picks up this information during the
construction of an AudioWorkletNode, and then creates and links
AudioParam objects to the node accordingly.
/* A separate script file, like "my-worklet-processor.js" */
class MyWorkletProcessor extends AudioWorkletProcessor {
// Static getter to define AudioParam objects in this custom processor.
static get parameterDescriptors() {
return [{
name: 'myParam',
defaultValue: 0.707
}];
}
constructor() { super(); }
process(inputs, outputs, parameters) {
// |myParamValues| is a Float32Array of either 1 or 128 audio samples
// calculated by WebAudio engine from regular AudioParam operations.
// (automation methods, setter) Without any AudioParam change, this array
// would be a single value of 0.707.
const myParamValues = parameters.myParam;
if (myParamValues.length === 1) {
// |myParam| has been a constant value for the current render quantum,
// which can be accessed by |myParamValues[0]|.
} else {
// |myParam| has been changed and |myParamValues| has 128 values.
}
}
}
AudioWorkletProcessor.process() method
The actual audio processing happens in the process() callback method in the
AudioWorkletProcessor. It must be implemented by a user in the class
definition. The WebAudio engine invokes this function in an isochronous
fashion to feed inputs and parameters and fetch outputs.
/* AudioWorkletProcessor.process() method */
process(inputs, outputs, parameters) {
// The processor may have multiple inputs and outputs. Get the first input and
// output.
const input = inputs[0];
const output = outputs[0];
// Each input or output may have multiple channels. Get the first channel.
const inputChannel0 = input[0];
const outputChannel0 = output[0];
// Get the parameter value array.
const myParamValues = parameters.myParam;
// if |myParam| has been a constant value during this render quantum, the
// length of the array would be 1.
if (myParamValues.length === 1) {
// Simple gain (multiplication) processing over a render quantum
// (128 samples). This processor only supports the mono channel.
for (let i = 0; i < inputChannel0.length; ++i) {
outputChannel0[i] = inputChannel0[i] * myParamValues[0];
}
} else {
for (let i = 0; i < inputChannel0.length; ++i) {
outputChannel0[i] = inputChannel0[i] * myParamValues[i];
}
}
// To keep this processor alive.
return true;
}
Additionally, the return value of the process() method can be used to
control the lifetime of AudioWorkletNode so that developers can manage
the memory footprint. Returning false from process() method marks
the processor inactive, and the WebAudio engine no longer invokes the
method. To keep the processor alive, the method must return true.
Otherwise, the node and processor pair is garbage collected by the system
eventually.
Bi-directional communication with MessagePort
Sometimes, a custom AudioWorkletNode wants to expose controls that don't
map to AudioParam, such as a string-based type attribute
used to control a custom filter. For this purpose and beyond,
AudioWorkletNode and AudioWorkletProcessor are equipped with a
MessagePort for bi-directional communication. Any kind of custom data
can be exchanged through this channel.
MessagePort can be accessed with the .port attribute on both the node and
the processor. The node's port.postMessage() method sends a message to
the associated processor's port.onmessage handler and in reverse.
/* The code in the main global scope. */
context.audioWorklet.addModule('processors.js').then(() => {
let node = new AudioWorkletNode(context, 'port-processor');
node.port.onmessage = (event) => {
// Handling data from the processor.
console.log(event.data);
};
node.port.postMessage('Hello!');
});
/* "processors.js" file. */
class PortProcessor extends AudioWorkletProcessor {
constructor() {
super();
this.port.onmessage = (event) => {
// Handling data from the node.
console.log(event.data);
};
this.port.postMessage('Hi!');
}
process(inputs, outputs, parameters) {
// Do nothing, producing silent output.
return true;
}
}
registerProcessor('port-processor', PortProcessor);
MessagePort supports transferable, which lets you
transfer data storage or a WASM module over the thread boundary. This opens
up countless possibility on how the Audio Worklet system can be used.
Walk through: Build a GainNode
Here's a complete example of GainNode built on top of
AudioWorkletNode and AudioWorkletProcessor.
The index.html file:
<!doctype html>
<html>
<script>
const context = new AudioContext();
// Loads module script with AudioWorklet.
context.audioWorklet.addModule('gain-processor.js').then(() => {
let oscillator = new OscillatorNode(context);
// After the resolution of module loading, an AudioWorkletNode can be
// constructed.
let gainWorkletNode = new AudioWorkletNode(context, 'gain-processor');
// AudioWorkletNode can be interoperable with other native AudioNodes.
oscillator.connect(gainWorkletNode).connect(context.destination);
oscillator.start();
});
</script>
</html>
The gain-processor.js file:
class GainProcessor extends AudioWorkletProcessor {
// Custom AudioParams can be defined with this static getter.
static get parameterDescriptors() {
return [{ name: 'gain', defaultValue: 1 }];
}
constructor() {
// The super constructor call is required.
super();
}
process(inputs, outputs, parameters) {
const input = inputs[0];
const output = outputs[0];
const gain = parameters.gain;
for (let channel = 0; channel < input.length; ++channel) {
const inputChannel = input[channel];
const outputChannel = output[channel];
if (gain.length === 1) {
for (let i = 0; i < inputChannel.length; ++i)
outputChannel[i] = inputChannel[i] * gain[0];
} else {
for (let i = 0; i < inputChannel.length; ++i)
outputChannel[i] = inputChannel[i] * gain[i];
}
}
return true;
}
}
registerProcessor('gain-processor', GainProcessor);
This covers the fundamental of Audio Worklet system. Live demos are available at Chrome WebAudio team's GitHub repository.
Feature transition: Experimental to Stable
Audio Worklet is enabled by default for Chrome 66 or later. In Chrome 64 and 65, the feature was behind the experimental flag.