The Handwriting Recognition API allows you to recognize text from handwritten input as it happens.
 Christian Liebel [X](https://twitter.com/christianliebel) [GitHub](https://github.com/christianliebel) [Mastodon](https://mastodon.cloud/christianliebel) [Bluesky](https://bsky.app/profile/christianliebel.com) [Homepage](https://christianliebel.com/)  Thomas Steiner [GitHub](https://github.com/tomayac) [LinkedIn](https://www.linkedin.com/in/thomassteinerlinkedin) [Mastodon](https://toot.cafe/@tomayac) [Bluesky](https://bsky.app/profile/tomayac.com) [Homepage](https://blog.tomayac.com/)
## What is the Handwriting Recognition API?
The Handwriting Recognition API allows you to convert handwriting (ink) from your users into text.
Some operating systems have long included such APIs, and with this new capability, your web apps can
finally use this functionality. The conversion takes place directly on the user's device, works even
in offline mode, all without adding any third-party libraries or services.
This API implements so-called "on-line" or near real-time recognition. This means that the
handwritten input is recognized while the user is drawing it by capturing and analyzing the single
strokes. In contrast to "off-line" procedures such as Optical Character Recognition (OCR), where
only the end product is known, on-line algorithms can provide a higher level of accuracy due to
additional signals like the temporal sequence and pressure of individual ink strokes.
### Suggested use cases for the Handwriting Recognition API
Example uses include:
- Note-taking applications where users want to capture handwritten notes and have them translated into text.
- Forms applications where users can use stylus or finger input due to time constraints.
- Games that require filling in letters or numbers, such as crosswords, hangman, or sudoku.
## Current status
The Handwriting Recognition API is available from (Chromium 99).
## How to use the Handwriting Recognition API
### Feature detection
Detect browser support by checking for the existence of the `createHandwritingRecognizer()` method
on the navigator object:
if ('createHandwritingRecognizer' in navigator) {
// 🎉 The Handwriting Recognition API is supported!
}
### Core concepts
The Handwriting Recognition API converts handwritten input into text, regardless of the input method
(mouse, touch, stylus). The API has four main entities:
1. A *point* represents where the pointer was at a particular time.
2. A *stroke* consists of one or more points. The recording of a stroke starts when the user puts the pointer down (i.e., clicks the primary mouse button, or touches the screen with their stylus or finger) and ends when they raise the pointer back up.
3. A *drawing* consists of one or more strokes. The actual recognition takes place at this level.
4. The *recognizer* is configured with the expected input language. It is used to create an instance of a drawing with the recognizer configuration applied.
These concepts are implemented as specific interfaces and dictionaries, which I'll cover shortly.
#### Creating a recognizer
To recognize text from handwritten input, you need to obtain an instance of a
`HandwritingRecognizer` by calling `navigator.createHandwritingRecognizer()` and passing constraints
to it. Constraints determine the handwriting recognition model that should be used. Currently, you
can specify a list of languages in order of preference:
const recognizer = await navigator.createHandwritingRecognizer({
languages: ['en'],
});
> [!CAUTION]
> **Caution:** The current implementation on ChromeOS can only recognize one language at a time. It only supports English (`en`), and a gesture model (`zxx-x-gesture`) to recognize gestures such as crossing out words.
The method returns a promise resolving with an instance of a `HandwritingRecognizer` when the
browser can fulfill your request. Otherwise, it will reject the promise with an error, and
handwriting recognition will not be available. For this reason, you may want to query the
recognizer's support for particular recognition features first.
#### Querying recognizer support
By calling `navigator.queryHandwritingRecognizer()`, you can check if the target platform
supports the handwriting recognition features you intend to use. This method takes
the same constraint object as the `navigator.createHandwritingRecognizer()` method,
containing the list of requested languages. The method returns a promise resolving
with a result object if a compatible recognizer is found. Otherwise, the promise resolves to `null`.
In the following example, the developer:
- wants to detect texts in English
- get alternative, less likely predictions when available
- gain access to the segmentation result, i.e., the recognized characters, including the points and strokes that make them up
const result =
await navigator.queryHandwritingRecognizerSupport({
languages: ['en']
});
console.log(result?.textAlternatives); // true if alternatives are supported
console.log(result?.textSegmentation); // true if segmentation is supported
If the browser supports the feature
needed by the developer, its value will be set to `true` in the result object. Otherwise, it will be set to `false`.
You can use this information to enable or disable certain features within your application or to send a new query for a different set of languages.
> [!NOTE]
> **Note:** Due to fingerprinting concerns, you cannot request a list of supported features, such as particular languages, and the browser may ask for user permission or reject your request entirely if you send too many queries.
#### Start a drawing
Within your application, you should offer an input area where the user makes their handwritten
entries. For performance reasons, it is recommended to implement this with the help of a
[canvas object](https://developer.mozilla.org/docs/Web/API/Canvas_API/Tutorial). The exact
implementation of this part is out of scope for this article, but you may refer to the [demo](https://developer.chrome.com/docs/web-platform/handwriting-recognition#demo)
to see how it can be done.
To start a new drawing, call the `startDrawing()` method on the recognizer. This method takes an
object containing different hints to fine-tune the recognition algorithm. All hints are optional:
- The kind of text being entered: text, email addresses, numbers, or an individual character (`recognitionType`)
- The type of input device: mouse, touch, or stylus input (`inputType`)
- The preceding text (`textContext`)
- The number of less-likely alternative predictions that should be returned (`alternatives`)
- A list of user-identifiable characters ("graphemes") the user will most likely enter (`graphemeSet`)
The Handwriting Recognition API plays well with
[Pointer Events](https://developer.mozilla.org/docs/Web/API/Pointer_events) which provide an
abstract interface to consume input from any pointing device. The pointer event arguments contain
the type of pointer being used. This means you can use pointer events to determine the input type
automatically. In the following example, the drawing for handwriting recognition is automatically
created on the first occurrence of a `pointerdown` event on the handwriting area. As the
`pointerType` may be empty or set to a proprietary value, I introduced a consistency check to make
sure only supported values are set for the drawing's input type.
let drawing;
let activeStroke;
canvas.addEventListener('pointerdown', (event) => {
if (!drawing) {
drawing = recognizer.startDrawing({
recognitionType: 'text', // email, number, per-character
inputType: ['mouse', 'touch', 'stylus'].find((type) => type === event.pointerType),
textContext: 'Hello, ',
alternatives: 2,
graphemeSet: ['f', 'i', 'z', 'b', 'u'], // for a fizz buzz entry form
});
}
startStroke(event);
});
> [!CAUTION]
> **Caution:** The current implementation on ChromeOS does not support grapheme sets yet, they are silently ignored.
#### Add a stroke
The `pointerdown` event is also the right place to start a new stroke. To do so, create a new
instance of `HandwritingStroke`. Also, you should store the current time as a point of reference for
the subsequent points added to it:
function startStroke(event) {
activeStroke = {
stroke: new HandwritingStroke(),
startTime: Date.now(),
};
addPoint(event);
}
#### Add a point
After creating the stroke, you should directly add the first point to it. As you will add more
points later on, it makes sense to implement the point creation logic in a separate method. In the
following example, the `addPoint()` method calculates the elapsed time from the reference timestamp.
The temporal information is optional, but can improve recognition quality. Then, it reads the X and
Y coordinates from the pointer event and adds the point to the current stroke.
function addPoint(event) {
const timeElapsed = Date.now() - activeStroke.startTime;
activeStroke.stroke.addPoint({
x: event.offsetX,
y: event.offsetY,
t: timeElapsed,
});
}
The `pointermove` event handler is called when the pointer is moved across the screen. Those points
need to be added to the stroke as well. The event can also be raised if the pointer is not in a
"down" state, for example when moving the cursor across the screen without pressing the mouse
button. The event handler from the following example checks if an active stroke exists, and adds the
new point to it.
canvas.addEventListener('pointermove', (event) => {
if (activeStroke) {
addPoint(event);
}
});
#### Recognize text
When the user lifts the pointer again, you can add the stroke to your drawing by calling its
`addStroke()` method. The following example also resets the `activeStroke`, so the `pointermove`
handler will not add points to the completed stroke.
> [!NOTE]
> **Note:** If necessary, you can also use the drawing's `getStrokes()` method to list all strokes, and the `removeStroke()` method to remove a particular one from the drawing.
Next, it's time for recognizing the user's input by calling the `getPrediction()` method on the
drawing. Recognition usually takes less than a few hundred milliseconds, so you can repeatedly run
predictions if needed. The following example runs a new prediction after each completed stroke.
canvas.addEventListener('pointerup', async (event) => {
drawing.addStroke(activeStroke.stroke);
activeStroke = null;
const [mostLikelyPrediction, ...lessLikelyAlternatives] = await drawing.getPrediction();
if (mostLikelyPrediction) {
console.log(mostLikelyPrediction.text);
}
lessLikelyAlternatives?.forEach((alternative) => console.log(alternative.text));
});
This method returns a promise which resolves with an array of predictions ordered by their
likelihood. The number of elements depends on the value you passed to the `alternatives` hint. You
could use this array to present the user with a choice of possible matches, and have them select an
option. Alternatively, you can simply go with the most likely prediction, which is what I do in the
example.
The prediction object contains the recognized text and an optional segmentation result, which I will
discuss in the following section.
#### Detailed insights with segmentation results
If supported by the target platform, the prediction object can also contain a segmentation result.
This is an array containing all recognized handwriting segment, a combination of the recognized
user-identifiable character (`grapheme`) along with its position in the recognized text
(`beginIndex`, `endIndex`), and the strokes and points that created it.
if (mostLikelyPrediction.segmentationResult) {
mostLikelyPrediction.segmentationResult.forEach(
({ grapheme, beginIndex, endIndex, drawingSegments }) => {
console.log(grapheme, beginIndex, endIndex);
drawingSegments.forEach(({ strokeIndex, beginPointIndex, endPointIndex }) => {
console.log(strokeIndex, beginPointIndex, endPointIndex);
});
},
);
}
You could use this information to track down the recognized graphemes on the canvas again.
#### Complete recognition
After the recognition has completed, you can free resources by calling the `clear()` method on the
`HandwritingDrawing`, and the `finish()` method on the `HandwritingRecognizer`:
drawing.clear();
recognizer.finish();
## Demo
The web component `` implements a
[progressively enhanced](https://web.dev/articles/progressively-enhance-your-pwa), editing control capable of handwriting
recognition. By clicking the button in the lower right corner of the editing control, you activate
the drawing mode. When you complete the drawing, the web component will automatically start the
recognition and add the recognized text back to the editing control. If the Handwriting Recognition
API is not supported at all, or the platform doesn't support the requested features, the edit button
will be hidden. But the basic editing control remains usable as a `