Case Study · ML · Vision
OCR in the Browser
The idea
In a graduate ML course (CSE 5819) my group worked on optical character recognition with respectably heavy machinery. The project left me with an itch the course couldn’t scratch: OCR’s smallest interesting unit (one glyph, one answer) is compact enough to live inside a web page, and I already knew the route, because emotions-by-face had walked it: train in PyTorch, export to ONNX, quantize to int8, run in the browser with onnxruntime-web. This is that route, walked again on purpose. Pipelines you’ve only used once are anecdotes; the second pass is what makes them yours.
The dataset is EMNIST Balanced: 131,600 handwritten characters in 47 classes: ten digits, twenty-six uppercase letters, and the eleven lowercase forms (a, b, d, e, f, g, h, n, q, r, t) whose shapes don’t simply merge with their capitals. That last clause is the dataset quietly teaching its first lesson: for the other fifteen letters, no classifier can tell the cases apart, because at 28×28 a handwritten c and C differ only in size, and size is exactly what preprocessing normalizes away.
The plate below is the finished network, running entirely in your browser; draw a character first, then read how it was trained and shrunk to half a megabyte.
A 28×28 grid of gray values is all the model gets; no strokes, no order, no pen. From that it names one of 47 characters (digits, uppercase, and the eleven lowercase forms that don't merge with their capitals). The network is a ~430k-parameter CNN trained on EMNIST Balanced to 88.5% test accuracy, quantized to half a megabyte, and it runs entirely in this page; nothing you draw is uploaded anywhere.
The model waits until you ask for it; this plate downloads nothing on page load.
The approach
The model is a deliberately small CNN (three conv layers and a 128-unit head, ~430k parameters) trained for six epochs with Adam and cosine decay. It reaches 88.5% on the 18,800-image test set, which is close to the practical ceiling for single-glyph EMNIST; the residual errors are concentrated exactly where they should be. The worst classes are L (51%), F (53%), q (56%), g (60%), and 0 (64%): every one a member of a confusable family (l/I/L/1, q/9/g, 0/O) that no pixel grid can disambiguate without knowing the word the glyph sits in. The lesson generalizes: when a classifier’s confusion matrix matches human confusion, the remaining error usually isn’t in the model. It’s in the task.
Dynamic int8 quantization then cuts the weights from 1.9 MB to 0.5 MB and costs 0.02 accuracy points, a trade so lopsided it barely deserves the word.
Two engineering details carry the browser demo, and both are contracts between training and inference. First, EMNIST stores its images transposed relative to how anyone draws, so the training transform untransposes them once, cheaper than asking every future client to remember. Second, the drawing pad reproduces the classic MNIST preprocessing recipe in TypeScript: crop to the ink’s bounding box, scale it down, place the glyph’s center of mass at the center of the 28×28 frame, and normalize with the training set’s own statistics. Even that recipe needed measuring rather than trusting: the textbook says scale to 20 pixels, but pushing 400 real test images through this exact pipeline showed EMNIST’s own crops run larger (24 pixels scores three points higher), and re-preprocessing an already-normalized image costs accuracy no matter what, which is why the plate’s Sample button feeds the model untouched test images instead of round-tripping them through the pad’s pipeline. A preprocessing contract isn’t something you write once at training time; it’s something you verify at both ends.
What the network is actually reading
88.5% tells you how often the model is right. It says nothing about what it is
looking at, and those are different questions: a classifier can be right for
the wrong reasons. The textbook way to ask the second question of
a CNN is Grad-CAM, which weights the last convolutional feature map by the
gradient of the class score flowing into it. In this page, both halves of that
sentence are unavailable. onnxruntime-web is an inference runtime and has no
autograd, so there is no gradient to be had; and the graph declares exactly one
output, logits, so the feature map is not readable either (asking the session
for /conv3/Conv_output_0 is refused: “‘fetches’ contains invalid output
name”). The int8 export closes the door a second time, since its convolutions
are ConvInteger nodes and a gradient through integer arithmetic isn’t defined
without the dequantization scaffolding a training graph carries.
So the workable method is the older one, and it happens to be the one that needs nothing from inside the network at all: occlusion sensitivity, from Zeiler and Fergus’s 2014 paper on visualizing convnets. Run the model once on the untouched image and note its answer. Then slide a small square over the input, one position at a time, overwrite whatever is under it, and run the model again; the drop in the score for the original answer is that position’s attribution. It is a black-box method, which is exactly the relationship a web page is able to have with a model, and its whole cost is one forward pass per position: 784 of them here, one per pixel, about a second in the browser. Being pushed to a weaker technique by the deployment constraint is not a consolation prize; the constraint is this project’s subject, and it turns out the 2014 method survives it while the 2017 one does not.
28 × 28 px · EMNIST test set
7 × 7 · erased to the background
Drop in P(3) when the patch lands here
099.8 points
Shading is relative to the largest drop in this sweep, so the ramp is a scale for this map and no other. Blank means the answer did not move.
Untouched, the network reads this specimen as 3 at 99.9%. The costliest 7 × 7 patch is centred on column 24, row 16: cover it and the reading becomes Z at 98.8%.
Occluding never raised the score for 3 by more than 0.1 points anywhere in this sweep, so the map gives up nothing by shading drops only.
A sweep is 784 forward passes of the network, one per pixel, and takes a second or two in the browser. The map above is the recorded answer for these settings; load the model to recompute it, or to try another specimen, patch or occluder.
Read the default map and the striking thing is how local a 99.9% is. Almost all of this 3’s evidence sits in the lower right bowl: erase the 7×7 square centred on column 24, row 16 and the reading goes from 3 at 99.9% to Z at 98.8%, one confident answer swapped for another confident answer by twenty pixels. The e is better still. Erase its crossbar and the network reads C at 100.0%, which is not a failure at all but the dataset’s first lesson coming back around: an e without its bar is a c, and EMNIST Balanced’s C class stands for both cases. Patch size is the other knob worth turning, because it changes what question you asked. At 3×3 no single position costs the A more than 8.6 points; at 7×7 the same letter collapses by 99.3. “Which pixels matter” has no answer independent of the scale at which you ask.
The caveats are sharper than the findings, and the plate is built to make them visible rather than to hide them. Switch the occluder to Blotted and the map grows a second life away from the glyph: a 7×7 blot of ink dropped in the empty bottom-left corner, touching no part of the Q, costs the model 87.6 points of confidence. That region of the map is not reporting which pixels the network reads; it is reporting what happens when you put an out-of-distribution blob in the frame, which is a fact about the patch and not about the letter. The Erased occluder has the mirror-image problem: off the glyph its map is exactly 0.0 everywhere, necessarily, because erasing blank paper to background leaves the input unchanged. The blank there means this method cannot ask, not the model ignores this. Which is the general shape of the thing. A saliency map licenses one sentence: the answer depends on these pixels, at this patch size, with this fill. It does not license “and that is why the answer is right”. Models attend to the wrong thing and get the right answer all the time, and no heat map will tell you which case you are in.
Scope, and the next rung
The demo’s known failure mode is the gap between EMNIST’s pen-on-paper strokes and a finger on a phone screen: the model was trained on digitized handwriting, and thick uniform browser strokes are slightly out of distribution; the preprocessing narrows that gap but doesn’t close it. Second, the confusable families are a permanent ceiling by construction: this plate reads characters, and resolving l from 1 requires reading words, which is a different project (a sequence model over glyph lattices) and a natural next rung. Third, the pipeline lesson: the second use of a pipeline is where the generalization happens. Emotions-by-face took a full QA pass to get ort-web loading correctly; this plate inherited every one of those decisions (the /wasm build, self-hosted runtime, deferred download) for free, and went from repo to working demo in an afternoon.