Gregory Friedman The Greg Wide Web on the name →

Case Study · Computer Vision · ML

Emotions By Face

2025 · repository

The idea

emotions-by-face is a team course project (we were Team 3) built on FER2013, the classic facial-expression dataset: 35,887 photographs of faces, each a mere 48×48 grayscale pixels, labeled with one of seven emotions: angry, disgust, fear, happy, sad, surprise, neutral. The task sounds soft but the dataset is famously hard: the images are tiny, the lighting is wild, the labels are noisy enough that human agreement is estimated around 65%. Nobody gets 95% on FER2013; the interesting questions are all about what your errors look like.

Our approach was deliberate transfer learning on a budget: take a ResNet18 pretrained on ImageNet, freeze the entire backbone, and retrain only the final layer as a 7-way classifier. Upsample the 48-pixel faces to 112×112 (the notebook’s comment, “changed from 224 for CPU constraints”, is its own little engineering lesson), normalize with ImageNet statistics, add flips and small rotations, and let eleven million frozen parameters do the featurizing while 3,591 trainable ones do the deciding. Six epochs of Adam on the head, and the notebook reports 59.0% test accuracy across 7,178 held-out images.

The plate below is the model I ended up with, quantized to int8 and running entirely in your browser; hand it a face first, then read how it got here, including the reproduction that had to happen first.

Model card. ResNet18 head on FER2013

Per-class recall of the emotion classifier on the FER2013 test set.
classrecall
Angry33.5%
Disgust20.7%
Fear29.4%
Happy86.8%
Sad45.6%
Surprise66.3%
Neutral51.6%

Overall test accuracy 54.9% across 7,178 images; the classifier is a frozen ImageNet ResNet18 with only its final layer retrained, so these numbers are the price of keeping the backbone untouched. Running it on your own photo needs JavaScript.

Quantized weights (11.3 MB), the ONNX WebAssembly runtime (13.5 MB), and a 234 KB face-finding cascade, fetched from this site only when you ask. Inference runs entirely in your browser: photos and camera frames never leave the page.

Plate I The reproduced model, quantized to int8 and running in your browser via onnxruntime-web. Load it, then hand it a photo or take one with the camera: a Viola-Jones-style cascade locates the face, zooms in on it, and shows you the 48×48 grayscale crop that is all the model ever gets to see. Photos and camera frames never leave the page.

The reproduction

Putting this model on the site meant reproducing it first. The notebook saves its weights to a models/ directory that was never committed, and the data comes from Kaggle and was correctly left out of the repository too (it’s 100 MB and not ours to redistribute). The repo contained the complete recipe and none of the cake.

So I re-ran the notebook’s recipe, top to bottom: same architecture, same frozen backbone, same augmentations, optimizer, learning rate, epoch count, even the same seeded train/validation shuffle. The committed cells produce a model with 37.4% test accuracy. The notebook’s own recorded output says 59.0%.

A 21-point gap is not a random-seed story; it’s a stateful notebook story. Look closely and the evidence was there all along: the training cell only ever optimizes the head, but the file it ultimately saves is named resnet18_finetuned.pth. That name is a fossil: the residue of work done live in the session (training loops re-run, layers unfrozen, learning rates fiddled) that never crystallized back into code. The notebook’s results were real; its history was gone. Every notebook is an experiment log where the pages can rewrite themselves.

The model on this page therefore finishes the sentence the filename started: continue from the committed recipe’s checkpoint, unfreeze the last residual block, and fine-tune it alongside the head at a gentle learning rate. That lands at 55.5% test accuracy, within four points of the notebook’s number, reached by the path its artifact name implies. The full two-phase script lives in this site’s repo, pinned and runnable, so this number, unlike the last one, can be regenerated on demand.

Shipping it to the browser

Getting a PyTorch model into a web page is a pipeline with three shrinking stages, plus a doorman:

  1. Export: torch.onnx.export traces the model into an ONNX graph (44.7 MB of fp32 weights; ResNet18 is “small” only by ImageNet standards).
  2. Quantize: rewrite the weights as 8-bit integers: 11.3 MB, a 4× shrink. The first attempt, dynamic quantization, silently cost 6.5 points of accuracy; convolutions hate per-tensor scales chosen at runtime. Static quantization, calibrated on 512 training images with per-channel scales, costs 0.6 points (54.9% vs 55.5%, measured on the full test set, not assumed).
  3. Locate: FER2013 faces fill the frame, so your photo should too. Before the classifier sees anything, a face finder crops the photo down to the face: pico, a 234 KB cascade in the direct lineage of Viola-Jones (2001). The structure is the same idea that made real-time detection possible: slide a window over the image at every position and scale, and let a cascade of weak classifiers reject non-faces as early as possible, so almost all windows cost almost nothing. Where Viola-Jones used Haar features over an integral image, pico’s stages are small decision trees over pairwise pixel comparisons, which need no preprocessing at all; a whole frame takes milliseconds in plain JavaScript. The plate zooms in on the best detection (and falls back to a center crop, and tells you, when nothing clears threshold).
  4. Run: onnxruntime-web executes the graph in WebAssembly, entirely client-side. The face crop is preprocessed in a canvas; down to 48×48 grayscale (the model has never seen anything sharper, so neither should you), back up to 112, ImageNet-normalize; and then handed to the same graph the test set was scored on.

Nothing is uploaded anywhere; there is no server to upload to. The plate at the top of this page fetches the weights, the runtime, and the cascade once, on click, and then works on an airplane. The camera path is the same story: permission is asked only when you press the button, a single frame is snapped to a canvas, the stream stops that instant, and the frame walks the same locate-crop-classify pipeline without ever leaving the page.

What it looks with

The plate above shows what the model says. This one shows what it says it with: every kernel of its first convolutional layer, read out of the same 11.3 MB file the page has just offered you.

Only the first layer can be looked at, because it is the only convolution whose input is an image and therefore the only one whose kernels are pictures. layer4 slides 3×3 windows over 512 channels of feature map, and a picture of that is a picture of nothing. The one drawable layer is also the one layer this project never trained: the backbone was frozen through all three stages and only layer4 and the head ever took a gradient step. So the shapes below are ImageNet’s, arriving here unchanged up to a per-filter scalar and int8 rounding.

The scalars are another matter. BatchNorm’s running mean and variance are buffers rather than parameters, so freezing the weights never froze them; they kept tracking FER2013 through every epoch spent in model.train(). Export folds each BatchNorm into the convolution ahead of it, which multiplies those FER2013 statistics straight into ImageNet’s filters, and the layer that was never trained on faces turns out to have been rescaled by them anyway. Three things fall out of that, and all three are in the plate.

Show
Weight

negative0positive

Shading follows the square root of the weight, so a kernel two orders of magnitude down stays visible; each kernel is scaled against its own loudest cell, so these are shapes, not sizes. The sizes are in the panel below. Applied is the three planes added in the model's own input units, each divided by its ImageNet σ, which is all a grey face can excite. Dashed cells are the 8 filters that are numerically zero.

Applied
R
G
B

Filter 28 of 64

Largest weight
2.75
Folded bias
4.29
Reachable by grey
0.2%
int8 step
0.022

Filter 28: largest weight 2.75, rank 1 of 56 in the layer (the median is 0.0832). 0.2% of its weight lies along the one colour direction a grey face can excite, so 0.2% of it is what the Applied view shows.

Read straight out of emotions.int8.onnx, the 11.3 MB file this page downloads: 64 kernels of 3 × 7 × 7, dequantized from int8 with their per-filter scales. Batch-norm is folded into these numbers, so each is the product of a convolution weight and a normalization scale and the artifact cannot say which factor a zero came from. It can say the product is zero.

Plate II All 64 kernels of the first convolutional layer, dequantized from the shipped int8 graph with their per-filter scales. Ochre is a positive weight, green a negative one, on a square-root ramp so a faint kernel is faint rather than absent. Applied is the three colour planes added in the model's own input units, which is the only combination a grey face can excite; R, G and B are the planes on their own. Press a kernel to open it.

Eight of the 64 are off. Filters 2, 4, 7, 9, 13, 36, 38 and 48 have largest weights running from 2.0 × 10⁻⁸ down to 1.4 × 10⁻¹⁴, against a median of 0.083 for the other 56, and their folded biases are dead in the same proportion (1.7 × 10⁻⁸ to 3.5 × 10⁻⁵, against a median of 0.31). Multiply an image by that and you get zero; the ReLU behind them passes the zero on. An eighth of the first layer is doing nothing at all. Press one and the ImageNet filter is still legible underneath, drawn at some 10¹² times its own amplitude: a fossil of a filter, with the gain gone. What the artifact cannot tell you is which factor the zero lives in, since folding has already multiplied the convolution weight by the normalization scale and only the product survives.

Twelve more encode colour, and there is no colour. The pipeline hands this model a 48×48 grey crop replicated into three planes, then normalizes each plane with its own ImageNet σ, so the three inputs are affine copies of one grey image and the only part of a kernel that can respond to content is its component along (1/σR,1/σG,1/σB)(1/\sigma_R, 1/\sigma_G, 1/\sigma_B). Twelve filters have under a tenth of their weight in that direction, and seven of those have under a hundredth. Switch the plate from Applied to R and watch what those twelve turn into: a mottled residue under Applied resolves into a clean blob. They are colour-opponent detectors, and in every one of the twelve two of the three planes point in opposite directions (cosine under −0.7-0.7), which is exactly the job they were trained for on photographs that had colour in them.

And those twelve are the loudest filters in the layer. The five largest weights in the whole convolution belong to filters whose reachable share is under 1%. That is BatchNorm behaving exactly as designed: a channel whose output nearly vanishes has a nearly vanishing variance, and folding divides by its square root, so the filters with the least to say were amplified the most. The amplification restores the volume and not the content. What those twelve now apply to a grey face is the small achromatic residue of a colour detector, turned up until it is roughly as loud as its neighbours, and they end up carrying about 14% of the layer’s applied weight where an even twelve-of-fifty-six split would be 21%.

None of that is visible in the accuracy number, and none of it required running the model even once. A first layer is the one part of a network you can audit by reading the file, which is worth holding onto while reading the next section, where nothing can be read without running it.

Asking it what it looked at

The obvious next question about a 55% classifier is which pixels it is using, and the obvious tool is Grad-CAM. Neither the tool nor a good version of the figure is available here, for reasons worth writing down. Grad-CAM needs a gradient with respect to the last convolutional feature map, and onnxruntime-web is an inference runtime with no autograd; the int8 graph declares one output, logits, and hands back nothing in between. What survives the browser is occlusion sensitivity (Zeiler and Fergus 2014): cover a patch of the input, re-run, and record the drop. That needs only forward passes, and forward passes here are expensive. Measured on one laptop under the same single-threaded WebAssembly runtime this page uses, a pass of this ResNet18 costs about 28 ms against roughly 1 ms for the small OCR network next door, and the weights are 11.3 MB against 0.5 MB. The deciding objection isn’t the arithmetic though; it’s that this model has no input until you give it one, so an occlusion figure here could never be a finished figure, only a promise of one. So the sweep is built where it can be read for free, on the OCR case study, over a 28×28 grid and a real test image, with the method named and its failure modes on the same screen.

It is worth being clear about what such a map would and would not settle here, because the temptation on a page like this one is to treat a heat map as an audit. It is not one. A saliency map is a statement about one image: these pixels, this patch size, this fill value. The hard question this project raises is about a distribution over people, and the two do not meet. A map showing that the classifier’s “angry” reading rests on the mouth region of one face says nothing about whether that reading moves with the skin tone of the next thousand faces, and FER2013 records no demographics at all, so the grouping the audit would need does not exist in the data. Per-image explanation cannot substitute for per-group evaluation; it can only tell you a story compelling enough that you stop asking for the evaluation.

There is also a resolution trap specific to this model. The pipeline hands the network 112×112 inputs that contain 48×48 of information, because that’s what the frozen backbone expects. Any heat map drawn over that input inherits its apparent sharpness from the upsampling and not from the evidence, so a convincing-looking blob five pixels across is really a blob two pixels across, smeared. The polite fiction in the preprocessing becomes an impolite one the moment you draw a picture on top of it, which is a good reminder that interpretability figures are downstream of every choice made before them.

Whose faces, and whose emotions

There is a bias question underneath the accuracy number, and it is worth naming plainly rather than leaving in the confusion matrix. FER2013 was assembled from a web image search, and it ships with no demographic labels: age, race, and gender are simply unrecorded. So the composition of the training faces is unknown and almost certainly skewed, and per-group accuracy cannot even be measured from this dataset, let alone corrected. A model is only as auditable as the data behind it, and this one is not auditable at all.

That silence matters because automated emotion reading carries documented demographic bias. Rhue (2018) ran official NBA headshots through commercial facial-analysis systems and found that comparably smiling players were scored as angrier or more contemptuous when the player was Black. The mechanism is not exotic: if the training faces skew light, and the label “angry” ends up correlated with features a small model can’t separate from skin tone or physiognomy, the error stops being abstract and lands on real people. A 59% test number says nothing about whether those 41 points of error fall evenly across who is in front of the camera. Here they almost certainly do not, and the demo has no way to show you where.

The deeper problem is the taxonomy itself. The seven “basic” emotions come from Ekman’s universality thesis, and that thesis is contested: reviewing the evidence, Barrett and colleagues (2019) concluded that facial configurations do not map to internal emotional states reliably across people or cultures. The premise that a face simply has one of seven emotions is a strong, culturally particular assumption, and it is baked into the labels before any network sees a pixel. This plate is a study object, not a product: it is here to show what a small classifier does and does not learn, and it should never be used to judge a real person’s feelings. The blind spot is not a footnote to the demo; on a task like this it is half the lesson.

What I would build next

  • Frozen backbones pay you in speed and charge you in accuracy. Head-only training was the right call for a course timeline (minutes per epoch on a laptop) but ImageNet features were tuned for objects in photographs, not low-resolution grayscale faces. Fine-tuning the last residual block (or training a small CNN from scratch at the native 48×48, which strong FER2013 baselines do) is where the next several points are.
  • The confusion matrix is the real report card. Happy is easy (the notebook’s recall: 88%) because smiles are geometric; fear is brutal (24%) because at 48 pixels fear and surprise share a face. And disgust has only 111 test examples; 1.5% of the set; so the class the model is worst at is also the one the accuracy number barely notices. Class imbalance isn’t a footnote; on FER2013 it’s the plot.
  • Commit the weights, or commit to regenerating them. Reproducing this model from its notebook is what made the number on this page regenerable, and the pinned two-phase script is the form that lesson took. Pin the seed, save the artifact, write down the numbers you got; your future self is the peer reviewer who shows up with the least context.
  • Upsampling 48→112 is a polite fiction. The model “sees” 112×112 inputs that contain 48×48 of information, because that’s what the pretrained backbone expects. It works, but it spends ~5× the compute the information content justifies; one more argument for the native- resolution CNN in the previous bullet.

Keys on this page

A figure's keys are its own: they answer once the figure has focus, so tab to it or click it first. Esc closes this sheet.