Gregory Friedman The Greg Wide Web on the name →

Case Study · NLP · ML

Sentiment, Four Ways

2025 · demo

The idea

During my study-abroad term at the University of Essex I took CE807, the text-analytics module, and its final assignment was a small model zoo: classify appliance reviews as positive or negative four different ways, then explain not just who wins but why each model fails the way it does. The four were VADER, a rule-based lexicon scorer that needs no training data at all; TF-IDF with logistic regression, the classic discriminative baseline; a weighted ensemble of the two; and a fine-tuned DistilBERT as the state-of-the-art ceiling. The dataset was graded coursework material, so it stays where it is, and the numbers below are quoted as coursework results. But the lesson of the assignment deserved better than a PDF: the interesting part was never the leaderboard, it was the error analysis. Sarcasm, negation scope, Spanish reviews, creative spelling, idioms: each error mode defeats a different model for a different structural reason.

So this page rebuilds the comparison as something you can poke. The two models that are small enough to run in a browser really do run in your browser, live, on whatever you type; the two that are not are recorded on a fixed gallery of hand-written trick sentences, one pair per error mode. Everything is self-hosted and nothing you type leaves the page.

Model A · Lexicon + rules live

positive compound +0.64

scored words

  • puddle +1.4flipped
  • quiet +1.8

Model B · Hashed n-gram LR live

positive p(positive) = 0.76

strongest n-grams

  • no +0.30
  • runs +0.26
  • quiet +0.24
  • no puddle +0.18
  • whisper quiet +0.13
  • whisper +0.08
Plate I Two models score your sentence as you type. Model A is a compact lexicon scorer in the style of VADER: each word carries a valence, negators flip it, intensifiers boost it, and the word chips show every rule that fired. Model B is a logistic regression over hashed unigrams and bigrams, trained offline on a synthetic corpus; its chips show the n-grams that pulled the verdict. The gallery beneath compares all four models on ten sentences, two per error mode.

The four models

The lexicon. VADER (Hutto & Gilbert 2014) scores text with a dictionary of words weighted by sentiment intensity plus a handful of context rules. Model A above follows that recipe with an original curation of about 280 valence entries in [-4, 4]: a negator within three tokens flips a word’s valence (multiplied by -0.74, VADER’s empirical flip factor), intensifiers add or subtract intensity (“very quiet” beats “quiet”), ALL-CAPS words and exclamation marks amplify, and a contrastive “but” reweights the clauses around it. The word scores sum to a total xx and normalize to a compound score x/x2+15x/\sqrt{x^2+15} in [−1,1][-1, 1]. No training data anywhere; every verdict is auditable down to the word, which is exactly the appeal of the approach.

The n-gram classifier. The coursework model vectorized reviews with TF-IDF over unigrams through trigrams, then fit a class-weighted logistic regression: with features x\mathbf{x} and learned coefficients β\boldsymbol\beta,

P(y=1∣x)=11+e−(β0+β⊤x).P(y = 1 \mid \mathbf{x}) = \frac{1}{1 + e^{-(\beta_0 + \boldsymbol\beta^\top \mathbf{x})}}.

Model B is the browser-sized version of the same idea: binary unigram and bigram features, stopwords removed but negators and amplifiers kept (the coursework preprocessing that mattered most), hashed into 2,048 dimensions with FNV-1a so the whole model is one weight vector instead of a vocabulary. The weights are trained offline by a script checked into this site’s repo and shipped as 6.5 KB of quantized JSON.

The ensemble. The coursework recipe blends the two probabilities, pens=α plr+(1−α) plexp_{\text{ens}} = \alpha\, p_{\text{lr}} + (1-\alpha)\, p_{\text{lex}} with α=0.78\alpha = 0.78 tuned for macro-F1: mostly trust the classifier, let the lexicon pull on the margins. DistilBERT, fine-tuned for three epochs, sits at the top of the table: contextual embeddings resolve most of what breaks the others, at roughly 32 times the inference cost of the logistic regression.

VADER lexicon (unsupervised)

accuracy 0.830
macro-F1 0.704

TF-IDF + logistic regression

accuracy 0.874
macro-F1 0.830

Weighted ensemble (α = 0.78)

accuracy 0.901
macro-F1 0.857

Fine-tuned DistilBERT

accuracy 0.923
macro-F1 0.881

Validation-split results from the CE807 coursework runs (n = 700, 79.6% positive), quoted as reported there. Accuracy flatters everyone: the all-positive baseline already scores 0.796, which is why macro-F1, which weighs the scarce negative class equally, is the honest column.

Plate II Accuracy and macro-F1 for the four models on the coursework validation split, quoted as reported there. The dashed line is the all-positive baseline; the distance beyond it, not the raw accuracy, is each model's real achievement.

The split was 79.6% positive, so accuracy starts at 0.796 for free and macro-F1 is the column to read: it averages the two classes equally, and the gap between VADER’s 0.830 accuracy and 0.704 macro-F1 is the sound of a model coasting on the majority class.

What each model misses, and why

The gallery in Plate I is the error analysis made interactive; every claim here is a row you can press.

Sarcasm defeats both live models at once, and for once the ensemble cannot help: “Great, now my fridge doubles as a waterfall” contains a genuinely positive word and no negative one either model recognizes, so the blend inherits the shared mistake. This is the structural weakness of averaging: an ensemble corrects disagreements, never consensus errors.

Negation splits cleanly in two. Local negation is easy for both: the lexicon flips “no puddle” to positive, and the bigram no_leaks was learnable. Scope is the killer. In “it was easy to install, but I can’t honestly say it ever worked”, the negator sits four tokens from the verb it negates: outside the lexicon’s three-token window, and beyond any bigram’s reach. Both live models call it positive; only the transformer, whose self-attention connects “can’t” to “worked” at any distance, reads the sentence right.

Multilingual text is a coverage question, not an intelligence question. The lexicon knows “encanta” because I put it there; the classifier learned “muy practico” because Spanish examples were in its training data. Neither generalizes an inch past its inventory, which is the difference between memorizing languages and modeling them.

Misspellings blind the two models differently. “This vacume is amazeing” contains no lexicon word at all, so Model A scores it a flat zero, and every hashed n-gram lands on untrained (or collided) weights, so Model B’s verdict is hash noise; it happens to guess wrong. Subword tokenization is the transformer’s quiet superpower here: “amaze” + “ing” still means something.

Idioms are the lexicon’s original sin: it prices words, never phrases. “Fired right up” nets negative because “fired” is negative, exactly the failure the coursework VADER showed; the classifier gets it right only because that exact bigram was in its training data. “A steal at this price” adds an ensemble lesson: the classifier lands on the correct verdict by the width of a rounding error (p=0.50p = 0.50), but the lexicon is so confidently wrong that the blend drags the final answer back below the line. A fixed α\alpha cannot know which component to believe today.

Rebuilding it for the web

Nothing from the coursework ships here: not the dataset, not the trained models. The lexicon is an original curation that borrows VADER’s mechanism, and the logistic regression is trained by scripts/train-sentiment-lr.mjs on a synthetic corpus the script itself generates: hand-written appliance-review predicates expanded through templates with a seeded PRNG, 3,260 labeled examples, 15% held out (93.7% held-out accuracy, a number that flatters the model precisely because the test sentences come from the same templates). The corpus is shaped deliberately: local negation patterns appear in training so bigrams can learn them; long-scope negation, misspellings, and unseen languages do not, so Model B fails on the gallery for the same structural reasons the coursework model failed on the validation set. The gallery’s ensemble column is the α\alpha-blend computed offline over the two live scorers; the DistilBERT column is recorded from the coursework model’s behavior on the validation cases the gallery sentences are rewritten from, since a 66-million-parameter transformer has no business in a page budgeted at 15 KB of JavaScript.

Which words did you lean on

Every model above has a verdict, and a verdict is worth roughly what your ability to interrogate it is worth. So the last figure asks all four the same question, which words did you lean on, and the first thing it shows is that the question does not survive contact with the four architectures intact. The lexicon can answer exactly, because answering is the same operation as deciding. The logistic regression can answer exactly at the level of its features and only approximately at the level of your words. The ensemble can answer only once you have decided what combining two answers means. The transformer cannot answer from the inside at all, and has to be interrogated from outside, by breaking its input and watching what changes. Four columns, four methods, each named in its own column head, because pretending they are one method would be the whole error this project was assigned to notice.

The lexicon column is the rule arithmetic itself: the base valence, then whichever of negation, boosting, capitalization and the contrastive “but” fired on that word, and the per-word numbers sum to the pre-normalization total with nothing left over. The classifier column is weight times feature value, which is exact per hashed slot, but getting from a slot to a word costs two approximations. A bigram belongs to two words at once, and distinct n-grams can share a slot. Both are split evenly, which is a choice and not a derivation, and every token whose weight came from a shared slot is flagged. The specimen labeled Collision contains a real one: leaks and extremely quiet both hash to slot 711, so the single weight stored there was pushed around during training by every occurrence of either. That is the deeper cost and it is mostly invisible, because the collisions a sentence can flag are the rare ones where both n-grams happen to be in front of you; enumerating the rest would need precisely the vocabulary the hashing trick exists to avoid storing.

The ensemble column carries the judgement call, so the judgement is printed rather than buried. Both members are additive in their own scores and neither is additive in probability, so each member’s exact contribution is passed through the local derivative of its own squash, the sigmoid for the classifier and the compound normalizer for the lexicon, and the two are combined with the same α=0.78\alpha = 0.78 the blend itself uses. That is a first-order approximation: the parts do not quite sum to the whole, and the figure prints the residual instead of quietly absorbing it. The transformer column is leave-one-out occlusion, one forward pass per word: delete the word, run the model again, take the change in P(positive)P(\text{positive}). An attention heat map would have been easier and would have been wrong, because attention is not an explanation (Jain & Wallace 2019): a high attention weight on a token is perfectly compatible with the output not depending on that token. Integrated gradients would be stronger than occlusion and was never on the table, because onnxruntime-web has no autograd. Occlusion needs nothing but forward passes, which is exactly why it works on a model you cannot open.

Specimen

A negator three tokens from what it negates: inside the lexicon window, and "not bad" is a bigram the classifier could have learned.

Signed word-importance contributions for one sentence under four models. Each column is shaded relative to its own largest magnitude and printed in its own units.
TokenLexiconexact rule arithmeticvalenceN-gram LRweight × feature, split across tokenslog-oddsEnsemblelinearized blend of its two membersprobabilityDistilBERTleave-one-out occlusion, recordedchange in P(positive)
+0.00+0.000+0.000−0.000
−1.40+0.052−0.031−0.000
+0.00−0.099−0.016+0.000
+0.00+0.000+0.000+0.004
+0.00−0.338−0.054+0.999
+1.63−0.019+0.043+0.998
+0.00+0.000+0.000+0.001
+0.00−0.520−0.084+0.001

Green is a push toward positive, ochre a push toward negative, and the shading is normalized within each column: it ranks tokens inside one model and means nothing across models, because valence, log-odds and probability are not the same unit. The printed numbers are each in that model's own units. A ◦ marks a token whose weight comes from a hashed slot shared with another n-gram.

Word 5 of 8 · not

Lexicon

not in the lexicon, so it contributes exactly 0. A negator or a booster contributes nothing on its own either: it is an operator on other words, never a term.

N-gram LR

not −0.306 · level not −0.033 (bigram, half of it here) · not bad +0.000 (bigram, half of it here) · −0.338 log-odds

Ensemble

0.78 × 0.206 × (−0.338) + 0.22 × 0.1284 × (+0.00) = −0.0544 of probability

DistilBERT

P(positive) = 0.9988 with the word, 0.0002 without it · +0.9986

Lexicon exact

Sum of word valences
+0.23
Total
+0.23
Compound
+0.23 / √(0.05 + 15) = +0.06

N-gram LR exact per slot

Bias
+0.031
Sum of token contributions
−0.925
Log-odds z
−0.894
P(positive)
σ(z) = 0.290
Hashing
9 n-grams into 2048 slots · no two share a slot here

Ensemble first order

Blend
0.78 · p(LR) + 0.22 · p(lexicon)
Local slopes
dp/dz = 0.206 · dp/dt = 0.1284
Empty-sentence baseline
0.506
Explained by the tokens
−0.142
Residual (curvature, not a token)
−0.021
P(positive)
0.343

DistilBERT no additivity

P(positive), whole sentence
0.9988
Sum of the deltas
+2.002
Why they do not reconcile
Occlusion measures what each word is necessary for, one deletion at a time. Two words can each be necessary for the whole verdict, so the deltas can sum past 1, and nothing requires them to sum to anything at all.
Plate III The same sentence under four attribution methods, one per model, because the four models do not admit the same question. Read down a column rather than across a row: the shading is normalized within each column and the four sets of numbers are in four different units (valence, log-odds, probability, and a change in probability), so only sign and rank travel between columns. Click any word for the arithmetic behind its four numbers. Start with the Negation specimen and the word “not”: the lexicon scores it exactly zero, correctly, because a negator is an operator and not a term, while deleting it moves the transformer by 0.999 of probability.

The transformer’s column is recorded rather than live, and the reason is a fact about deployment rather than about NLP. The smallest usable ONNX export of a fine-tuned DistilBERT is 67 MB, and Cloudflare Pages, which serves this site, rejects any single asset over 25 MiB. There is no version of this page in which the transformer runs. So scripts/occlude-distilbert.mjs runs it offline over the seven specimens, one forward pass per word plus one for each whole sentence, and checks in the sixty-odd numbers that fall out; type anything else and that column goes blank, which is the constraint made visible rather than a bug. The model it runs is the public distilbert-base-uncased-finetuned-sst-2-english checkpoint, not the coursework fine-tune, which is not mine to republish. It is the same architecture standing in for it, and it is emphatically not the same model: it reads “Great, now my fridge doubles as a waterfall” as positive, where the coursework model in Plate I read it correctly. A stand-in that agreed with everything would be a worse stand-in.

Two things fall out of reading the columns beside each other. The first is a scale warning that is easy to skip and wrong to skip: valence, log-odds, probability and a probability delta are four different units, and stacking them on one axis would produce a picture with no meaning at all. Hence normalization within each column and the raw number in every cell. The second is the answer to the question the project started with. Look again at “not” in the Negation specimen. The lexicon scores it zero and is right to: a negator has no valence of its own, it is an operator on the words near it. Occlusion scores it +0.999+0.999, because deleting it inverts the sentence. Both numbers are correct, they are not measurements of the same quantity, and no amount of care will make them commensurable. The lexicon can tell you everything it did and is often wrong; the transformer is usually right and can only be questioned from outside, one deletion at a time. That is not a single axis with interpretability at one end and accuracy at the other. It is two different kinds of knowledge about a sentence, and the conclusion of a model zoo is that you choose one knowing which kind you are buying.

What the zoo taught

  • The live classifier is a demonstration, not a benchmark. Trained on synthetic templates, it proves the shape of n-gram behavior (local patterns learnable, scope invisible, out-of-vocabulary chaos), and its 93.7% means nothing beyond its own little distribution. The benchmark numbers are in Plate II, and they belong to the coursework models.
  • Hashing costs interpretability exactly when you want it. The coursework model could name its coefficients (“jammed”, β=−3.4\beta = -3.4). With 2,048 hashed slots, collisions mean a chip in the plate occasionally credits an n-gram for a weight it merely shares. A vocabulary would fix it at the price of shipping one.
  • The ensemble’s failure modes were the best teaching material. I expected the blend to be a boring average; the gallery shows it inheriting consensus errors and overruling a correct component. If I extended this, a context-dependent gate (even “trust the lexicon less when it scored only one word”) would beat the static 0.78.
  • What the transformer buys is scope. Every gallery row the recorded DistilBERT column wins over both live models involves a dependency longer than a bigram. That is a cleaner summary of a decade of NLP than any leaderboard: the architecture that can look four tokens away wins the sentences where meaning lives four tokens away.

Keys on this page

A figure's keys are its own: they answer once the figure has focus, so tab to it or click it first. Esc closes this sheet.