Case Study · NLP · ML
Sentiment, Four Ways
The idea
During my study-abroad term at the University of Essex I took CE807, the text-analytics module, and its final assignment was a small model zoo: classify appliance reviews as positive or negative four different ways, then explain not just who wins but why each model fails the way it does. The four were VADER, a rule-based lexicon scorer that needs no training data at all; TF-IDF with logistic regression, the classic discriminative baseline; a weighted ensemble of the two; and a fine-tuned DistilBERT as the state-of-the-art ceiling. The dataset was graded coursework material, so it stays where it is, and the numbers below are quoted as coursework results. But the lesson of the assignment deserved better than a PDF: the interesting part was never the leaderboard, it was the error analysis. Sarcasm, negation scope, Spanish reviews, creative spelling, idioms: each error mode defeats a different model for a different structural reason.
So this page rebuilds the comparison as something you can poke. The two models that are small enough to run in a browser really do run in your browser, live, on whatever you type; the two that are not are recorded on a fixed gallery of hand-written trick sentences, one pair per error mode. Everything is self-hosted and nothing you type leaves the page.
Model A · Lexicon + rules live
positive compound +0.64
scored words
- puddle +1.4flipped
- quiet +1.8
Model B · Hashed n-gram LR live
positive p(positive) = 0.76
strongest n-grams
- no +0.30
- runs +0.26
- quiet +0.24
- no puddle +0.18
- whisper quiet +0.13
- whisper +0.08
The error-mode gallery · all four models
| Sentence | Mode | Truth | Lexicon | N-gram LR | Ensemble | DistilBERT | Load into the input |
|---|---|---|---|---|---|---|---|
| Great, now my fridge doubles as a waterfall. | sarcasm | neg | pos | pos | pos | neg | |
| Oh wonderful, the oven quit the night before Thanksgiving. | sarcasm | neg | pos | pos | pos | neg | |
| No leaks, no noise, no problems. | negation | pos | pos | pos | pos | pos | |
| It was easy to install, but I can't honestly say it ever worked. | negation | neg | pos | pos | pos | neg | |
| Me encanta, es muy practico. | multilingual | pos | pos | pos | pos | pos | |
| El dryer nunca funciono bien. | multilingual | neg | neg | neg | neg | neg | |
| This vacume is amazeing, buyed a second one. | misspelling | pos | neut | neg | neg | pos | |
| Absolutly hopless. Retunred it the next day. | misspelling | neg | neut | neg | neg | neg | |
| Fired right up on the first try. | idiom | pos | neg | pos | pos | pos | |
| Honestly a steal at this price. | idiom | pos | neg | pos | neg | pos |
The Lexicon and N-gram LR columns are the two live models above, scored at build time; press try to hand any row to them yourself. The Ensemble column is the coursework recipe, p = 0.78 pLR + 0.22 plex, computed offline over the two live scorers by the training script. The DistilBERT column is recorded, not run: a fine-tuned transformer is far too heavy for a web page, so these verdicts follow the behavior of the coursework model on the validation cases these sentences are rewritten from. Green means the verdict matches the intended sentiment; ochre means it misses (a neutral verdict counts as a miss).
The four models
The lexicon. VADER (Hutto & Gilbert 2014) scores text with a dictionary of words weighted by sentiment intensity plus a handful of context rules. Model A above follows that recipe with an original curation of about 280 valence entries in [-4, 4]: a negator within three tokens flips a word’s valence (multiplied by -0.74, VADER’s empirical flip factor), intensifiers add or subtract intensity (“very quiet” beats “quiet”), ALL-CAPS words and exclamation marks amplify, and a contrastive “but” reweights the clauses around it. The word scores sum to a total and normalize to a compound score in . No training data anywhere; every verdict is auditable down to the word, which is exactly the appeal of the approach.
The n-gram classifier. The coursework model vectorized reviews with TF-IDF over unigrams through trigrams, then fit a class-weighted logistic regression: with features and learned coefficients ,
Model B is the browser-sized version of the same idea: binary unigram and bigram features, stopwords removed but negators and amplifiers kept (the coursework preprocessing that mattered most), hashed into 2,048 dimensions with FNV-1a so the whole model is one weight vector instead of a vocabulary. The weights are trained offline by a script checked into this site’s repo and shipped as 6.5 KB of quantized JSON.
The ensemble. The coursework recipe blends the two probabilities, with tuned for macro-F1: mostly trust the classifier, let the lexicon pull on the margins. DistilBERT, fine-tuned for three epochs, sits at the top of the table: contextual embeddings resolve most of what breaks the others, at roughly 32 times the inference cost of the logistic regression.
VADER lexicon (unsupervised)
TF-IDF + logistic regression
Weighted ensemble (α = 0.78)
Fine-tuned DistilBERT
Validation-split results from the CE807 coursework runs (n = 700, 79.6% positive), quoted as reported there. Accuracy flatters everyone: the all-positive baseline already scores 0.796, which is why macro-F1, which weighs the scarce negative class equally, is the honest column.
The split was 79.6% positive, so accuracy starts at 0.796 for free and macro-F1 is the column to read: it averages the two classes equally, and the gap between VADER’s 0.830 accuracy and 0.704 macro-F1 is the sound of a model coasting on the majority class.
What each model misses, and why
The gallery in Plate I is the error analysis made interactive; every claim here is a row you can press.
Sarcasm defeats both live models at once, and for once the ensemble cannot help: “Great, now my fridge doubles as a waterfall” contains a genuinely positive word and no negative one either model recognizes, so the blend inherits the shared mistake. This is the structural weakness of averaging: an ensemble corrects disagreements, never consensus errors.
Negation splits cleanly in two. Local negation is easy for both: the lexicon flips
“no puddle” to positive, and the bigram no_leaks was learnable. Scope is the killer.
In “it was easy to install, but I can’t honestly say it ever worked”, the negator sits
four tokens from the verb it negates: outside the lexicon’s three-token window, and
beyond any bigram’s reach. Both live models call it positive; only the transformer, whose
self-attention connects “can’t” to “worked” at any distance, reads the sentence right.
Multilingual text is a coverage question, not an intelligence question. The lexicon knows “encanta” because I put it there; the classifier learned “muy practico” because Spanish examples were in its training data. Neither generalizes an inch past its inventory, which is the difference between memorizing languages and modeling them.
Misspellings blind the two models differently. “This vacume is amazeing” contains no lexicon word at all, so Model A scores it a flat zero, and every hashed n-gram lands on untrained (or collided) weights, so Model B’s verdict is hash noise; it happens to guess wrong. Subword tokenization is the transformer’s quiet superpower here: “amaze” + “ing” still means something.
Idioms are the lexicon’s original sin: it prices words, never phrases. “Fired right up” nets negative because “fired” is negative, exactly the failure the coursework VADER showed; the classifier gets it right only because that exact bigram was in its training data. “A steal at this price” adds an ensemble lesson: the classifier lands on the correct verdict by the width of a rounding error (), but the lexicon is so confidently wrong that the blend drags the final answer back below the line. A fixed cannot know which component to believe today.
Rebuilding it for the web
Nothing from the coursework ships here: not the dataset, not the trained models. The
lexicon is an original curation that borrows VADER’s mechanism, and the logistic
regression is trained by scripts/train-sentiment-lr.mjs on a synthetic corpus the
script itself generates: hand-written appliance-review predicates expanded through
templates with a seeded PRNG, 3,260 labeled examples, 15% held out (93.7% held-out
accuracy, a number that flatters the model precisely because the test sentences come
from the same templates). The corpus is shaped deliberately: local negation patterns
appear in training so bigrams can learn them; long-scope negation, misspellings, and
unseen languages do not, so Model B fails on the gallery for the same structural reasons
the coursework model failed on the validation set. The gallery’s ensemble column is the
-blend computed offline over the two live scorers; the DistilBERT column is
recorded from the coursework model’s behavior on the validation cases the gallery
sentences are rewritten from, since a 66-million-parameter transformer has no business
in a page budgeted at 15 KB of JavaScript.
Which words did you lean on
Every model above has a verdict, and a verdict is worth roughly what your ability to interrogate it is worth. So the last figure asks all four the same question, which words did you lean on, and the first thing it shows is that the question does not survive contact with the four architectures intact. The lexicon can answer exactly, because answering is the same operation as deciding. The logistic regression can answer exactly at the level of its features and only approximately at the level of your words. The ensemble can answer only once you have decided what combining two answers means. The transformer cannot answer from the inside at all, and has to be interrogated from outside, by breaking its input and watching what changes. Four columns, four methods, each named in its own column head, because pretending they are one method would be the whole error this project was assigned to notice.
The lexicon column is the rule arithmetic itself: the base valence, then whichever of negation,
boosting, capitalization and the contrastive “but” fired on that word, and the per-word numbers sum
to the pre-normalization total with nothing left over. The classifier column is weight times feature
value, which is exact per hashed slot, but getting from a slot to a word costs two approximations.
A bigram belongs to two words at once, and distinct n-grams can share a slot. Both are split evenly,
which is a choice and not a derivation, and every token whose weight came from a shared slot is
flagged. The specimen labeled Collision contains a real one: leaks and extremely quiet both hash
to slot 711, so the single weight stored there was pushed around during training by every occurrence
of either. That is the deeper cost and it is mostly invisible, because the collisions a sentence can
flag are the rare ones where both n-grams happen to be in front of you; enumerating the rest would
need precisely the vocabulary the hashing trick exists to avoid storing.
The ensemble column carries the judgement call, so the judgement is printed rather than buried. Both members are additive in their own scores and neither is additive in probability, so each member’s exact contribution is passed through the local derivative of its own squash, the sigmoid for the classifier and the compound normalizer for the lexicon, and the two are combined with the same the blend itself uses. That is a first-order approximation: the parts do not quite sum to the whole, and the figure prints the residual instead of quietly absorbing it. The transformer column is leave-one-out occlusion, one forward pass per word: delete the word, run the model again, take the change in . An attention heat map would have been easier and would have been wrong, because attention is not an explanation (Jain & Wallace 2019): a high attention weight on a token is perfectly compatible with the output not depending on that token. Integrated gradients would be stronger than occlusion and was never on the table, because onnxruntime-web has no autograd. Occlusion needs nothing but forward passes, which is exactly why it works on a model you cannot open.
A negator three tokens from what it negates: inside the lexicon window, and "not bad" is a bigram the classifier could have learned.
| Token | Lexiconexact rule arithmeticvalence | N-gram LRweight × feature, split across tokenslog-odds | Ensemblelinearized blend of its two membersprobability | DistilBERTleave-one-out occlusion, recordedchange in P(positive) |
|---|---|---|---|---|
| +0.00 | +0.000 | +0.000 | −0.000 | |
| −1.40 | +0.052 | −0.031 | −0.000 | |
| +0.00 | −0.099 | −0.016 | +0.000 | |
| +0.00 | +0.000 | +0.000 | +0.004 | |
| +0.00 | −0.338 | −0.054 | +0.999 | |
| +1.63 | −0.019 | +0.043 | +0.998 | |
| +0.00 | +0.000 | +0.000 | +0.001 | |
| +0.00 | −0.520 | −0.084 | +0.001 |
Green is a push toward positive, ochre a push toward negative, and the shading is normalized within each column: it ranks tokens inside one model and means nothing across models, because valence, log-odds and probability are not the same unit. The printed numbers are each in that model's own units. A ◦ marks a token whose weight comes from a hashed slot shared with another n-gram.
Word 5 of 8 · not
Lexicon
not in the lexicon, so it contributes exactly 0. A negator or a booster contributes nothing on its own either: it is an operator on other words, never a term.
N-gram LR
not −0.306 · level not −0.033 (bigram, half of it here) · not bad +0.000 (bigram, half of it here) · −0.338 log-odds
Ensemble
0.78 × 0.206 × (−0.338) + 0.22 × 0.1284 × (+0.00) = −0.0544 of probability
DistilBERT
P(positive) = 0.9988 with the word, 0.0002 without it · +0.9986
Lexicon exact
- Sum of word valences
- +0.23
- Total
- +0.23
- Compound
- +0.23 / √(0.05 + 15) = +0.06
N-gram LR exact per slot
- Bias
- +0.031
- Sum of token contributions
- −0.925
- Log-odds z
- −0.894
- P(positive)
- σ(z) = 0.290
- Hashing
- 9 n-grams into 2048 slots · no two share a slot here
Ensemble first order
- Blend
- 0.78 · p(LR) + 0.22 · p(lexicon)
- Local slopes
- dp/dz = 0.206 · dp/dt = 0.1284
- Empty-sentence baseline
- 0.506
- Explained by the tokens
- −0.142
- Residual (curvature, not a token)
- −0.021
- P(positive)
- 0.343
DistilBERT no additivity
- P(positive), whole sentence
- 0.9988
- Sum of the deltas
- +2.002
- Why they do not reconcile
- Occlusion measures what each word is necessary for, one deletion at a time. Two words can each be necessary for the whole verdict, so the deltas can sum past 1, and nothing requires them to sum to anything at all.
The transformer’s column is recorded rather than live, and the reason is a fact about deployment
rather than about NLP. The smallest usable ONNX export of a fine-tuned DistilBERT is 67 MB, and
Cloudflare Pages, which serves this site, rejects any single asset over 25 MiB. There is no version
of this page in which the transformer runs. So scripts/occlude-distilbert.mjs runs it offline over
the seven specimens, one forward pass per word plus one for each whole sentence, and checks in the
sixty-odd numbers that fall out; type anything else and that column goes blank, which is the
constraint made visible rather than a bug. The model
it runs is the public distilbert-base-uncased-finetuned-sst-2-english checkpoint, not the
coursework fine-tune, which is not mine to republish. It is the same architecture standing in for
it, and it is emphatically not the same model: it reads “Great, now my fridge doubles as a
waterfall” as positive, where the coursework model in Plate I read it correctly. A stand-in that
agreed with everything would be a worse stand-in.
Two things fall out of reading the columns beside each other. The first is a scale warning that is easy to skip and wrong to skip: valence, log-odds, probability and a probability delta are four different units, and stacking them on one axis would produce a picture with no meaning at all. Hence normalization within each column and the raw number in every cell. The second is the answer to the question the project started with. Look again at “not” in the Negation specimen. The lexicon scores it zero and is right to: a negator has no valence of its own, it is an operator on the words near it. Occlusion scores it , because deleting it inverts the sentence. Both numbers are correct, they are not measurements of the same quantity, and no amount of care will make them commensurable. The lexicon can tell you everything it did and is often wrong; the transformer is usually right and can only be questioned from outside, one deletion at a time. That is not a single axis with interpretability at one end and accuracy at the other. It is two different kinds of knowledge about a sentence, and the conclusion of a model zoo is that you choose one knowing which kind you are buying.
What the zoo taught
- The live classifier is a demonstration, not a benchmark. Trained on synthetic templates, it proves the shape of n-gram behavior (local patterns learnable, scope invisible, out-of-vocabulary chaos), and its 93.7% means nothing beyond its own little distribution. The benchmark numbers are in Plate II, and they belong to the coursework models.
- Hashing costs interpretability exactly when you want it. The coursework model could name its coefficients (“jammed”, ). With 2,048 hashed slots, collisions mean a chip in the plate occasionally credits an n-gram for a weight it merely shares. A vocabulary would fix it at the price of shipping one.
- The ensemble’s failure modes were the best teaching material. I expected the blend to be a boring average; the gallery shows it inheriting consensus errors and overruling a correct component. If I extended this, a context-dependent gate (even “trust the lexicon less when it scored only one word”) would beat the static 0.78.
- What the transformer buys is scope. Every gallery row the recorded DistilBERT column wins over both live models involves a dependency longer than a bigram. That is a cleaner summary of a decade of NLP than any leaderboard: the architecture that can look four tokens away wins the sentences where meaning lives four tokens away.