Late September, at TeknSEO in Toulouse, I spent 80 minutes explaining that a small model trained on your own data often does better, faster and cheaper than a big model you ask questions to. Early November, I read Dejan’s post about his seven-level sentiment classifier. Three days later, an open dataset and two models in six languages were online. And you can pitch in: I’m building a hand-labeled reference set (a “gold” dataset), and your help would be very welcome (this way). Here is how it came together.
Where it started: a workshop on small models
The workshop was called “Google classifies everything, even you. Do the same.” The starting idea: Google sorts every query, every page, every site into boxes, and an SEO has every reason to be able to do the same on their own data. In the meantime Jev came out, the zero-shot classifier that answers with probabilities, and for a while I thought I’d have to redo the whole workshop. Digging in, the argument held up, even better than before: the engine is within everyone’s reach, so the edge comes from the rules you give it and the dataset you check it against.
Three lessons from the workshop carried over almost as they were:
- synthetic data flatters itself. On French queries generated by an LLM, my classifier showed 98% in cross-validation. On 22,354 real queries, 91.7%. A score on generated text mostly measures how regular the generator is;
- the silver recipe: real text, labeled in bulk by an LLM, then synthetic text only to fill the gaps;
- one encoder, small heads. An SVM on embeddings reached 91.7% in 31 seconds of CPU. For N tasks, you deploy one encoder and N heads of a few KB, instead of N full models.
Big unease in the room when I showed tests some people had proudly posted online, sorting their emails or their support tickets. Yes: user data, with sensitive and personal details, sent without a second thought to a server in the US or in China. All good.
Dejan’s post
A few days ago, Dejan (Dan Petrovic) released a seven-level sentiment classifier, from -3 to +3. It clicked right away: classifying text with small models is my playground of the moment.
From his post, I adopted three ideas on the spot:
- the seven-level scale. The positive / neutral / negative trio of the classic datasets flattens everything an SEO cares about: polite reservation and cold anger end up in the same box;
- labeling in context: each sentence is scored by an LLM that has the whole text in front of it. “They did it again” reads very differently depending on whether it’s about a rock band or a spammer;
- the 26 formats of web text: product page, review, forum thread, press release, FAQ… The sentiment of a product page and that of a forum thread aren’t measured with the same yardstick.
His model is trained on synthetic English text. I wanted to see what the same idea gives when fed with real web text, in six languages, using the workshop recipe.
Real web text, and a little synthetic
I drew passages at random from FineWeb (English) and FineWeb-2 (French, Spanish, German, Italian, Portuguese), then had Claude Sonnet score every sentence, with the full passage in view and the seven-level grid. The result: 452,410 sentences from 27,409 passages, 86% of them real web text.
A trap along the way: although it’s supposed to be “cleaner” than CommonCrawl, FineWeb still contains unspeakable junk. Always filter if you build anything public on it, it can be scary!
Real web text has another flaw: it’s lukewarm. About half the sentences are neutral, and -3 makes up less than 1% of the lot. To find very negative sentences, I first tried having them written. A partial miss: when I ask Claude for a “very negative” text and then have it scored sentence by sentence, only 11% of the sentences come back as -3, and half as -2. The same model that writes “very negative” reads its own text back as merely negative :).
A learned preselection worked better. A small logistic regression,
trained on my first labels, predicts “this passage contains a -3”. Among the
passages it picks, 21% of the sentences are scored -3, against less than
1% at random. The synthetic part stayed, marked in a source column, for
formats the web supplies poorly (customer emails, ad scripts, support
replies). For training the model it added nothing on natural web text, and
I left it out.
A frozen encoder and a 119 KB head
I tested the route Dan took: fine-tuning a full small LLM. With the same data, it did no better than a frozen encoder with a logistic regression on top, and it varied much more from one random seed to the next. So I kept the workshop solution: EmbeddingGemma-300m, left untouched, and a linear seven-class head. The head fits in a 119 KB JSON file: 7 × 768 weights, 7 biases and a mean vector.
An analogy I like: the encoder is a librarian who shelves each sentence according to what it’s about, and the head is a ruler you lay on the shelf to read the sentiment. You can lay other rulers on the same shelves (topic, intent, format) without hiring another librarian.
On 9 human-labeled test sets, in six languages, the head reaches 0.673 macro-F1 on three classes. The LLM that labeled the data scores 0.726 on the same sets: a model that learns its labels has little chance of beating them, and the head gets within 5 points.
Another setting that matters: centering. You subtract the mean from the vectors before applying the head. Centering on the mean of your own corpus gives 0.673, centering on the reference mean shipped with the head gives 0.641. Three points for one subtraction, that’s a bargain!
The missing prefix
EmbeddingGemma expects a prefix before the text depending on the task, here
task: classification | query: . That was already a workshop slide: not all
prefixes are equal. Except that several tools that produce embeddings don’t
let you add a free prefix. A head trained with the prefix and fed vectors
without it loses 3 to 10 points depending on the test set.
Hence a second head, trained on vectors without the prefix: it recovers most of it (0.662). In the browser, with the encoder quantized to ONNX (309 MB in q8, 197 MB in q4), the head loses 0.3 and 0.8 points, and the sentences never leave the machine.
Reading the sentence with its page
The labeler reads each sentence with the whole passage. The model only sees the sentence. So some labels depend on information the model never gets: who “they” are, or the ironic setup two sentences earlier.
The simplest attempt: give the head two vectors, the sentence’s and the full passage’s, end to end. On natural web text, at seven levels, macro-F1 goes from 0.539 to 0.565. To check that the gain comes from the context and not from the extra dimensions, I trained the same head with the passage of another page, drawn at random in the same language: 0.534, no gain. The context does the work, mostly on the extreme levels. It became a separate model, experimental, also freely available.
The V-shaped galaxy
For the project page, I wanted to show what sentiment looks like in the encoder’s space. I projected 20,000 sentences the model had never seen onto the three axes that best separate the seven levels (a linear discriminant analysis). The result has a V shape: neutral at the tip, one arm for positive, one arm for negative, and intensity growing as you go up the arms. The first axis carries polarity (0.77 correlation with the score), the second carries intensity.
It’s the image at the top of this post. The interactive galaxy, in 2D or 3D, to spin it around.
The thing that blew me away looking at this visual, fresh out of the specs? To me it works like an optical illusion. In 2D mode, although nothing really moves, I see the whole cloud of points “spin” if I stare into space without focusing. Does it do that to you too?
Still hard: irony and mixed sentences
I had the 10,708 sentences of the test sets tagged by linguistic phenomenon (irony, negation, contrast, comparison…) to see where the errors cluster. Irony and mixed sentences, one part positive and one part negative, multiply by about 2.5 the risk of flipping the polarity. That holds for every system tested, the labeler included. My head has one more weakness: negation, contrast (“but”, “however”) and comparisons raise its flips by 1.4 to 1.9 times. A frozen encoder probably grasps the scope of small logical words less well; I haven’t tested that yet. The goal here was a small, fast model that runs in a browser, rather than the best possible quality.
The bill
About $112 of API for the whole project: format classification, synthetic data, labeling, two rounds of collection. Encoding and training ran on my GPUs. Synthetic data costs about half a cent per passage, generation and labeling included. Real web text costs less and gives better models.
Online since November 4
- WebSentiment ML-7, the dataset: 452,410 sentences, 6 languages, ODC-By licence, DOI 10.5281/zenodo.23964284.
- websentiment-egemma-ml-7, the main model, with and without prefix, in Python or in the browser, CC BY 4.0 licence, DOI 10.5281/zenodo.23981493.
- websentiment-egemma-ml-7-ctx, the context model, experimental, DOI 10.5281/zenodo.23982301.
- The demo: paste a text, each sentence is scored and highlighted in green or red, alone or in the context of the whole text: https://huggingface.co/spaces/SDeaure/websentiment-demo
Everything is described, with the licences and how to credit, at https://sylvain-deaure.fr/en/websentiment/. The scale, labeling in context and the formats come from Dejan’s work, and I thank him for it: it’s the kind of post that makes you want to open a terminal.
A hand with the human gold set
No sentence in the dataset was scored by a human. The labeler’s agreement with human annotators is measured on existing research sets, but those are mostly tweets and film reviews. What’s missing is a human reference set on real web text, in all six languages.
If you read one of the six languages and have a moment, you can score a few sentences here: https://sdeaure-websentiment-gold.hf.space. You sign in with your Hugging Face account, each sentence shows up with its passage, you pick a level from -3 to +3, and on to the next one. Those who wish will be thanked by name in the card of the final set.
Further reading
- Dejan’s post that started it all: Sentiment model.
- Johan Puisais’s write-up of the TeknSEO workshop (in French): JEV et SEO : les limites que Sylvain Deauré rappelle aux SD Days 2026.
This page was translated from French to English with the help of AI. Read the original in French.