Ian Kellar.

Behavioural and implementation science

ian@k3llar.com
ORCID
Bluesky
GitHub

Writing

LLM sentiment analysis: the Medium post and the evidence behind it

29 October 2026

Part A. The post, summarised

The claim

The post argues that large language models handle sentiment analysis far better than earlier methods, but that their reliability is not established, so they need a “crash test”.[1] Traditional algorithms fall short on irony, sarcasm and similar features, and the opacity of LLMs leaves performance guarantees and confidence bounds thin, so that they “work, but it’s extremely complicated to understand why, how, and how well”.[1] The post cites a market estimate of about USD 3.15 billion for automated sentiment analysis in 2021, growing at 14.4 per cent a year.[1]

The design

The author and a co-author built, in the post’s words, a “small, but controlled and curated battery of scenarios, deliberately tricky”, translated into ten languages: English, Italian, German, Japanese, Russian, Chinese, French, Spanish, Polish and Portuguese.[1] The preprint behind the post specifies 20 scenarios.[2] A pool of international raters scored each scenario from 1 (very bad) to 10 (very good); the preprint reports 62 volunteers segmented by native language and blind to the model results.[1][2] The same prompts then went to ChatGPT 3.5 and 4, LLaMA2 7b and Gemini Pro, using default settings and repeated runs.[1][2]

The findings the post reports

For “many scenarios and most languages”, the average discrepancy between model and human rating is reported as below 0.5 point.[1] The post also gives specific absurd answers.[1] LLaMA2 rated the opening scenario, “At the restaurant the food was good but the waiter shot someone with a gun”, as 8 out of 10.[1] It rated a Polish rendering of “The food was good, but I spent the whole night in the bathroom” as 8 out of 10, and read the sarcastic “Great idea to go to Thailand in the monsoon season” as enthusiasm.[1] One scenario about whitewater rafting drew 3 from Gemini in Russian and 7 from ChatGPT in Italian.[1] The post reports that LLaMA2 never scored a scenario below 7, that Gemini censored some prompts unevenly by language with Spanish the most censored and French the least, and that the same model could score the same scenario differently across languages or across versions.[1]

What the author concludes

Proverbio concludes that nuanced, ironic and sarcastic text is now handled much more effectively than before, that this partly justifies the hype around the tools, but that “getting better” does not mean “being wonderfully good”.[1] He recommends “crash tests” for LLMs, developed and maintained alongside public agencies, and asks what error rate users are willing to accept.[1]

Appraisal

What the post reports as measured and what it asserts as opinion are separable. The divergence figures, the quoted model answers, the language-dependent failures and the censoring pattern are observations from the authors’ own experiment, and the preprint sets out the design behind them.[1][2] The claim that the hype is “somehow justified”, the crash-test agency proposal, and the framing of a “leap forward” are the author’s interpretation rather than findings.[1]

The post is also a personal blog piece promoting the author’s own preprint, posted within weeks of it appearing, and it links straight to the manuscript.[1][2] The scenario set is small and curated: 20 hand-written items chosen by the authors to be ambiguous, and the post names only a handful of them.[1][2] A set of that size can demonstrate that a failure mode exists and can illustrate it with concrete examples. It cannot estimate a failure rate, and it cannot show that the reported divergence holds for uncurated text. The preprint itself states that the work “is limited in the number of examples provided”, and that the human questionnaires may carry the authors’ own filter bubble bias.[2]

Part B. The background literature

Twenty-six sources, weighted to peer-reviewed work, with preprints marked as preprints. Every one below was checked by fetching the DOI, arXiv abstract or publisher page.

B1. Can an LLM stand in for a human annotator on a subjective text task?

  • Gilardi, Alizadeh and Kubli (2023), “ChatGPT outperforms crowd workers for text-annotation tasks” PNAS 120(30) e2305016120. link. Zero-shot ChatGPT beat crowd workers by about 25 percentage points on average across four datasets, with higher intercoder agreement and roughly thirty times lower cost. [3]
  • Törnberg (2023), “ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning” arXiv:2304.06588 (preprint). link. GPT-4 classified the political affiliation of US politicians’ tweets with higher accuracy and reliability and equal or lower bias than expert and crowd annotators. [4]
  • Ziems, Held, Shaikh, Chen, Zhang and Yang (2024), “Can Large Language Models Transform Computational Social Science?” Computational Linguistics. link. A peer-reviewed framing of what LLMs can and cannot do for social-science text tasks, including annotation, with explicit caution about validation. [5]
  • Bermejo, Gago, Gálvez and Harari (2025), “LLMs outperform outsourced human coders on complex textual analysis” Scientific Reports 15:40122. link. On 210 Spanish news articles and five coding tasks, four LLMs outperformed outsourced human coders on every metric when measured against expert labels. [6]
  • Rønningstad, Velldal and Øvrelid (2024), “A GPT among Annotators: LLM-based Entity-Level Sentiment Annotation” Proceedings of LAW-XVIII. link. The clearest negative result in this set: the best GPT-generated labels reached Cohen’s kappa 0.425 against curated gold, below the level of every one of five trained human annotators. [7]
  • Calderon, Reichart and Dror (2025), “The Alternative Annotator Test for LLM-as-a-Judge” ACL 2025. link. Proposes a leave-one-out statistical test for whether an LLM can replace a human annotator; closed models such as GPT-4o passed on some datasets, open models generally did not, and success was task- and prompt-dependent. [8]
  • Nakamura, Tan and Yean (2026), “Observational Equivalence of LLM and Human Annotation” arXiv:2609.22133 (preprint). link. Replicating 14 political-science coding tasks with ten LLMs, three experts and 165 crowd workers, they found recent LLMs agreed with experts about as closely as experts agreed with each other, and that disagreement tracked item ambiguity. [9]
  • Pangakis, Wolken and Fasching (2023), “Automated Annotation with Generative AI Requires Validation” arXiv:2306.00176 (preprint). link. Replicating 27 annotation tasks, they argue that LLM annotation must be validated against human labels task by task, because performance is highly contingent on the dataset and the type of task. [10]
  • Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber and Hovy (2025), “Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation” arXiv:2509.08825 (preprint). link. Shows that annotation can look acceptable on standard quality metrics while downstream statistical conclusions are corrupted, and that passing an alt-test does not guarantee valid downstream inference. [11]
  • Santurkar, Durmus, Ladhak, Lee, Liang and Hashimoto (2023), “Whose Opinions Do Language Models Reflect?” ICML 2023. link. Documents annotator-population skew: a model’s opinion distribution tracks some demographic groups much more closely than others, so a model is not a neutral panel of annotators. [12]
  • Durmus, Nguyen, Liao and colleagues (2023), “Towards Measuring the Representation of Subjective Global Opinions in Language Models” arXiv:2306.16388 (preprint). link. Builds a global-opinion benchmark and finds that model answers align better with the survey responses of some countries than others. [13]
  • Sclar, Choi, Tsvetkov and Suhr (2024), “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design” ICLR 2024. link. Prompt-formatting changes alone moved accuracy by up to 76 points on one model; the authors argue that performance should be reported as a range over plausible prompt formats, not a single format. [14]
  • Zhao, Wallace, Feng, Klein and Singh (2021), “Calibrate Before Use: Improving Few-Shot Performance of Language Models” ICML 2021. link. Shows that label-set order and format anchor predictions strongly, and that the skew can be measured and corrected. [15]
  • Atil and colleagues (2024), “Non-Determinism of ‘Deterministic’ LLM Settings” arXiv:2408.04667 (preprint). link. Even with temperature set to zero, repeated runs can return different answers, so run-to-run variance has to be estimated rather than assumed away. [16]

B2. Sarcasm, irony, negation and double negation

  • Zhang, Zou, Lian, Tiwari and Qin (2024), “SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding” arXiv:2408.11319 (preprint). link. Benchmarks eleven LLMs against supervised baselines on six sarcasm datasets; current LLMs underperform supervised pre-trained models on several of them, although GPT-4 Turbo leads among LLMs. [17]
  • Abu Farha, Oprea, Wilson and Magdy (2022), “SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic” SemEval-2022. link. The first shared task using the text authors’ own sarcasm labels; the top system reached F1 0.563 on sarcasm detection, which shows how hard intended sarcasm is. [18]
  • Van Hee, Lefever and Hoste (2018), “SemEval-2018 Task 3: Irony Detection in English Tweets” SemEval-2018. link. The first irony shared task: the best F1 was 0.71 for binary irony detection and 0.51 for fine-grained irony type. [19]
  • Truong, Baldwin, Verspoor and Cohn (2023), “Language models are not naysayers: An analysis of language models on negation benchmarks” arXiv:2306.08189 (preprint). link. LLMs were insensitive to the presence of negation, failed the lexical semantics of negation, and reasoned under negation worse than chance on most NLI datasets. [20]
  • García-Ferrero, Altuna, Álvez, Gonzalez-Dios and Rigau (2023), “This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models” EMNLP 2023. link. With about 400,000 sentences, LLMs classified affirmative sentences competently but fell below the random baseline on negative ones, relying on superficial cues. [21]
  • Ettinger (2020), “What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models” TACL 8:34-48. link. A pre-LLM diagnostic suite showing that BERT-type models showed clear insensitivity to the contextual impact of negation. [22]
  • Kassner and Schütze (2020), “Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly” ACL 2020. link. Pre-trained models did not distinguish “Birds cannot [MASK]” from “Birds can [MASK]”, the direct antecedent of the double-negation problem that the post raises. [23]

B3. Multilingual and cross-cultural reliability

  • Joshi, Santy, Budhiraja, Bali and Choudhury (2020), “The State and Fate of Linguistic Diversity and Inclusion in the NLP World” ACL 2020. link. Quantifies the resource imbalance across the world’s languages and challenges the claim that current models are language-agnostic. [24]
  • Artetxe, Labaka and Agirre (2020), “Translation Artifacts in Cross-lingual Transfer Learning” EMNLP 2020. link. Translated datasets carry artefacts that change model behaviour and that have invalidated earlier cross-lingual findings, so a translated benchmark is not equivalent to a native one. [25]
  • Zhang, Njuguna and Feng (2026), “Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili” Proceedings of MeLLM 2026. link. With 4,900 symmetric English and Swahili prompt pairs, bias “transforms rather than transfers”: stereotype rates moved by up to 12 percentage points, and one model refused 169 English prompts and none in Swahili. [26]

One point about the post’s own design belongs here. Its claim to cross-cultural coverage rests on ten translations scored by 62 raters in total, so each language is represented by a handful of people.[2] The preprint states that the number of questionnaire responses “does not allow a thorough quantitative evaluation of biases associated to each language”.[2] Translation was done by automated tools and then checked by native speakers, which reduces but does not remove the translation-artefact problem.[2][25]

B4. Validation practice, and how the post’s design measures against it

What a credible validity study of an LLM annotator looks like, drawn from the work above: a human baseline with an agreement statistic, prompts fixed in advance, held-out items, and failure cases reported.

  • Plank (2022), “The ‘Problem’ of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation” EMNLP 2022. link. Argues that disagreement in annotation is often genuine variation rather than noise, so a single gold label is the wrong target and inter-annotator agreement is essential context rather than an optional extra. [27]
  • Bail (2024), “Can Generative AI improve social science?” PNAS 121(21) e2314021121. link. Sets out the promise and the limits of generative AI for social science, including bias in training data and risks to replication, and argues for shared open infrastructure. [28]

Measured against that standard, the post’s study does well on some elements and not on others.

  • Human baseline: present. 62 native-speaker volunteers scored all 20 scenarios and were blind to the model results.[2]
  • Agreement statistic: absent. The preprint reports human variability and per-language mean rates but no reliability coefficient such as kappa or ICC for the human raters, and it concedes that the response count is too small for per-language bias estimates.[2] Without that number, the “human average” the models are compared against has no stated reliability.
  • Prompts fixed in advance: present. A single query wording is given in the preprint, settings were left at default, and models were run repeatedly with 95 per cent confidence intervals reported for model ratings.[2]
  • Held-out items: absent, but not expected. All 20 scenarios are the study’s whole set; there is no held-out split, because the study demonstrates a failure mode rather than training a classifier.[2]
  • Failure cases reported: present and detailed, including the irony failure, the LLaMA2 optimism, and the Gemini censorship pattern.[1][2]

Part C. What it means here

For behavioural-science work such as BCT and TDF coding, fidelity checklists and free-text triage, the rule is that an LLM’s output is a candidate label until it has been validated on your own task with your own codebook. What must be validated first: agreement against a human baseline with a reported agreement statistic, a prompt fixed in advance, and a sample held back from prompt development. The BCT literature shows what that baseline costs even for trained humans. The BCTTv1 taxonomy was developed with expert consensus and validated on 40 trained coders,[29] coder training improved validity but did not improve intercoder reliability,[30] and a later expert-reviewed BCT coding exercise reported Cohen’s kappa 0.79 only after training and reconciliation.[31] The automated end of the same field shows the gap between a plausible label and a valid one: an information-extraction system built for behaviour change reports correctly extracted only 58.6 per cent of the BCTs present, and never extracted outcome values against the correct trial arm, which was judged unfit for deployment.[32] A model that returns labels is not yet a model that returns data.

What the evidence does not settle

  • Whether the post’s headline divergence figure holds beyond its 20 curated scenarios. No estimate of the error rate on uncurated text follows from that set.[1][2]
  • Whether the language-dependent failures generalise, because the human baseline has a handful of raters per language and no agreement statistic.[2]
  • Which model generations the pattern applies to. The models tested were the 2023 to early-2024 generation, and later work shows that version changes alone shift outputs, so the findings should be read as a dated snapshot.[1][2][16]
  • Whether passing an annotation-validity test implies valid downstream inference. One study finds it does not, so a validated annotator is necessary but not sufficient.[11]

References

  1. [1] Daniele Proverbio (2024), "LLMs make leap forward in sentiment analysis - but it's not all glittering gold", Medium, 20 February 2024. https://medium.com/@daniele.proverbio94/llms-make-leap-forward-in-sentiment-analysis-but-its-not-all-glittering-gold-ad1ad0b9c86b
  2. [2] Alessio Buscemi and Daniele Proverbio (2024), "ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis", arXiv:2402.01715 (preprint). https://arxiv.org/abs/2402.01715
  3. [3] Gilardi, Alizadeh and Kubli (2023), "ChatGPT outperforms crowd workers for text-annotation tasks", PNAS 120(30) e2305016120. https://www.pnas.org/doi/10.1073/pnas.2305016120
  4. [4] Petter Törnberg (2023), "ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning", arXiv:2304.06588 (preprint). https://arxiv.org/abs/2304.06588
  5. [5] Ziems, Held, Shaikh, Chen, Zhang and Yang (2024), "Can Large Language Models Transform Computational Social Science?", Computational Linguistics. https://arxiv.org/abs/2305.03514
  6. [6] Bermejo, Gago, Gálvez and Harari (2025), "LLMs outperform outsourced human coders on complex textual analysis", Scientific Reports 15:40122. https://www.nature.com/articles/s41598-025-23798-y
  7. [7] Rønningstad, Velldal and Øvrelid (2024), "A GPT among Annotators: LLM-based Entity-Level Sentiment Annotation", Proceedings of LAW-XVIII. https://aclanthology.org/2024.law-1.13
  8. [8] Calderon, Reichart and Dror (2025), "The Alternative Annotator Test for LLM-as-a-Judge", ACL 2025. https://aclanthology.org/2025.acl-long.782
  9. [9] Nakamura, Tan and Yean (2026), "Observational Equivalence of LLM and Human Annotation", arXiv:2609.22133 (preprint). https://arxiv.org/abs/2609.22133
  10. [10] Pangakis, Wolken and Fasching (2023), "Automated Annotation with Generative AI Requires Validation", arXiv:2306.00176 (preprint). https://arxiv.org/abs/2306.00176
  11. [11] Baumann, Röttger, Urman, Wendsjö, Plaza-del-Arco, Gruber and Hovy (2025), "Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation", arXiv:2509.08825 (preprint). https://arxiv.org/abs/2509.08825
  12. [12] Santurkar, Durmus, Ladhak, Lee, Liang and Hashimoto (2023), "Whose Opinions Do Language Models Reflect?", ICML 2023. https://arxiv.org/abs/2303.17548
  13. [13] Durmus, Nguyen, Liao and colleagues (2023), "Towards Measuring the Representation of Subjective Global Opinions in Language Models", arXiv:2306.16388 (preprint). https://arxiv.org/abs/2306.16388
  14. [14] Sclar, Choi, Tsvetkov and Suhr (2024), "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design", ICLR 2024. https://arxiv.org/abs/2310.11324
  15. [15] Zhao, Wallace, Feng, Klein and Singh (2021), "Calibrate Before Use: Improving Few-Shot Performance of Language Models", ICML 2021. https://arxiv.org/abs/2102.09690
  16. [16] Atil and colleagues (2024), "Non-Determinism of 'Deterministic' LLM Settings", arXiv:2408.04667 (preprint). https://arxiv.org/abs/2408.04667
  17. [17] Zhang, Zou, Lian, Tiwari and Qin (2024), "SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding", arXiv:2408.11319 (preprint). https://arxiv.org/abs/2408.11319
  18. [18] Abu Farha, Oprea, Wilson and Magdy (2022), "SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic", SemEval-2022. https://aclanthology.org/2022.semeval-1.111
  19. [19] Van Hee, Lefever and Hoste (2018), "SemEval-2018 Task 3: Irony Detection in English Tweets", SemEval-2018. https://aclanthology.org/S18-1005
  20. [20] Truong, Baldwin, Verspoor and Cohn (2023), "Language models are not naysayers: An analysis of language models on negation benchmarks", arXiv:2306.08189 (preprint). https://arxiv.org/abs/2306.08189
  21. [21] García-Ferrero, Altuna, Álvez, Gonzalez-Dios and Rigau (2023), "This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models", EMNLP 2023. https://arxiv.org/abs/2310.15941
  22. [22] Allyson Ettinger (2020), "What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models", TACL 8:34-48. https://aclanthology.org/2020.tacl-1.3
  23. [23] Kassner and Schütze (2020), "Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly", ACL 2020. https://aclanthology.org/2020.acl-main.698
  24. [24] Joshi, Santy, Budhiraja, Bali and Choudhury (2020), "The State and Fate of Linguistic Diversity and Inclusion in the NLP World", ACL 2020. https://aclanthology.org/2020.acl-main.560
  25. [25] Artetxe, Labaka and Agirre (2020), "Translation Artifacts in Cross-lingual Transfer Learning", EMNLP 2020. https://arxiv.org/abs/2004.04721
  26. [26] Zhang, Njuguna and Feng (2026), "Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili", MeLLM 2026. https://aclanthology.org/2026.mellm-1.17
  27. [27] Barbara Plank (2022), "The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation", EMNLP 2022. https://arxiv.org/abs/2211.02570
  28. [28] Christopher A. Bail (2024), "Can Generative AI improve social science?", PNAS 121(21) e2314021121. https://www.pnas.org/doi/10.1073/pnas.2314021121
  29. [29] Michie and colleagues (2015), "Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions", Health Technology Assessment 19(99). https://pubmed.ncbi.nlm.nih.gov/26616119
  30. [30] Wood, Richardson, Johnston, Abraham, Francis, Hardeman and Michie (2015), "Applying the behaviour change technique (BCT) taxonomy v1: a study of coder training", Translational Behavioral Medicine 5(2):134-148. https://pmc.ncbi.nlm.nih.gov/articles/PMC4444702
  31. [31] Zhao, Guo, Miao, Luo, Wang and Wu (2026), "Behavior Change Content and Implementation of Large Language Model-Driven Conversational Agents in Cardiometabolic Care: Scoping Review", Journal of Medical Internet Research 28:e89190. https://pmc.ncbi.nlm.nih.gov/articles/PMC13372078
  32. [32] Finnerty Mutlu and colleagues (2024), "Automated information extraction for behaviour change intervention reports", Wellcome Open Research 9:493. https://wellcomeopenresearch.org/articles/9-493/v1