For financial sentiment, the published tests agree about dictionaries and split on FinBERT against prompted language models. Word-count dictionaries score below the leading model in every head-to-head study reviewed here [5, 11, 12, 17]. Fine-tuned encoders such as FinBERT and RoBERTa score about the same as or above a language model prompted zero-shot when they were trained on the label being tested [11, 13, 14]. A prompted model comes out ahead when the label differs from the one the encoder learned, as with forex headlines or a five-step scale of Federal Reserve stance [12, 15].
The choice has a second half. A prompted model's labels can change with the wording of the prompt, across repeated runs at fixed settings and when the vendor updates or retires the model [21, 24, 25, 26]. Open resources change too: the Loughran-McDonald word lists have added and removed words, and the public FinBERT checkpoint gained a license on 12 October 2026 [2, 8]. A sentiment label can be reproduced only when its definition, method, version, prompt and date are stored with it.
Define the label before choosing a method
Each benchmark asks a different question. The Financial PhraseBank asked annotators whether a news sentence "may have positive, negative or neutral influence on the stock price" [5]. The FOMC dataset in Trillion Dollar Words calls a sentence hawkish when it "would indicate a future monetary policy tightening" [11]. Fatouros et al. label forex headlines by their likely short-term effect on one currency pair [15]. Kirtac and Germano use no human labels: an article counts as positive when the stock's three-day excess return after it is positive [17].
These definitions do not transfer. The FOMC authors note that a sentence containing the word "increase" can be either dovish or hawkish [11]. Mapped from positive and negative to dovish and hawkish, Araci's FinBERT reached an F1 score of 0.425 on their data [11]. In an event study on public communications, the label decides which statements count as events.
People disagree as well. PhraseBank annotators agreed 98.7% of the time on positive against negative and 75.2% on positive against neutral [5]. Model scores rise with annotator agreement [18], so results on the full-agreement subset describe the sentences people found easiest.
The Loughran-McDonald finance dictionary
In 10-Ks from 1994 to 2008, 73.8% of the negative word counts from the Harvard list came from words such as tax, cost, capital and liability that are usually not negative in a financial context [3]. Loughran and McDonald built a negative list of 2,337 words for financial text [3]. The current Master Dictionary, updated March 2026 and covering 1993 to 2025, has seven sentiment categories, and the lists are free for academic research, with commercial licenses on request [1].
Gentzkow, Kelly and Taddy say dictionary methods suit cases where prior information is strong and reliable and the data carry little information of their own [4]. Word counts miss context: increase and decrease read differently when revenues or costs follow [3]. On PhraseBank sentences with full agreement, a Loughran-McDonald word count recalled 12.5% of positive sentences [5].
FinBERT and other fine-tuned encoders
Araci's FinBERT is BERT trained further on financial text and fine-tuned on the PhraseBank, where it reached an accuracy of 0.86 on all sentences and 0.97 on the full-agreement subset [7]. Yang, Uy and Huang released a different model under the same name, pretrained on filings, earnings calls and analyst reports [9]. Huang, Wang and Yang report that their FinBERT substantially outperforms the Loughran-McDonald dictionary on researcher-labeled analyst report sentences [10]. A label record therefore needs the checkpoint identifier.
FinBERT vs prompted language models: the head-to-head results
| Study | Text, label and metric | Dictionary | BERT-family encoder | Language model |
|---|---|---|---|---|
| Shah, Paturi and Chava, 2023 [11] | FOMC sentences, three-way stance, weighted F1 | 0.497 (rule-based) | RoBERTa-large 0.717 | 0.587, zero-shot, May 2023 |
| Hansen and Kazinnik, 2023 [12] | FOMC sentences, five-step stance, accuracy | Loughran-McDonald 0.28, Henry 0.35 | BERT 0.25 | 0.37, zero-shot |
| Shah and Chava, 2023 [13] | PhraseBank, full agreement, weighted F1 | Not tested | RoBERTa-large 0.976 | 0.893, zero-shot |
| Li et al., 2023 [14] | PhraseBank, 50% agreement, accuracy | Not tested | FinBERT 0.86 | 0.78 to 0.86, zero- to five-shot, July 2023 |
| Fatouros et al., 2023 [15] | 2,291 forex headlines, accuracy | Not tested | FinBERT 0.561 | 0.730 to 0.790, four prompts |
| Vamvourellis and Mehta, 2025 [18] | PhraseBank, full agreement, macro F1 | Not tested | FinBERT 0.962 | 0.776 to 0.895 across 12 setups, 0.890 and 0.895 with no reasoning prompt |
| Kirtac and Germano, 2024 [17] | 965,375 articles, sign of later returns, accuracy | 50.1% | FinBERT 72.2% as published, BERT 72.5% fine-tuned on the label | Not prompted. OPT fine-tuned on the label: 74.4% |
The language models are prompted hosted commercial models, except in the last row [17]. In Shah and Chava's 2023 FOMC test, the reported average labeling time per sentence was 3.85 milliseconds for RoBERTa-base and 3.91 seconds for the hosted model [13].
Financial PhraseBank benchmark
In Li et al., five-shot prompting brought one hosted model level with FinBERT, at 0.86 and 0.97 on the two agreement levels [14]. The public FinBERT checkpoint was trained on much of the PhraseBank, so Vamvourellis and Mehta treat its score as "a reference point rather than a fair competitor" [18].
Hawkish and dovish classification of FOMC text
On FOMC text, two studies point in opposite directions. With three classes and an encoder fine-tuned on those labels, RoBERTa-large led and the zero-shot hosted model trailed [11]. On five steps with no matching encoder, the hosted model led, and after fine-tuning on 400 sentences it reached 0.61 on 100 held-out ones [12]. People disagree on that scale too: on 264 of 500 sentences, one reviewer differed by more than one step [12].
Headlines with a different label
The figures in the abstract of Fatouros et al., about 35% better performance in sentiment classification and a 36% higher correlation with market returns than FinBERT, come from one setup: one 2023 hosted model, used zero-shot, on headlines the authors labeled themselves [15]. One of the four prompts mirrored their annotation approach [15].
How reliable are language model labels?
Across four datasets of tweets and news, a hosted model's zero-shot accuracy exceeded crowd workers' by about 25 percentage points, and two runs agreed on 97% of labels on average at a temperature of 0.2 [20].
Reiss measured a Krippendorff's alpha of 0.75 between two temperature settings, rising to 0.91 after pooling ten runs, and changing one word in the instruction kept alpha at or below 0.6 [21]. At a temperature of 0 with a fixed seed, Atil et al. saw accuracy vary by up to 15% across ten runs [24]. Prompt format alone moved one open model's few-shot accuracy by up to 76 points [23]. None of these studies used financial sentiment text. Pangakis et al. found precision or recall below 0.5 on 9 of 27 annotation tasks and conclude that model labels must be validated against human ones [22]. The same checks for agents appear in a black-box protocol for evaluating LLM agents.
Version drift in models and word lists
One hosted service's accuracy at identifying prime numbers fell from 84.0% in March 2023 to 51.1% in June 2023 [25]. The vendor's documentation lists its dated March 2023 snapshots as shut down, on 13 September 2024 for the smaller chat model and on 26 March 2026 for the larger one [26]. The 14 March 2023 snapshot that Lopez-Lira and Tang recorded for their labels is among them, so those labels cannot be regenerated [19, 26].
The public FinBERT checkpoint had no commits between 23 May 2023 and 12 October 2026, when it gained a CC BY-NC-SA 4.0 license that restricts commercial use and a copy of its weights in a new file format, which the commit states gives identical outputs [8]. Transformers can load a checkpoint pinned to a commit id [29]. The Loughran-McDonald negative list held 2,337 words in 2011, 2,356 in the version Hansen and Kazinnik used and 2,345 in the 2025 file linked from the dictionary page, by this note's count [1, 3, 12].
What to store with every label
- The label definition and codebook, including whose view and what horizon, shared by people and model [5, 11, 15, 22].
- The label map from integers to names, which differs across resources [8, 11].
- The method and exact version: dictionary file, checkpoint identifier and commit, or hosted snapshot, with the date of the run [1, 8, 19, 29].
- The prompt text and settings, the number of runs and their agreement, and parse failures [13, 21, 23].
- The human benchmark: annotators, agreement and the subset used [5, 12].
- Overlap between training data and test text, and the model's training cutoff against the text's date, since a model may know what followed a text it labels [13, 18, 30].
Rillor's public communications and market response data holds selected public statements as issued, with their entities, timestamps and classifications, each classification with its classifier version, and pairs each statement with price observations in a declared window around it.
Read the full report
The full report adds each dataset's label definition, the head-to-head table with run dates, the dictionary's change log and the reliability studies.