Look-ahead bias is the use of information that was not available on the date a forecast is supposed to have been made [9, 18]. In an LLM backtest it has a route that older backtests lacked: the model's training data may already hold the outcomes being forecast, so a strong historical score can reflect memorized outcomes as well as skill [2, 9]. Testing for it takes four things: a recall test on the model, a reference that separates leakage from recency, inputs fetched as they stood at each forecast date, and a count of every variant tried [3, 4, 11, 19].
Many published studies do not mention it. A 2026 preprint by Kong and colleagues reviewed 164 papers from 2023 to 2025 that evaluate language models in finance, and only 26.8% acknowledged look-ahead bias [9].
How look-ahead bias gets into an LLM backtest
Kong and colleagues name two channels: knowledge stored in the model's weights, and documents brought in through retrieval [9]. The usual fixes are a declared knowledge cutoff and a publication-date filter on retrieved documents. They fail, the authors write, because cutoffs "are opaque and not verifiable" and retrieved documents "are edited and re-ranked using future information" [9].
Paleka and colleagues set out the retrieval problem in a 2025 preprint [5]. "Many search engines do not robustly implement date restrictions." [5]. Pages change without a new publication date, and the ranking itself can rely on knowledge from after the forecast date [5]. In their example, a search for "January 6th" restricted to 2020 returns documents with "an abnormally strong association to U.S. politics (by 2020 standards)" [5]. They suggest an index of sources with clear and reliable dates, or one kept for each period [5]. Halawi and colleagues treat each retrieval date as the "simulated date" of a forecast and warn that choosing it from the resolution date leaks information about the outcome [12].
The question set can leak too. A 2021 question on whether Queen Elizabeth would live to 100 can only resolve "no" when it is backtested in 2025, and at least 3.8% of one published question set resolved early [5]. Paleka and colleagues also call knowledge cutoffs "more of a guideline than a guarantee" [5]. A widely used chat model with a stated October 2023 cutoff, told in its system prompt that its cutoff was November 2023, described a meeting first announced on 8 November 2023, though the authors say they cannot confirm this as leakage, since the visit had been discussed in October 2023 [5].
Memorization before the training cutoff
The direct evidence comes from Lopez-Lira, Tang and Zhu, in a preprint revised in December 2025 [2]. Before its training cutoff, a widely used chat model recalled S&P 500 index levels on certain dates and unemployment rates to a tenth of a percentage point [2]. Told to ignore information after 2010, it still called the direction of quarterly GDP growth with 97.6% accuracy before that date and 98.0% after, against 40% on quarters past its real cutoff [2]. After the cutoff the authors saw no recall [2]. The summary in LLM agents as market participants and the section on frozen snapshots and memorization in A black-box protocol for evaluating LLM agents cover what this means for agent studies. This note turns to detection and repair.
Anonymizing the text cuts both ways
Hiding company names looks like the obvious repair. Glasserman and Lin tested it on two sets of headlines, 129,431 on 6,723 companies and 181,908 on 678 S&P 500 companies [1]. Within the model's training window, anonymized headlines did better. The authors read this as a distraction effect, in which general knowledge of a company interferes with measuring a text's sentiment, having a greater impact than look-ahead bias [1]. The effect was particularly strong for larger companies, and out of sample look-ahead bias was not a concern [1].
Masking can also leak. Lopez-Lira, Tang and Zhu found that a model given earnings call transcripts with names, numbers and dates removed still identified the company and the quarter [2]. And it can cost information. In a preprint on credit downgrades revised in September 2026, Wu and colleagues found raw texts predicted better than anonymized ones, and look-ahead bias did not explain the difference [17]. The trade-offs between scoring methods are covered in dictionaries, FinBERT or language models for financial sentiment.
Test the model with Lookahead Propensity
Gao, Jiang and Yan, in a preprint revised in June 2026 and titled "Detecting Lookahead Bias in LLM Forecasts", ask the model about an outcome without showing it any text [3]. The query holds a firm's name, ticker and target date and nothing else [3]. Lookahead Propensity (LAP) is the probability the model puts on knowing whether the outcome went up or down [3].
On stock news before the cutoff, roughly 23% of firm-days were saturated, with LAP at 0.95 or more [3]. Across all 7,806 firm-day queries in 2024, after the cutoff, the maximum was below 0.0001 [3]. To turn LAP into a test, the authors add its interaction with the model's forecast to an accuracy regression. A positive interaction "indicates lookahead-bias contamination" [3]. The test needs token probabilities, which open-weight models give directly. Without them, answer frequencies across repeated samples approximate them "more coarsely" [3].
Older contamination detectors ask a different question. Guided instruction and Min-K% Prob test whether a model saw a given text [13, 14]. LAP tests whether the model knows an outcome [3].
Why a before-and-after-cutoff split proves little
The common check compares a model's score before its training cutoff with its score after. Zhang and Stadie, in an August 2026 preprint, call this check "uninformative" [4]. Four of five flagship models failed it on questions that all resolved at least 30 days after their documented cutoffs, with false leakage gaps up to +0.061 in improvement over the crowd [4]. Their scores are Brier score reductions [4], the measure explained in Brier score and log loss for probability forecasts.
The reason is recency. Models know more about times close to their cutoff, so performance fades after it even without any leakage [4, 15]. A flat profile does not prove a backtest clean either [4]. The authors offer two references instead [4]. A known cutoff identifies leakage at the boundary, where a jump in the loss means memorization switching off [4]. A matched clean control, an older model vintage that is clean on the evaluation window and shares the target's recency profile, gives a leakage-adjusted score [4]. Lopez-Lira, Tang and Zhu add that a post-cutoff sample small relative to the pre-cutoff one cannot reliably tell skill from memorization [2]. Where the question allows, live forecasting avoids leakage altogether: ForecastBench asks only about events with no known answer at submission [21].
Point-in-time models trained to a date
A point-in-time model is trained only on text available up to its cutoff. In a 2026 preprint, Yan and colleagues trained twelve 1.3-billion-parameter models with annual cutoffs from 2013 to 2024, and perplexity probing showed each model's knowledge bounded by its cutoff year [7]. In another, Kelly and colleagues trained models of up to 4 billion parameters on 1 trillion date-filtered tokens, and a performance gap to unrestricted models remained on several tasks [8]. He and colleagues, in a 2025 preprint, found look-ahead bias "modest" in their news application and call it model- and application-specific [16].
A date in a corpus needs checking. FineWeb's card defines its date field as the crawl date [27], and Yan and colleagues note that a page crawled in 2015 may have been written in the 1990s [7].
Inputs need vintages too
The numbers a backtest feeds the model are revised as well. The default real-time period of the St. Louis Fed's FRED data API is today, so a call left at the default returns the past as it is known now [11]. The bank's archival service, ALFRED, returns "each economic data release (vintage) that was available on a specific date in history" [10]. SEC filing data carry an acceptance time, and filings accepted after 5:30pm EST count as filed on the next business day [26]. Paleka and colleagues, citing earlier work, note that Enron drastically changed its 1998 to 2001 earnings in 2002 and that many repositories no longer report the original values [5].
Count every variant you try
Searching over variants inflates a backtest too [19, 20]. Bailey and colleagues write that when a researcher tries more than one configuration, "overfitting is always present" [19]. Bailey and López de Prado add that virtually all backtests published in academic journals and investment offerings leave out the number of trials attempted [20]. Applied to an LLM backtest, each prompt wording, model, retrieval setting and input set counts as a trial in the sense of [19, 20].
Rillor builds forecasting and backtesting pipelines around a client's questions, and their backtests replay only what was known at each point in time.
Read the full report
The full report sets out each test from the paper that defines it, with what it needs and where its authors say it stops, adds a ten-step checklist for an LLM backtest with each step traced to a source, and lists all 28 sources.