Deep learning on limit order books has a long record of rising benchmark scores, and the studies that retested those models found the scores fall on new data. LOBCAST, a 2023 benchmark study, retrained 15 deep learning limit order book models, 13 of them published from 2017 to 2022, and measured F1 scores of 48% to 61% on Nasdaq stocks from 2021 and 2022, significantly lower than on the FI-2010 dataset the models were first scored on [1]. A 2025 study found that the DeepLOB model forecast large-tick stocks far better than small-tick ones [2].
This post sets the benchmark and replication papers against the original claims: the FI-2010 dataset, the DeepLOB model, Sirignano and Cont's universal features, and the order flow imbalance model of Cont, Kukanov and Stoikov. Each figure stays with its paper's own dataset, label and horizon, because those differ across papers and no source maps one onto another [1, 2, 3, 12].
What deep learning on limit order books predicts
A limit order book records "all the outstanding bid and ask orders currently available on an exchange or a trading platform" [1]. The models read the prices and volumes of the top levels on each side and predict whether the mid-price, the average of the best bid and the best ask, goes up, down or stays stationary [1, 2].
In the classic label, the mean of the next k mid-prices is compared with the current mid-price against a threshold θ. FI-2010 sets that threshold at a percentage change of 0.002 [1, 3]. In FI-2010, LOBCAST and the 2025 tick-size study, horizons count order book events, not seconds [1, 2]. For 13 of the 15 stocks in one study, 10 updates are more likely to take under a second [2].
The book has to be rebuilt before any model reads it. Nasdaq's TotalView-ITCH feed "uses a series of messages to track the life of a customer order" [9], and LOBSTER reconstructs historic Nasdaq books from that data as message and order book files for researchers, for an annual fee [1, 6]. How a book is rebuilt from snapshots and updates, and why one missed update leaves it wrong, is covered in rebuilding an order book from snapshots and diffs.
FI-2010, the original benchmark
FI-2010 holds five stocks traded in Helsinki over ten days from 1 to 14 June 2010, about 4 million samples, or 394,337 events when sampled every 10 events [1, 3]. Its authors normalized the open version "in order to prevent reconstruction of the original NASDAQ data" [3], so the original book cannot be recovered [1]. The dataset paper's two baselines reached an average F1 of about 46% [3].
F1 is the usual score because the classes are unbalanced. At k = 1, 63% of FI-2010's labels are stationary, and θ was chosen to balance the classes at k = 5 [1].
Three papers question FI-2010 as a test. DeepLOB's authors called it "far too short, downsampled and taken from a less liquid market" [12]. LOBCAST notes it is "already pre-processed (filtered, normalized, and labelled)" [1], and Briola, Bartolucci and Aste call it "too simplistic, leaving ample space for models' overfitting" [2]. Scores on it kept climbing: TLOB, a 2025 transformer, reports an F1 of 92.81 at horizon h = 100 [4]. FI-2010 is still in use: TensorLOB, posted on 2 November 2026, reports results on FI-2010 alone [17].
DeepLOB and the replication that followed
DeepLOB reported F1 of 83.40% at k = 10 on FI-2010 and 70.15% at k = 20 on London Stock Exchange data from 2017, and its authors read similar scores on five stocks left out of training as a sign the model can "extract universal features" [12]. Sirignano and Cont, whose data covered about 1,000 Nasdaq stocks, compared models on about 500 of them out of sample in June to August 2015 and found a universal model improved accuracy on the next price move "by around 10%" over stock-specific linear models [10]. That is two-class accuracy, so it does not compare with three-class F1 [10, 12].
LOBCAST retrained 15 models on FI-2010 and on two new LOBSTER datasets of six Nasdaq stocks, each of 10 trading days, in July 2021 and February 2022 [1]. Its Table 2 gives F1 in percent, as a mean over horizons and five seeds [1]:
| Model | Claimed on FI-2010 | Measured on FI-2010 | LOB-2021 | LOB-2022 |
|---|---|---|---|---|
| DeepLOB | 78.9 | 71.4 | 59.5 | 59.5 |
| TransLOB | 87.3 | 59.4 | 57.7 | 50.4 |
| BiN-CTABL | 80.1 | 82.6 | 61.2 | 59.2 |
Source: [1], Table 2.
About half of the hyperparameter search runs diverged to an F1 of 33% or less, the ranking changed considerably against the claimed scores, and ensembles did not exceed the top single models [1]. The authors give two possible reasons for the gaps: "the lack of the original code and the missing hyperparameters declaration" [1]. On Intel data from 2019, a separate study found a multilayer perceptron "performs comparably to or better than" CNN-LSTM models [14].
Tick size and predictability
Briola, Bartolucci and Aste ran DeepLOB on LOBSTER data for 15 Nasdaq stocks from 2017 to 2019, with 45 days of training, 5 of validation and 10 of testing each year [2]. They grouped the stocks by mean spread, small-tick at about 3 ticks or more and large-tick at about 1.5 ticks or less, and scored forecasts with the Matthews correlation coefficient (MCC), on which 0 means a random forecast and +1 a perfect one [2].
| Tick class | H = 10 | H = 50 | H = 100 |
|---|---|---|---|
| Small-tick | 0.11 | 0.04 | 0.01 |
| Medium-tick | 0.13 | 0.085 | 0.036 |
| Large-tick | 0.29 | 0.36 | 0.26 |
Source: [2], MCC without a probability threshold. H counts order book updates.
For small-tick stocks at 100 updates, the results "suggest that the model is producing random forecasts" [2]. The authors suggest tick size "could be used as a proxy measure of a stock's predictability" [2].
From classification scores to complete transactions
A high classification score does not show that a forecast leads to complete transactions [2]. Backtesting on history alone assumes forecasts acted on in time, every order executed, and zero market impact and costs, assumptions the 2025 study rejects [2]. Its measure pT divides correct transactions by all the transactions implied by the true labels and the forecasts together [2].
At 10 updates, CSCO scored an MCC of 0.31 and a pT of 0.10 [2]. Raising the probability threshold lifts MCC and lowers pT: at 50 updates with a 0.9 threshold, BAC's MCC was 0.69 and its pT 0.02 [2]. When TLOB's authors set the label threshold to the average spread on Tesla, its F1 at h = 50 fell from 43.48 to 41.39 [4].
Order flow imbalance as the baseline
Cont, Kukanov and Stoikov found that order flow imbalance, the imbalance between supply and demand at the best bid and ask, explained 10-second mid-price changes with an average R² of 65% in a linear model, on trades and quotes data for 50 stocks in April 2010 [5]. That data source is described in TAQ data explained. The relation is contemporaneous, so it explains moves in the same interval and makes no forecast [5].
As a model input, order flow did well. In a study of ten Nasdaq stocks from 2019 to January 2020, a deep model fed order flow was in the statistically superior set of models at 89% of the stock and horizon pairs where predictability was found, and DeepLOB fed book states at 11% [6].
Generative models and LOB-Bench
Replaying recorded history "does not capture the market response to the experimental agents' actions", so the papers turn to simulators [1, 24]. LOB-Bench, presented at ICML 2025, scores generated order book messages on their distributions, a trained discriminator and market impact [7, 8]. In its tests "The LOBS5 model beats the baseline and coletta model on almost all scores", and adding one model's synthetic data to a mid-price forecaster's training "markedly reduces the resulting prediction precision and recall" [7]. A 19 October 2026 preprint found impact exponents of 0.63 to 1.21 in five generative models [27]. Research on LLM agents in simulated markets is reviewed in LLM agents as market participants and measuring LLM agents in simulated auctions.
What to check in a published result
The later papers point to a short list of checks [1, 2, 4, 6]:
- the data: FI-2010 alone, or other stocks and periods
- the label and horizon, in the paper's own units
- whether a score is claimed or reproduced, and over how many seeds
- the tick sizes of the stocks tested
- a measure beyond F1, such as MCC or pT
- normalization that uses only past data, such as a rolling z-score
- public code and hyperparameters
- a simple baseline, such as a multilayer perceptron or order flow inputs
Rillor's market data supports forecasting models tested out of sample on recorded history and scored against baselines, and sets the spreads, depth and order flow of simulated markets from real books.
Read the full report
The full report adds each paper's data and splits, the full LOBCAST and tick-size tables, the transaction measure in detail, the generative benchmark's rankings and the evaluation practices the papers recommend.