The first step in how to create a dataset for LLM training is a document, written before anyone collects a record. It states what the data is for, which sources are in scope and on what terms, what each record holds, the file format, how much data is needed, which records are held out for evaluation and how the delivered set is accepted. Datasheets for Datasets asks dataset creators to read its composition questions "prior to any data collection" and to answer them once collection is complete [1].
Data problems found late are common. In interviews with 53 practitioners in high-stakes fields, 92% reported at least one data cascade, the study's term for "compounding events causing negative, downstream effects from data issues", and 45.3% reported two or more in a project [3]. The authors describe cascades as "delayed" and "often avoidable" [3].
How to create a dataset for LLM training: write the specification first
Hutchinson and colleagues (FAccT 2021) adapt five documents from software development to datasets [2]. The dataset requirements specification comes first, then a design document, an implementation diary, a testing report and a maintenance plan [2]. Their template covers the vision, intended and non-intended uses, two or three mock records, where the data comes from, the numbers and types of instances, sampling, processing, maintenance, sharing and a changelog [2].
Other templates cover the same ground. Datasheets for Datasets asks 57 questions in seven sections (counted for this note) [1]. Data Cards document 31 themes whose importance varies by task [4]. Data statements for language data open with a curation rationale: which texts were included and why [5].
Start from the use
The specification opens with the task the data serves. Datasheets asks "For what purpose was the dataset created?" [1], and Hutchinson's template names intended and non-intended uses [2]. ISO/IEC 5259-2 ties each data quality requirement to a "data usage scope", with acceptance criteria that "can be quantitative or qualitative" [8].
For high-risk systems, Article 10 of the EU AI Act requires training, validation and testing data sets to be "relevant, sufficiently representative" and, as far as possible, "free of errors and complete in view of the intended purpose" [6]. One model provider's fine-tuning guide asks for evaluations to be set up before any fine-tuning, so that improvement over a base model can be measured [15].
Name the source universe and the rights
List each source in scope and each one left out, with the terms that apply to it, read at the original source. An audit of 1,858 fine-tuning datasets in 44 collections found over 70% of licenses for popular datasets on two hosting platforms "Unspecified", and three platforms gave licenses that were too permissive 16% to 29% of the time [9]. Terms also move. An audit of 14,000 web domains found that restrictions added in one year, 2023 to 2024, left roughly 5% or more of all tokens in the C4 corpus, or "28%+ of the most actively maintained, critical sources in C4", fully restricted [10].
The European Commission's template for public summaries of training content sorts sources into six categories: publicly available datasets, private datasets obtained from third parties, data crawled from online sources, user data, synthetic data and other sources [11]. The same list works as a checklist, with a date on each entry. Record whether synthetic records are allowed, their share and how they were made. A Nature study found that indiscriminate training on model-generated content causes defects in which the tails of the original distribution disappear [28]. Provenance at the record level is the subject of datasets that state their limits. This note is research and gives no legal advice.
Define entities, fields and labels
Write the record schema: each entity type that must resolve and join across sources, each required field and each label with its definition. How records are matched is set out in how entity resolution matches records across sources.
Labeling instructions belong in the specification, because they shape the data. In two natural language inference datasets, where each example pairs a premise with a hypothesis sentence, a simple classifier that read only the hypothesis predicted the label for about 67% of SNLI and 53% of MultiNLI examples, because the protocol used to write them left clues [13]. Models also picked up traits of the people who wrote the examples and generalized poorly to new writers, so Geva and colleagues recommend that test set annotators be disjoint from training set annotators [14]. Test sets carry label errors too, an average of at least 3.3% across ten widely used datasets [12].
LLM training dataset format
Format follows the training method. One model provider's fine-tuning guide asks for JSON Lines, "one complete JSON structure on every line", in a chat format, with at least 10 lines [15]. One open-source training library separates the format, standard or conversational, from the type, such as prompt-only, prompt-completion or preference, and defines each type by its columns: a preference record has "prompt", "chosen" and "rejected" [16]. Croissant, an MLCommons vocabulary in JSON-LD, describes delivered files with their checksums, record sets and typed fields [17].
State coverage, history and geography
Name the time range, the languages and the regions, and tie each to the period and place the model serves. In a study of 28 pretrained models, a time gap between pretraining and evaluation data lowered performance, and fine-tuning did not overcome it [18]. Check language on a read sample: an audit of 205 language corpora from five public datasets found at least 15 with no usable text, and many mislabeled [19]. For high-risk systems, Article 10(4) of the EU AI Act requires data sets to take into account the geographical, contextual, behavioural or functional setting of use, as far as the intended purpose requires [6].
Write each cleaning filter down and ask for a sample of what it removed. In the C4 corpus, a blocklist removed African American English and Hispanic-aligned English at 42% and 32%, against 6.2% and 7.2% for White-aligned and other English [20]. In the FineWeb study, global deduplication of a 2013 crawl kept 10% of its data, and that 10% was of worse quality than the 90% removed [27].
LLM fine-tuning dataset size
State size as a range tied to the job, and count unique records apart from repeats. One provider's guide sets a minimum of 10 examples, reports gains from 50 to 100 and suggests starting with 50 well-crafted demonstrations [15]. The LIMA study tuned a 65-billion-parameter open model on 1,000 curated examples, about 750,000 tokens [21]. In its ablations, doubling the training set did not improve response quality, while more diverse and higher-quality data did [21].
Pretraining is another scale. Compute-optimal estimates grow tokens with model size, to 20.2 billion tokens for a model of 1 billion parameters [22]. Up to four epochs of repeated data changed loss little against unique data [23].
Hold out evaluation data in the specification
Fix the evaluation split before collection and remove its near duplicates from the training set. In one study, 4.6% of the C4 validation set and 14.4% of the RealNews validation set had an approximate duplicate in training, which can inflate scores [24]. C4 also contains examples from other benchmark datasets [20]. A review of data leakage in research using machine learning found errors in 329 papers across 17 fields [25].
Size the split for the smallest difference you need to detect. Typical test sets of 2,000 sentences have about 75% power to detect a difference of 1 BLEU point on a translation task [26]. Contamination in agent tests is covered in what agent benchmarks measure, and what they miss.
Write acceptance criteria you can test
Hutchinson and colleagues call this step requirements testing, or acceptance testing [2]. Ideal tests return pass or fail, and requirements should be traceable and in many cases measurable [2]. Write each criterion as a measure, a threshold and a sample size: completeness of required fields, label accuracy on a reviewed sample, the near-duplicate rate against the held-out split. The Open Data Contract Standard lets quality rules be library metrics, such as row counts and null values, each with an operator that checks the expected range [31].
Acceptance sampling decides whether a delivery is likely acceptable. It does not estimate its quality [29]. A reviewed sample of 100 records with no errors still allows a true error rate up to 3.62% at 95% confidence (two-sided), and 1,000 clean records bring that limit to 0.37% (exact binomial method, derived) [30]. Scoring rules for each field and intervals for small samples are worked through in how to test document extraction accuracy before you buy.
Plan refresh, versions and delivery
Datasheets asks whether a dataset is updated, how often and by whom, and whether older versions stay available [1]. Hutchinson's maintenance plan asks how stale data is detected and fixed [2]. The specification sets the refresh cadence, what a source change triggers, how versions are numbered and which documents travel with each version.
Rillor's custom datasets service scopes, builds and keeps current training and evaluation data to the requirement a client sets, and each dataset comes with its schema, records with resolved identities, source provenance, lineage, version history, quality notes and stated limits.
Read the full report
The full report describes the published templates, gives the evidence and limits for each line of the specification, adds a checklist with the test for each line and an interval table for acceptance samples, and lists the sources.