Data preparation for LLM fine-tuning starts with the record. A fine-tuning file is a list of records, and the training method decides what each one holds: an instruction and its answer, a conversation of role-tagged messages, a prompt with a chosen and a rejected reply, or one reply labelled desirable or undesirable. One open-source training library describes every dataset by a format, standard or conversational, and a type, such as prompt-completion or preference, with each type "characterized by its columns" [1].
The layout then decides how a record becomes tokens, which of those tokens are trained and which checks a file has to pass. Small defaults matter. One library truncates sequences longer than 1,024 tokens by default, keeping the start [4].
Data preparation for LLM fine-tuning: pick the record type first
| Record type | What one record holds | Method that reads it | Sources |
|---|---|---|---|
| Prompt-completion | A prompt and the completion to learn | Supervised fine-tuning (SFT) | [1, 10] |
| Chat | A list of messages, each with a role and content | SFT | [1, 2, 8] |
| Preference | A prompt, a chosen reply and a rejected reply | Direct preference optimization (DPO), KTO and reward models | [1, 21, 23] |
| Unpaired preference | A prompt, one reply and a true or false label | KTO | [1, 22] |
| Prompt-only | A prompt alone | GRPO and RLOO | [1] |
Hosted services rename the parts. One fine-tuning service calls a preference record's fields input, preferred_output and non_preferred_output [2], and one cloud provider names the assistant role model [9]. The library converts records between types and recommends doing so before the chat template is applied [1].
Instruction records: the SFT dataset format
A 52,000-example dataset that Stanford researchers released in 2023 uses three fields: an instruction that describes the task, an optional input with context and an output [25]. Around 40% of its records have an input, and two prompt templates, with and without the input, wrap the fields before training [25]. One cloud provider's guide uses a plain prompt and completion pair [10]:
{"prompt": "What is the capital of France?", "completion": "The capital of France is Paris."}
Templates matter even here. FLAN, a 2021 study of instruction tuning, rewrote 62 datasets with ten hand-written templates each [17]. Self-Instruct varied its prefixes, such as "Task:" and "Input:", to make the tuned model robust to formats [20]. The Flan Collection found that mixing zero-shot, few-shot and chain-of-thought templates in training improves results in all three settings [26].
Chat records and tool calls
A chat record holds one conversation as a list of messages. One fine-tuning service requires each conversation to start with a system or user message and then alternate user and assistant [2]. One model provider asks for JSON Lines in its chat format, in a file of at least 10 lines, and lets a weight of 0 or 1 switch training off for single assistant messages in a multi-turn record [8, 12].
{"messages": [{"role": "system", "content": "<instructions>"}, {"role": "user", "content": "<question>"}, {"role": "assistant", "content": "<answer>", "weight": 1}]}
Tool-calling records put a tool_calls field on the assistant message, return the tool's reply in a message with the role tool, and carry a tools column that the chat template usually uses to build the system prompt [1]. How tool use is scored once a model is tuned is covered in agent evaluation metrics and what each one tells you.
A few dialogues can change behaviour. In LIMA, a 2023 study, adding 30 multi-turn dialogue chains to the training set raised the share of excellent responses from 45.2% to 76.1% [13].
Chat templates turn messages into tokens
A model reads tokens. One model library's guide explains that messages are "converted to a token sequence, often with control tokens", and that models fine-tuned from the same base model can use different templates [3]. With the wrong control tokens, it warns, such models "would have drastically worse performance" [3].
Its training advice is to apply the chat template as a preprocessing step with add_generation_prompt=False, then tokenize the result with add_special_tokens=False so that special tokens are not duplicated [3]. The end-of-sequence token has to match the template so that replies stop correctly [4].
Records should also look like production prompts. One cloud provider's guide gives the example of a "question:" and a "context:" in training records, which production prompts then need in the same order [9]. One model provider asks that all training examples share the format expected at inference [12].
Which tokens are trained
The loss decides what the model learns from each record. By default, one open-source library trains a prompt-completion record on its completion only, and one fine-tuning service trains a chat record on its assistant messages only [2, 4]. In the library, assistant-only training is off by default and needs a chat template that marks the assistant's text [4]. The service also accepts per-message weights of 0 or 1 [2].
One study tested the other choice. Applying the loss to the instruction as well as the output helped especially with long instructions and short outputs, or with few training examples, and its authors do not propose it as a replacement for current fine-tuning [5]. Since tools differ, the setting belongs in the notes that travel with the data.
Preference records: the DPO dataset format
DPO trains on a static dataset of comparisons, each a prompt with a preferred and a dispreferred completion [21]. One open-source library stores them as prompt, chosen and rejected, and recommends explicit prompts whenever possible [1]:
{"prompt": "The sky is", "chosen": " blue.", "rejected": " green."}
One model provider's preference format trains on one-turn conversations only, and the provider reports that supervised fine-tuning on preferred replies before a preference job can significantly enhance results [23].
Pairs can be built from rankings or ratings. One model developer's 2022 study ranked 4 to 9 replies per prompt, which gives 6 to 36 pairs per ranking (derived) [6]. It trained on all pairs from one prompt as a single batch element, since pairs from one ranking are "very correlated" [6]. A 2023 distillation study rated four replies per prompt, took the one with the highest mean rating as chosen and one of the other three, at random, as rejected [7].
Raters disagree. In the 2022 study, training labelers agreed with each other 72.6% of the time [6]. The measures for that are set out in measuring label quality with inter-annotator agreement.
Unpaired feedback records
KTO needs only a signal of whether one output is desirable or undesirable for an input, stored as a prompt, a completion and a true or false label [1, 22]. In its paper, KTO matched or exceeded DPO at scales from 1 to 30 billion parameters [22]. The library warns that chosen and rejected replies "can be both good or bad", so pairs split into unpaired records need checking first [1].
JSON Lines files and the checks they need
All four hosted services read for this article take text records as JSON Lines [2, 8, 9, 10]. The format is UTF-8 with no byte order mark and one valid JSON value per line, and a blank line is not a value [11]. Under JSON's own rules, an object with repeated names makes the receiving software's behaviour unpredictable [24].
One fine-tuning service checks the encoding, the parsing of each line, the line count and the file size on the client, and checks roles and tool calls on its server after upload [2]. One cloud provider lists per-record limits from 4,096 tokens at a batch size of 1 to 32,000 tokens across its base models, and 2,048 tokens for some models at larger batch sizes [10]. Under the library's 1,024-token default, a longer chat record loses its end, where the final reply sits (derived) [4].
A validation split from the same collection measures the job [8, 9]. Size, acceptance tests and held-out data belong in the specification, as set out in how to specify a training dataset before anyone builds it.
What goes wrong in fine-tuning records
Generated records need review. In a Self-Instruct sample of 200, 54% had every field valid [20]. Quality counts for more than volume in the guidance read. One cloud provider calls quality "far more important than the quantity" [9], and LIMA tuned a 65-billion-parameter model on 1,000 curated prompts and responses [13].
Evaluation items can leak in. The Tülu 3 project checked its training prompts for 8-gram overlap with its evaluations [15], and a 2023 study found that paraphrased or translated test items pass string-matching checks [16]. How contamination distorts scores is covered in what agent benchmarks measure, and what they miss.
Fine-tuning can also undo safety behaviour. A 2023 study broke one hosted model's safety guardrails with 10 adversarially designed examples, for less than $0.20, and benign datasets also degraded safety, to a lesser extent [14]. Licenses travel with the records too. The Stanford instruction data allows non-commercial use only [25].
When Rillor builds a custom dataset, the specification names the file formats and record layout your training and evaluation pipelines read.
Read the full report
The full report sets out each record type with its fields, the chat template and loss settings, how preference pairs are built from rankings and ratings, the file rules and checks, and all 33 sources.