Checking your training data before you fine-tune a language model

Why checking the data comes before training

No training setting compensates for wrong answers in the data. In Hugging Face TRL's SFTTrainer the loss is token-level cross-entropy, and for prompt-completion datasets it is computed on the completion tokens by default. The model is rewarded for reproducing your recorded answers token by token, careless labels and retired policies included.

OpenAI's fine-tuning documentation, written for a hosted platform now closed to new users, suggests starting with about 50 well-crafted demonstrations and rethinking the task or prompt if they make no difference. The advice holds for open-weight models too.

Check the recorded answers against expert judgment

Historical records look like labels but often are not. Ticket categories get chosen in a hurry, fields get corrected downstream without the fix flowing back, and fraud flags reflect rules that have since changed. Have a domain expert label a random sample without seeing the recorded answer, and sort each disagreement into one of these groups.

• Wrong label: the record is simply incorrect. Fix it or drop it.
• Guideline gap: two experts disagree because the rules do not decide the case. Write the rule before relabelling.
• Outdated answer: correct under an old policy, wrong under the current one. Filter by date or relabel.
• Hindsight label: the answer depends on information that arrived later, such as a chargeback weeks after the decision. The input must contain only what was known at decision time.

If two experts agree with each other less often than you need the model to agree with them, fix the task definition first.

Compare the examples with the traffic the model will receive

A correct dataset can still be unrepresentative. Compare a recent month of production inputs with the training set by category, language, document template, customer segment and input length.

Count rare classes separately. The Hugging Face Evaluate guidance notes that with 99 normal cases and one fraud case, a model that always answers normal scores 99 percent accuracy and catches nothing. Report the costly cases on their own.

Length is the check teams forget. In the TRL release documented at the time of writing, SFTTrainer truncates sequences longer than max_length, which defaults to 1,024 tokens, and keeps the start. Count tokens with the base model's tokenizer. If long documents exceed your limit, raise it or restructure the input, or the model trains on examples whose answer was cut away.

Record where every example came from

Gebru and colleagues proposed that every dataset come with a datasheet describing its motivation, composition, collection process and recommended uses. For a fine-tuning set, one page per data source is enough if it records the following.

• Source system and extraction query, with the date range covered
• Who produced each label: a person, a rule, a later outcome or another model
• The guideline version the labels follow
• Personal or confidential fields, and how they were removed or kept
• The basis on which the data may be used for training, as confirmed by your data protection and legal advisers

Flag model-generated or synthetic examples, check a sample of them as carefully as human labels, and keep them out of the test set, so you can remove them if a problem turns up later.

Keep test cases out of the training data

A test set only tells you something if the model has not seen it. Lee and colleagues found that train-test overlap affected over 4 percent of the validation examples in standard datasets they examined. Business data can be worse: templated emails, recurring invoices and one case logged as several tickets all produce near-copies.

Deduplicate before you split, with normalised text or a similarity threshold rather than exact matches. Then decide what the model has to generalise to. If it will meet new suppliers or templates, hold out whole suppliers. If it will mostly see future invoices from suppliers it already knows, split by time so the test period follows the training period. A random row split answers neither question.

Freeze the test set and give it an owner outside the training work.

A worked audit for a hypothetical invoice dataset

Suppose a finance team wants a small model to extract twelve fields from supplier invoices and has 8,000 invoices corrected by clerks. All figures are hypothetical, and the thresholds are agreed before anyone looks at results.

1. 1

   Sample audit

   An expert relabels 300 random invoices blind. Target: 97 percent field-level agreement. Result: 93 percent, because VAT corrections made in the ERP were never written back.

2. 2

   Fix the source

   Labels are rebuilt from the ERP's final values, and a fresh sample now clears the target.

3. 3

   Coverage table

   Scanned invoices are 9 percent of last quarter's traffic but only 80 of the 8,000 examples. The team adds 700 labelled scans, bringing them to 780 of 8,700, close to the 9 percent target.

4. 4

   Length check

   6 percent of invoices exceed 1,024 tokens. The limit is raised, and multi-page invoices get their own test slice.

5. 5

   Deduplicate and split

   Removing near-duplicates leaves 7,350 of 8,700 records, and the team checks that scans are still close to 9 percent. New suppliers join every month, so 20 percent of suppliers are held out entirely, and a second test set covers the latest month from known suppliers.

6. 6

   Datasheet

   One page records the sources, label origin, guideline version, removed personal fields and the advisers' approval.

Training starts after step six. Where outcomes are never recorded, or experts disagree on many cases, define the task before any fine-tuning.