Testing a fine-tuned model against the model it would replace

Write the acceptance criteria before the first score

Once a result is on the table, it is tempting to argue that a two-point drop on one category does not matter. Agree the bar in advance with the product owner and the domain expert who will live with the model. For a hypothetical support-ticket router replacing a hosted frontier model, the criteria might read as follows.

• Overall: macro F1 on the held-out set no worse than the current model's
• Critical slice: recall on legal complaints at least equal to the current model's, because a missed one is expensive
• Format: every output parses against the routing schema
• Latency: p95 under 800 milliseconds in the production serving setup
• Cost: lower total cost per 1,000 requests, serving and operation included

Each line names a metric, a comparison and a threshold. If a line cannot be written that precisely, the team does not yet agree on what better means.

Compare against the setup you would actually replace

OpenAI's fine-tuning documentation, written for a hosted platform now closed to new users, recommends carving out holdout data before training and using it as a control group. The principle applies to any model. For the business, the control that matters is not the untuned base model but the current production setup: the same hosted model, prompt, examples and post-processing that run today. Beating a weak baseline proves little.

Give the incumbent a fair chance. If its prompt has not been touched in a year, improve it before the comparison. Run both models through the same harness, with the same parsing and scoring code, so differences come from the models and not the plumbing.

What the held-out cases must look like

The comparison set must contain no training cases and no near-copies of them, and its split must match what the model will face: a later time period for recurring work, unseen customers or templates if those will arrive. If you oversample rare cases to measure the critical slice, report that slice separately or reweight the overall score to production proportions. The Hugging Face Evaluate guidance suggests keeping 10 to 30 percent of a dataset for evaluation, with the test set as close to production data as possible.

Small differences need care, so compare the models case by case. In a hypothetical set of 600 tickets, suppose the candidate is right on 31 cases the incumbent gets wrong and wrong on 19 the incumbent gets right. The net gain is 12 cases, or 2 percent, and a paired test such as McNemar's tells you whether that exceeds noise. Read the 19 regressions one by one, because one serious miss can outweigh the net gain.

Score the way the decision is made

Accuracy hides failure on rare classes, so use per-class precision and recall or F1 where classes are imbalanced, and score extraction field by field. For free text, OpenAI's evaluation guidance recommends combining metrics with human judgment and trusting a model as judge only after its grades agree with your experts' labels.

For outputs that resist automatic scoring, use a blind pairwise review: an expert sees two answers to the same case without knowing which model wrote which, and picks the better one or marks a tie.

Put cost and latency in the same table as quality

Hugging Face's evaluation notes state the trade-off plainly: a slightly less accurate model can be the better choice when it is faster and needs less hardware. Measure latency in the serving setup you would run, under realistic load, and take cost per request from the full bill, including idle GPU hours, the hosted fallback and retraining. The readout below is hypothetical and assumes 1.5 million requests a month, one dedicated GPU running around the clock for the fine-tuned model and the hosted model's current per-token price.

• Current hosted model: macro F1 0.86, legal-complaint recall 0.91, p95 1.6 s, €4.10 per 1,000 requests
• Fine-tuned 8B model: macro F1 0.88, legal-complaint recall 0.89, p95 0.5 s, €1.30 per 1,000 requests

Under the criteria above, this candidate fails. It is cheaper and faster, but recall on legal complaints dropped, and that line was agreed as a hard requirement. The next step is more labelled legal complaints and another run, not a debate about whether two points matter.

From a passing result to production

A pass on held-out data is an offline result. Before switching, run the candidate in shadow on live traffic while the current model still answers, and compare their decisions. Hugging Face's guidance distinguishes offline evaluation before deployment from online evaluation on production data, and a model switch needs both.

The method weakens where true outcomes arrive months later or never, and for open-ended tasks with no fixed answer. There, blind expert review carries more weight and the sample has to be larger. Keep the hosted model as a fallback until online numbers agree with offline ones.