Deciding whether a task needs retrieval, fine-tuning or both

Knowledge problem or behaviour problem?

A quick test: if a document changed tomorrow, would the correct answer change too? If yes, the task depends on knowledge, and that knowledge should be looked up at question time. If the correct output is a pattern applied to new input, such as a category, a field layout or a tone, the task depends on behaviour.

Mixed tasks are common. A support reply needs current product facts and a consistent house style. Splitting the requirement this way tells you which part each technique should carry.

Why changing facts belong in retrieval

The original RAG paper from 2020 named two weaknesses of knowledge stored only in model weights: it is hard to show where an answer came from, and hard to update. Retrieval addresses both, because the source passage can be cited and replaced. A 2023 study by Ovadia and colleagues compared unsupervised fine-tuning with retrieval for adding knowledge and found that retrieval consistently performed better, both for facts the model had seen and for new ones. The authors also reported that models struggled to learn new facts through fine-tuning unless they saw many variations of each fact.

There are operational reasons as well. Retrieval can filter documents by user permissions, cite its sources and drop a withdrawn document at the next sync, with a delay that depends on how sync is configured. Fine-tuning alone provides none of these mechanisms: a fact in the weights carries no access list and no source link, so scoping or removing it depends on controls around the model or on another training run.

Where fine-tuning earns its place

OpenAI's model optimisation guide describes fine-tuning as an optional step after evaluations and prompt work, and lists what each method suits: supervised fine-tuning for classification, specific output formats and fixing instruction-following failures; preference optimisation for tone and style; reinforcement fine-tuning for domain reasoning where expert graders agree on the right answer. Its stated benefits include training on more examples than fit in a prompt, shorter prompts and a smaller, faster model for one task.

Fine-tuning needs labelled examples of correct output, a task definition that stays stable for a while, and held-out cases to compare against the current model. Check availability before planning. When we read that guide in October 2026, it stated that OpenAI's fine-tuning platform was no longer open to new users. Open-weight models trained on your own infrastructure are the other route.

Five tasks, decided

The table applies the test to five invented tasks. The right column is a first choice to evaluate, not a final answer.

Task (hypothetical)                                   Problem type   First choice
----------------------------------------------------  -------------  ------------------------------
Answer staff questions from HR policies revised       knowledge      retrieval with citations
each quarter
Route incoming tickets to one of 40 queues,           behaviour      prompt baseline, then
high and steady volume                                               fine-tune a smaller model
Draft customer replies in house style, quoting        both           retrieval for facts; prompt or
current product documentation                                        fine-tune for style
Extract fixed fields from supplier invoices           behaviour      prompt with a schema first;
                                                                     fine-tune only if it falls short
Answer questions about one short manual that          knowledge      whole manual in the prompt,
changes once a year                                                  if size and cost allow

The last row shows a third option. If the material is small enough, it can go into the prompt whole, with no retrieval and no training. Whether that is affordable depends on the model and its pricing, so it belongs in the evaluation as well.

An order of work that keeps both options open

1. 1

   Build the evaluation set

   Real inputs with known correct outputs, including cases the system should refuse.

2. 2

   Measure a prompt baseline

   Use a capable hosted model and a careful prompt, and sort the failures by type.

3. 3

   Fix missing or outdated facts with retrieval

   If failures are about what the model knows, add retrieval and measure again.

4. 4

   Consider fine-tuning for remaining behaviour gaps

   If the remaining failures are format, consistency, latency or cost at volume, compare a fine-tuned model on held-out cases, including cost per request.

Combining the two is legitimate, for example a model trained to answer only from supplied passages and to decline when they are missing. It also doubles what you maintain: a retrieval pipeline with its sync jobs, and a training set that has to be refreshed when the base model is replaced.