Choosing which high-volume tasks to give a small language model

What makes a task suitable for a small model

Belcak and colleagues argue in a position paper, first published in 2025, that agentic systems make many model calls that perform a few specialised tasks repetitively and with little variation, and that small models are often capable enough and cheaper for those calls. OpenAI's latency guide makes the practical point: well-defined tasks with limited scope are good candidates for a smaller, possibly fine-tuned model, while open-ended assistant replies are better left to a larger one.

Turned into questions about your own workload:

• Is the output constrained? A category, a JSON object with known fields or a yes or no is easier than free prose.
• Do inputs repeat in structure, such as the same form, ticket template or document type?
• Can someone tell whether an answer is right? Without a checkable answer you cannot test a smaller model.
• Does the task need facts that are not in the input? Then a larger model or retrieval is the safer choice.
• What does a wrong answer cost, and is there a fallback or a reviewer?

Screening a hypothetical task inventory

Suppose an insurer's platform team lists its five highest-volume model calls. The volumes and judgments are hypothetical, written the way a team might fill in the screen during a workshop.

• Email triage into 14 queues, 40,000 a day: constrained output, repetitive inputs, labels from past routing. Strong candidate.
• Field extraction from claim forms, 25,000 a day: fixed fields, known templates, corrected values in the claims system. Strong candidate.
• Claim file summaries for adjusters, 6,000 a day: free text, long inputs, quality judged by people. Weak candidate.
• Customer questions about policies, 3,000 a day: depends on current policy wording and broader reasoning. Better served by retrieval with a larger model.
• Fraud referral notes, 400 a day: high stakes, low volume. Even at an assumed €10 per 1,000 calls, 12,000 calls a month cost about €120, far less than building, evaluating and reviewing a model. A dedicated model would need a quality case, not a cost case.

Two candidates survive on cost. The team starts with email triage, because its labels are cleaner and a misrouted email costs minutes rather than a wrong claim decision.

Test before you train

Find out how far a small model gets without fine-tuning. Run one or two small hosted or open-weight models on held-out cases from the task, with the same prompt and parser as the current model. Each of three outcomes tells you what to do next.

• It already matches the current model: switch with a model and prompt change, and skip training.
• It comes close, with errors concentrated in a few categories: labelled examples can probably teach what is missing.
• It falls behind everywhere: the task is probably less narrow than it looked, so revisit the screen.

OpenAI's fine-tuning documentation describes a further route, and it works equally with an open-weight student trained in your own environment: tune a larger model's prompt until it performs well, keep its outputs that meet your criteria and use them to train a smaller model. That helps when labelled data is scarce, but the small model inherits the larger one's mistakes.

Do the economics with your own volume

Smaller hosted models are often priced lower per token, but the gap depends on provider and workload, and a self-hosted model costs money whether or not requests arrive. Compare monthly cost at your volume: the current cost per request times volume, against fixed serving cost plus the work of building data, training, evaluating and monitoring.

A hypothetical illustration with stated assumptions: the current model costs €3 per 1,000 triage emails, so 1.2 million emails a month cost €3,600. One GPU at an assumed €1.50 an hour costs about €1,100 a month running continuously, before the hosted fallback for peaks. The difference of roughly €2,500 a month has to pay for the engineering and retraining. Under these assumptions, at a tenth of the volume the GPU would cost more than the model it replaced. A smaller hosted model would then be worth testing, if its price, its results on your held-out cases and its data handling are acceptable.

Some reasons sit outside the invoice: response time, and data that must stay on your own systems. Hugging Face's evaluation notes add that a model which fits on one GPU is far simpler to deploy than one split across several.

Where small models disappoint

The usual failure is a task that is narrower on paper than in production. Email triage looks like 14 labels until some emails contain two requests, a legal threat in a postscript or a language nobody planned for. Measure these tail cases and route them to the larger model rather than forcing a label onto them.

The other failure is drift. Queues, products and policies change, and a small model trained on last year's labels does not notice. Decide who retrains it and on what signal before it goes live.