Does This Need a Language Model?

The decision

Somewhere in your organization the same small decision gets made over and over. Which queue does this request belong in. Is this document missing something. Should this action run, or should a person look first.

You have to decide what makes that decision.

Most teams reach straight for a language model. Two cheaper things usually come first, and neither is a model. Measure what the work costs you now, and write down the rules for the cases that are obvious. What is left is a smaller problem than the one you started with.

When something is left, you have four choices. A rule, a general language model told what you want, a model trained on your own examples, or two of those working together.

This guide is for the manager who owns that call, and for the AI assistant helping them make it.

Does this apply to you?

Use it when a decision recurs often enough that its cost shows up somewhere. In a bill, in a queue, or in the hours people spend.

It applies whether people do the work today, or a language model does, or a set of rules nobody has touched in years. It also applies when nobody does it at all. That last case is easy to miss, and often the most useful: something goes unchecked today because checking it properly was never affordable.

This guide carries a few dated prices, because the conversion method matters more than the figures and you cannot see the method without them. Treat every number as a worked example to redo, not a fact to quote. Depending on the task, the price of a given level of capability has fallen between 9 and 900 times a year (Epoch AI, 2025).

For choosing among language models once you know you need one, see Which AI Model for Which Job. For deciding whether to automate a task at all, see When to Let AI Do It. For where a person should sit in the finished system, see Where Should the Human Check Go.

The options

Ordered by how much you have to build.

• A rule you write. A condition someone can read: this sender, that amount, this word in the subject line. It costs nothing per decision and answers instantly. It cannot handle a case you did not anticipate, and it needs an owner (see below).
• A general language model, told what you want. You describe the possible answers in the instructions, and paste in a few examples of each. Nothing is trained and nothing is hosted. This is the option teams most often skip past, because "we have examples" gets heard as "so we should train something."
• A model trained on your examples. You hand over labelled past cases and get back a model that has learned your pattern. Cheap per decision, often very accurate on work resembling what it learned from, and blind to anything else. Vendors call this fine-tuning.
• Two of these working together. A cheap one handles what it can and passes the rest up. This is its own option, with its own cost and its own ways of failing.

The first three often all appear in one working system. A rule catches the obvious cases, something cheaper handles the routine, and the most capable option sees only what is left over.

What actually matters

• How many possible answers there are. This predicts more than anything else on this list. A yes-or-no question and a question with eighty answers are different problems.
• Whether you have labelled examples. Labelled examples are past cases where someone recorded the right answer. You often have them without realizing: a year of resolved tickets carries the queue each one ended up in. They are useful to every option here, not only to training.
• Whether the input keeps changing. Work where next year resembles last year differs from work where someone is actively changing their behaviour.
• Whether you can check the answer cheaply. A required format, a calculation, or an exact match opens up your options. Needing a second judgment to judge the answer narrows them.
• What a wrong answer costs, and whether you can take it back. A misfiled ticket and an irreversible action are different risks, and they lead to different designs.
• Where the text is allowed to go. If the material carries customer or personal data, your contracts and your regulator may decide this before cost does. Settle that first, because it rules options out rather than ranking them.
• Volume. Not as a measure of how sophisticated your answer should be. It decides which cost dominates: the bill, the engineering, or the people reviewing output.

How the options compare

Rules are cheap to run and expensive to keep. This is the cost nobody prices. In an earlier generation of rule-based automation, EY put the share of initial projects that fail at "30 to 50%" (EY, 2016). Forrester found $3.41 spent on services for every $1 of software (Forrester, 2019). A rule set written for the obvious cases accretes, one exception at a time, usually added by whoever is on call. Write the obvious rules, give them an owner, and stop there.

Two more cautions on rules. Harvesting the easy cases raises the average difficulty of everything downstream, so what remains is not simply a smaller version of what you had. And where someone is working against you, as in fraud or abuse, a readable rule is a published attack surface.

Ten labelled examples per category go a long way. On a banking benchmark with 77 possible answers, a trained model reached 83.42% accuracy from 10 examples per category (Casanueva et al., 2020). The full training set took it to 93.66% (Casanueva et al., 2020).

Read that pair twice, because both framings are true. Ten examples got most of the way up the scale. The full set still removed about three fifths of the remaining errors, which is the difference between one in six wrong and one in sixteen. Which framing matters depends on what a wrong answer costs you.

That benchmark is also the friendly case: balanced categories, cleanly separable, professionally labelled. Real work has one category taking half the volume, several that overlap, and cases two people would file differently. Imbalance and overlap push the number of examples you need up sharply.

Every accuracy figure on this page comes from a curated benchmark, and curated benchmarks are the easy end. Treat published accuracy as an optimistic bound and published decay as a conservative one.

Trained models are excellent until the world moves. Researchers evaluated 19 trained language models across eight tasks. Average performance fell by 20.05% once the input stopped resembling the training data (Yang et al., 2023). Humans given the same shift lost 7.82%, against 16.33% for the best model tested (Yang et al., 2023).

A security study shows what that looks like in practice. A phishing detector lost 28.0 points of F1 when the people it was detecting switched tools for writing their emails (Gutierrez et al., 2026). F1 is a combined measure of precision and recall. Nothing about the detector changed. The world on the other side of it did.

When a score drops, check the threshold before you replace the model. A trained model does not answer yes or no. It returns a score, and a person picks the score above which you act. That number is the threshold. In the study above, researchers recalibrated the cut-off on a labelled sample of the new material, a few hundred examples. That recovered 86% of the lost score, and they retrained nothing (Gutierrez et al., 2026).

There is a cheap test for whether the threshold is the problem. Ask whether the model still ranks cases correctly and only the cut-off has slipped, or whether its ranking has fallen apart. If the ranking survived, recalibrate. If it did not, the model itself has broken, and no threshold will save it. Note also that recalibrating needs freshly labelled examples, which is usually the expensive part of retraining too.

A score from one system does not mean the same thing as a score from another. It may not mean the same thing across two kinds of question inside one system, or across a version change. Check a supplied score against your own recorded outcomes before you act on it.

Combining a cheap decider with an expensive one costs something before it saves anything. You pay the cheap one on everything. You only avoid the expensive one on the part the cheap one finishes alone. Academic tests found savings from 50% to 98%, depending on the dataset (Chen, Zaharia and Zou, 2023). Those tests tune the arrangement per dataset, so treat the top of that range as a ceiling. The bottom is not a floor either: arrangements that cost more than they saved do not get written up.

The arithmetic is unforgiving, and you can do it before building anything. Say the cheap option costs a fifth of the expensive one. It must then finish more than a fifth of the work alone before you save a penny. Below that line it is a surcharge. Teams find this out after building, because the share it will finish alone is the one number nobody estimates first.

A check that runs before every action has different arithmetic again. It pays its cost on everything and earns it back only when it stops a real problem, which is rare. Its false alarms are the expensive part, not its compute. One study tested five commercial moderation services across four curated hate-speech benchmarks and found false-alarm rates from 1.2% to 76.8% (Kolla et al., 2025). A rate measured on someone else's material tells you very little about yours.

A check that blocks by default is also a target. Researchers built an adversarial string roughly 30 characters long that made one safeguard model reject over 97% of user requests (Zhang et al., 2025). If being blocked is your failure mode, causing blocks is the attack.

Escalating to a person has the clearest economics, because the thing you are replacing is expensive. A person handling one case costs many times what a machine attempt costs. So the share the machine must finish alone to pay for itself is small.

The risk here is not cost. The share a machine handles is easy to measure, and easy to improve by making the path to a human hard to find. Klarna publicized a large deployment. It later said cost "seems to have been a too predominant evaluation factor" and that quality suffered (Siemiatkowski via Customer Experience Dive, 2025). Notably, Klarna did not switch the system off. As of May 2025 its chatbot still handled two-thirds of inquiries (Customer Experience Dive, 2025). What the company restored was the human path.

The managed middle is worth pricing before you assume it. Cloud providers sell custom classification services that look like the obvious low-effort choice here. They price in units that rarely compare to anything else on your list, so convert before you choose. Here is the conversion for one decision that reads about 2,000 characters, done in September 2026.

AWS Comprehend custom classification bills in units of 100 characters. That is 20 units, at $0.0005 each in batch mode, so one cent a decision, or $10 per thousand. Azure AI Language custom classification bills in records of up to 1,000 characters, so two records. It reached the same figure by that different route. A small general language model reads those 2,000 characters as roughly 500 tokens, which came to about 3 cents per thousand decisions.

That gap is wide enough to change decisions, so check it rather than assuming the managed option is the economical one. Two things the figure hides, both arguing the same way. The batch price is the cheap one: an always-on endpoint bills by the second whether or not traffic arrives. And both managed services charge separately to train and to host.

Running a model yourself is a volume decision. The Organisation for Economic Co-operation and Development (OECD) found no saving from self-hosting below 100 million tokens a month (OECD, 2026). A token is a fragment of a word. Those figures assume buying hardware; the same paper says renting can pay off at more modest volumes, under some assumptions (OECD, 2026).

Your supplier's clock is shorter than your system's life. The notice periods are set out in Build vs Buy and Which AI Model for Which Job. What matters here is that a model you trained on your own data is not exempt. OpenAI's documentation states that running a fine-tuned model stops when the model it was built on goes away (OpenAI, 2026).

Cohere went further in September 2025. It retired fine-tuning for its classification models, and "previously fine-tuned models will no longer be accessible" (Cohere, 2025). Customers lost the trained thing itself, not just the ability to make more. Your labelled examples are the part of that investment that survives. Keep them somewhere you control.

When each one wins

A rule wins when:

• The answer is obvious from something you can point at: a sender, a threshold, a field that is present or missing.
• You need to explain the decision afterwards to a customer, an auditor, or a court.
• The occasional case it cannot handle costs little, because those cases fall through to something else.

A general language model, told what you want, wins when:

• Volume is modest enough that the per-decision bill is noise against the cost of building anything.
• The answers are open-ended, or the list of possible answers keeps changing.
• You have no engineers to spare, or no appetite for owning a threshold and a retraining cycle.
• Your labelled examples are contaminated, few, or stale. You can still paste the good ones into the instructions.

A model trained on your examples wins when:

• The decision has a fixed list of possible answers, and the list is long.
• You have many clean labelled cases per answer, and a way to keep getting more after the model goes live.
• Volume is high enough that the difference in cost per decision outweighs building and keeping it.
• You have a way to notice when it stops working, and somebody owns that.

Two working together wins when:

• The work splits into a large pile of easy cases and a tail of hard ones. If everything is moderately hard, the cheap component has nothing to take.
• You can check the cheap component's answer without asking something expensive to judge it.
• Wrongly handling something cheaply costs well below just using the expensive option throughout.

That last condition rules this option out more often than the other three combined. Work through the arithmetic in steps 6 and 7 before you commit to it.

How to decide

Work through these in order, for one recurring decision at a time. Steps 1 to 3 often end the exercise.

1. Write down what the work costs you today, and how often it is wrong. Per decision, including people's time. A rough order of magnitude is enough. The error rate matters as much as the cost, because without it you cannot say whether any replacement is good enough.
2. Settle where the text is allowed to go. Contracts, personal data, and your regulator rule options out rather than ranking them. Doing this first saves pricing something you cannot use.
3. Write the rules for the obvious cases, and give them an owner. A rule you can read beats a prediction you cannot, when the answer is not in doubt. Stop at the obvious ones. Count what is left, because that is the real size of your problem.
4. Decide what good enough looks like, before you look at options. A number you would accept, agreed with whoever lives with the results. Without it, step 8 produces a measurement nobody can act on.
5. Count the possible answers, and look at what your labelled examples are actually worth. Check whether the recorded answer is the one that was knowable at the time, or one added later. Tickets get re-routed; the final queue is not always the right first guess. Contaminated labels are still useful as examples in instructions, and dangerous as training data.
6. If you are weighing two components, do the division first. Work out each option's cost per decision, counting only what you pay to run it. Divide the cheap one by the expensive one. That fraction is the share of work the cheap one must finish alone to break even. If checking the cheap answer costs something too, subtract that from the expensive option before dividing.
7. Then test it against a pessimistic error rate. Of the work the cheap component keeps, assume one case in ten should have gone up and did not. Multiply by what one such mistake costs. Compare against the mistakes the expensive option would have made anyway, not against perfection. If the answer still says no, stop now and save the pilot.
8. Set every threshold on your own data, and name who owns it. A number carried over from a vendor's default was set against material you do not have. Write down the date it was set and when someone will look again.
9. Decide how many results a person checks, and keep the two reasons separate. To estimate an error rate, a few hundred is the floor, and it tells you as much at 30,000 decisions as at 3,000. To catch bad outcomes before they reach someone, you need a share of the flow, and that share grows with volume. Sample each category, not just the total: an overall figure can look healthy while one category is entirely broken.

A worked example

A team routes about 4,000 support tickets a month into twelve queues by hand.

Step 1. Handlers spend roughly a minute each sorting. At a loaded rate of $40 an hour that is about 67 cents a ticket. Four thousand of them is over eight working days a month. Nobody knows the current error rate, so they sample fifty and find handlers disagree on about one in eight.

Step 2. Tickets carry customer names and order numbers. Their contract with their existing helpdesk vendor already covers processing this text, and adding a new processor needs a review. That rules out nothing yet, but it dates the decision.

Step 3. Three senders account for a fifth of the volume and always go to the same queue. A rule takes an afternoon, and the support lead owns it. The problem is now 3,200 tickets.

Step 4. They agree that routing needs to beat the handlers' own one-in-eight disagreement rate, and that a wrong queue costs a few minutes of somebody's time.

Step 5. Twelve answers, which is not a long list. The helpdesk records the queue each resolved ticket ended in, but a check shows a good share were re-routed after diagnosis. Those labels describe where a ticket finished, not what its opening text supported. Useful as examples, risky as training data.

Steps 6 and 7. Skipped for now. They are choosing one component.

The choice. At 3,200 tickets a month, the per-decision bill is noise whichever option they pick. What is not noise is building something. A trained model means labels they have just found to be contaminated, a hosting bill, a threshold to own, and a retraining cycle. So they write the twelve queue descriptions into the instructions of a general model and paste in five good examples of each. They run it against last month's tickets before it touches anything live.

Steps 8 and 9. They set the score above which routing happens automatically using their own tickets. The support lead owns it, with a diary note for six months. A handler checks a few hundred routed tickets to estimate the error rate, sampled across all twelve queues rather than at random. Every low-score ticket goes to a person.

The answer to this page's title, for this team, is yes. Not because a language model is clever, but because at their volume anything they build costs more than it saves.

Watch out for

• A creeping escalation rate. If the share of work reaching the expensive component is rising, the cheap one has gone stale. This shows up in the bill long before accuracy. Watch it falling too: a model that has become confidently wrong escalates less, and the bill improves while the work gets worse.
• Rules nobody owns. They accrete one exception at a time and outlive the person who wrote them. Give the set a name, an owner, and a date.
• Checking only the results that passed. If nobody looks at what a filter rejected, nobody can measure its false-alarm rate. The people affected rarely complain. They retry, or they leave.
• Free labels that are not free. Labels drawn from resolved work record where something ended, not what was knowable at the start. Check before training on them.
• Losing your label supply. Once a model routes the work, people only see what it sent them, and the independent record you were learning from stops arriving.
• Counting a deflection as a resolution. A deflection is work the machine handled without a person. If the same customer comes back through another channel two days later, you paid for the machine attempt and the human one.

Sources

• Epoch AI — LLM inference prices have fallen rapidly but unequally across tasks, 2025. Across all of these benchmarks and performance thresholds, we found prices declining between 9x per year and 900x per year, with a median of 50x per year. View source · verified 2026-07-01 · primary
• EY — Get ready for robots: Why planning makes the difference between success and disappointment, 2016. as many as 30 to 50% of initial RPA projects fail. View source · verified 2026-09-19 · primary
• Forrester (via CIO.com) — 4 surprising RPA research findings from Forrester, 2019. for every dollar spent on RPA software, $3.41 was spent on services to make it work. View source · verified 2026-09-19 · ⚠ secondary mirror
• Casanueva, Temcinas, Gerz, Henderson and Vulic — Efficient Intent Detection with Dual Sentence Encoders, 2020. Table 3: Accuracy scores (×100%) on all three intent detection data sets with varying number of training examples (10 examples per intent; 30 examples per intent; Full training data). View source · verified 2026-09-19 · primary
• Yang, Zhang, Chen et al. — GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective (Findings of ACL), 2023. all pre-trained models involved in GLUE-X show significant performance decay under the OOD test compared to the ID performance (20.05% decay in average). View source · verified 2026-09-19 · primary
• Yang, Zhang, Chen et al. — GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective (Findings of ACL), 2023. The average performance decay between in- and out-of-domain tests of humans (87.10% - ID vs. 80.14% - OOD) is significantly lower than models, even for the best-performing model with the lowest performance decay (7.82% vs. 16.33%). View source · verified 2026-09-19 · primary
• Gutierrez, Villegas-Ch and Govea — Cross-model evaluation of phishing detectors against LLM-generated emails (Frontiers in Big Data), 2026. Recalibrating the threshold on a small target subset reduced the gap to 4.0 percentage points (an 86% reduction), and an aggregated-pool detector achieved F1 = 0.997 on each generator. View source · verified 2026-09-19 · primary
• Chen, Zaharia, Zou (Stanford University) — FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, 2023. Table 3 displays the overall cost savings of FrugalGPT, which range from 50% to 98%. This is feasible because FrugalGPT identifies the queries that can be accurately answered by smaller LLMs and, as a result, only invokes those cost-effective LLMs. View source · verified 2026-09-13 · primary
• Kolla, Salminen et al. — Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeting Hate Speech (CHI '25), 2025. HateXplain Dataset: OpenAI performed best for most metrics, including AUC (87.3%) and F1 (77.2%), while Google had the lowest F1 (58.9%) and highest FPR (76.8%). View source · verified 2026-09-19 · primary
• Zhang, Xiong and Mao — LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks (ACM CCS LAMPS Workshop), 2025. the attacker can use our optimization process to automatically generate seemingly safe adversarial prompts, approximately only 30 characters long, that universally block over 97% of user requests on Llama Guard 3. View source · verified 2026-09-19 · primary
• Customer Experience Dive — Klarna reinvests in human talent for customer service, quoting Sebastian Siemiatkowski to Bloomberg, 2025. "As cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality," Siemiatkowski told Bloomberg. View source · verified 2026-09-19 · ⚠ secondary mirror
• Customer Experience Dive — Klarna reinvests in human talent for customer service, 2025. The chatbot still handles two-thirds of all customer inquiries. View source · verified 2026-09-19 · ⚠ secondary mirror
• OECD — Benefits of AI openness: OECD discussion paper for the G7, 2026. Self-hosting of open-weight models becomes cost-effective only at scale. ... The economic benefits of self-hosting are not evident for small workloads (less than 100 million tokens per month). Under medium workloads (1 billion tokens per month), private hosting becomes cheaper than pay-as-you-go cloud services only after 2.5 years (around 30 months). ... Table 3.3. Breakeven months for cloud API vs private GPU installation scenarios ... Medium (500 million tokens per month) 30.4 ... Under certain assumptions, GPU rental can therefore be cost-effective even at relatively modest usage levels. View source · verified 2026-09-13 · primary
• OpenAI — Model deprecations (API docs), 2026. Fine-tuned models created from these base models are not affected by this deprecation, but you will no longer be able to create new fine-tuned versions with these models... Inference on fine-tuned models will be disabled only when the underlying base model is deprecated. View source · verified 2026-07-05 · primary
• Cohere — Deprecations, 2025. Fine-tuning for the command-light , command , command-r , classify , and rerank models is being retired. This covers both the Cohere dashboard and API. Previously fine-tuned models will no longer be accessible. View source · verified 2026-09-19 · primary