When LLMs are the right tool (and when they're definitely not)

We turn down about 30% of engagement inquiries because the proposed use case doesn't need an LLM. The team has heard about AI capabilities, has budget, and has a problem. The problem is usually real. The LLM is usually the wrong tool for it.

This isn't ideology. We sell LLM engagements; we'd benefit from saying yes to all of them. We say no because the engagements that say yes to the wrong tool end badly — the model under-performs, the team blames the model, and the firm that pushed the engagement carries the reputational hit. Better to redirect.

Here's the triage we run on the first call.

When LLMs genuinely earn their cost

An LLM is the right tool when the input is unstructured natural language AND the output requires understanding meaning, not just patterns. Both conditions matter.

Strong fits:

Customer support triage and response. Free-text inputs, judgment about intent, drafting natural-language replies.
Document classification at the conceptual level. "Is this contract a license, services agreement, or NDA?" — requires understanding what's actually in the text, not just keyword matching.
Question answering over a knowledge base. Free-text questions, free-text answers, fact retrieval grounded in source material.
Code generation, especially as a programmer assistant. Open-ended specification → structured code; quality matters but isn't binary.
Summarization of long documents. When the summary needs to reflect the actual semantic content rather than just extracting keywords.
Drafting and editing prose. Marketing copy, structured reports, legal language refinement.
Conversational interfaces. When the user benefits from natural-language back-and-forth rather than menu navigation.

These engagements end well because the model's strengths align with the problem.

When LLMs are the wrong tool

We see the same misapplications repeatedly.

Structured-data tasks that have shipped algorithms for 30 years. "Use an LLM to predict which customers will churn." Logistic regression on your customer data will outperform an LLM by a wide margin, runs in milliseconds, costs nothing, and the audit story is clean. We've watched teams spend three months and $100K on an LLM-based churn predictor that got beat by a 50-line scikit-learn model.

Numerical computation. "Use an LLM to forecast next month's sales." LLMs are bad at math at small scale, hilariously bad at large scale, and have no inductive bias for time series. Use Prophet or a classical model.

Exact-match search. "Use an LLM to look up records in our database." SQL exists. Elasticsearch exists. Don't pay an LLM to do a join.

Deterministic transformations. "Convert this XML to JSON using an LLM." A 10-line script with a real parser is correct 100% of the time. The LLM is correct 99% of the time and gives you the 1% to debug.

Real-time pricing or trading decisions. Latency, determinism, and auditability all matter more than the LLM's strengths. Use the model that ML/quant teams already use.

Hard fact retrieval where being wrong is unacceptable. Medical dosing, legal citations, regulatory thresholds. RAG mitigates this but doesn't eliminate it. Engagements where "occasionally wrong" is unacceptable need a different architecture, often involving the LLM only as a draft-suggestor that a human reviews.

Anything where a regex would work. Phone number extraction. Email validation. License plate parsing. The LLM is more expensive and less reliable than a five-line regex you can test exhaustively.

When LLMs are defensible but not optimal

A middle category: tasks where an LLM works, but a simpler tool would have worked too. We'll do these engagements if the client understands the tradeoff.

Classification with limited training data. A fine-tuned BERT classifier ships in two weeks and outperforms an LLM at 100× lower inference cost. But if the client has zero training data and needs to start tomorrow, an LLM with a few-shot prompt gets to 80% accuracy on day one. Trade-off: ongoing inference cost is higher; time-to-first-result is much lower.

Entity extraction. A spaCy pipeline does this well and runs in milliseconds. An LLM with a structured-output prompt also does it well at higher cost. Use the LLM if the domain has rare entities (medical compound names, unusual legal concepts) that the off-the-shelf NER doesn't know about and you don't want to train a custom one.

Simple text generation. Template strings with variable substitution work for most generation tasks. LLMs handle the harder cases (varied tone, edge cases) more gracefully.

Pick the LLM here if the client values flexibility over efficiency.

The triage script

We ask three questions on the first call:

1. What does success look like? If success is "X happens with Y reliability" and X is well-defined, often a classical approach works. If success is "the output is contextually appropriate to varied inputs," the LLM is the right place.

2. What does failure look like? If failure is "wrong answer becomes a regulatory issue / customer harm / financial loss," LLM is suspect. If failure is "minor inconvenience the user can correct," LLM is fine.

3. How does the current system handle this? If a script + database + simple rules would suffice, that's the answer. LLMs are warranted when the current system needs to interpret natural language or generate novel responses, not when it needs to follow deterministic rules.

If two of the three answers point at "classical tooling would do this," we recommend that and decline the engagement. If at least two point at "this needs language understanding," we proceed.

What this means for budget conversations

A common follow-up: "We need to justify a budget; doesn't 'use an LLM' sound better than 'use logistic regression'?" Sometimes yes. We acknowledge that. Budget approval narratives matter and "AI" still moves discretionary dollars in 2026.

We're honest about this trade-off. If the LLM is technically wrong but politically right, we'll say so explicitly and let the client decide. The engagement still has to ship value; we'll deliver the LLM solution, but we'll also build in measurement so the client can compare to a classical baseline. If the classical baseline wins (it often does), we deliver both and let the client choose what to deploy.

A pattern that reads as LLM-shaped

A heuristic: if you can write a clear test suite where each case has a single right answer, your problem is classical. If you can't — if a domain expert would look at the input and say "this is a judgment call," that's LLM territory.

Examples:

"Extract phone numbers from text." Test-suiteable. Classical.
"Determine whether this customer email is angry, frustrated, or neutral." Judgment-call. LLM territory.
"Sum the numbers in this column." Classical.
"Identify the implicit assumption in this argument." Judgment. LLM.
"Find files modified in the last 24 hours." Classical.
"Summarize what the customer needs from this support ticket." LLM.

When in doubt, run a few test cases through both approaches. The classical one is fast to build; if it works, you've saved months of engagement effort.

What you avoid by saying no

Two outcomes we've watched unfold on engagements that should have been declined:

The model never reaches acceptable accuracy. Six months of fine-tuning, prompt engineering, and architectural changes don't move the needle because the underlying task wasn't a language task. The team eventually gives up and either ships the inadequate model or rebuilds with classical tooling, having burned the budget twice.
The model works "well enough" but the cost story is broken. API spend balloons because every customer interaction triggers an LLM call when a $0.0001 SQL query would have done the same job. The deployment is technically functional, but the unit economics make it permanently unprofitable.

Both end the same way: the firm that recommended the LLM gets blamed, and the client's appetite for genuine LLM engagements drops to zero. Saying no early avoids both.

When to say yes

Once you've ruled out the wrong-tool failures, the remaining LLM engagements tend to share a shape:

The input is text the user wrote in their own words
The output is text the user will read or act on
The mapping between them benefits from understanding meaning rather than matching patterns
"Pretty good" answers are more valuable than "occasionally exact" ones
The user accepts that the model is a draft generator, not an oracle

When all five apply, we engage with confidence. The model does what it's good at, the client gets value, and the engagement compounds into a long-term relationship.