ProdPad: Stop Paying LLMs in Tokens to Imitate Databases

Changing the orchestration layer, not the model, reduced the cost of enterprise AI tasks by 33 to 61 per cent across six foundation models in a recent study. Tokens per task fell 38 per cent and median latency by 44 per cent, while headline quality broadly held. On that workload, architecture moved the bill more than switching from the cheapest model to the most expensive one. ProdPad, whose teams built the AI products CoPilot and Conductor, has now published a detailed argument for why: large language models are being asked to do deterministic work that databases, rules engines and mathematical systems already do better, cheaper and auditably.

The core mistake, ProdPad writes, is confusing capability with suitability. An LLM will often produce a plausible answer when asked to count records, enforce a permission, calculate a score or decide whether a workflow step completed. That does not make it the right place for the logic. The prompt slowly becomes the application, and the demo works — until the system has to be tested, explained, changed and operated at scale. In some systems, a single model now acts as search engine, analytics engine, rules engine, calculator, workflow engine and user interface at once, which ProdPad calls a fragile place to put business logic.

Where do LLMs genuinely belong?

Language models earn their cost where the task itself contains judgement: interpreting an incomplete request, inferring what a user probably means, connecting concepts expressed differently, synthesising qualitative evidence and explaining results to the reader. Ask for the main themes in customer feedback and several answers are defensible; probabilistic reasoning fits because the task is probabilistic. The mistake, ProdPad argues, is assuming every step inside the same workflow carries the same uncertainty.

Consider the query: which customer segment generated the highest number of unresolved feedback items linked to roadmap initiatives last quarter? The request arrives in natural language, but the underlying work is filtering, joins, grouping and counting. An analytical engine such as DuckDB performs those operations exactly, and the query can be inspected, tested, repeated and cached. An LLM may also get it right — or may omit a record, misread the grouping, or produce a persuasive explanation of a calculation it did not perform. The model should understand the question and explain the result; the database should establish the answer.

The same split applies to mathematics. Specialist mathematical systems are deterministic, reproducible and much cheaper to run, and their output can be checked in ways a long chain of model reasoning often cannot. The LLM can work out what the user wants calculated and select the operation; the mathematical engine should do the mathematics.

What about rules, state and safety controls?

Business rules are where flexibility becomes a liability. A model can interpret a policy written in plain language and explain it, but eligibility rules, approval thresholds, permissions and safety controls need to be explicit, versioned and tested. Higher-risk markets increasingly call this deterministic policy enforcement: the model can reason around a rule, but cannot quietly rewrite it while applying it. Runtime state is the same. A chat transcript may hint at what happened, but it is a poor source of truth for outstanding approvals, completed actions or what the system may do next.

Salesforce now describes the resulting pattern as a blended orchestration model spanning agents, humans, automation tools and deterministic systems. The terminology is new; the principle — use the right kind of system for each part of the work — is not.

How should context engineering work?

Context engineering, deciding what instructions, data, tools and memories the model sees, is a better framing than simply enlarging the context window. But ProdPad stresses its failure mode: good context engineering is not only a cleverer way to show the model more. It should identify information and computation the model never needs to see. If a tool returns an exact aggregation, send the result, not thousands of source records. If workflow state already marks a step complete, do not ask the model to infer it from the transcript. The cleanest context is the context you did not have to create.

The economics back this up. A large share of inference is compensating for earlier architectural choices — asking an LLM to perform work another system could finish in milliseconds with a predictable result. Cost accumulates in context assembly, repeated history, tool exposure, retries and failure handling. Those are orchestration decisions, not properties of the foundation model. ProdPad cautions that not every optimisation will reproduce the 33 to 61 per cent figures.

Whose incentives shape the defaults?

Frontier labs build and sell model inference, so their examples emphasise larger models, longer contexts, more reasoning and more autonomous agents. Their customers optimise for work completed reliably, visibly and at lower total cost. ProdPad sees no conspiracy — vendors demonstrate the world through the product they sell — but places responsibility on product teams to decide where that product belongs.

The practical routing: bounded classification to a small model; exact aggregation to the database; policy checks to deterministic rules; calculation to a mathematical engine; hard qualitative synthesis to a frontier model; consequential decisions to a human. Good orchestration makes the system more capable partly by using less LLM inference, reserving probabilistic reasoning where probability is an advantage rather than a defect.

The question for product teams is no longer which model should do this task, but which parts of the problem genuinely require a language model at all — the decision that will define the next generation of enterprise AI.