Engineering
RAG, fine-tuning or agents? A practical decision guide for business leaders
Three labels dominate conversations about AI applications: retrieval-augmented generation, fine-tuning and agents. They are often presented as competing options, as if a team must pick one. In practice they solve different problems, and the expensive mistakes come from choosing a technique before naming the problem.
Start with the problem, not the technique
Ask three questions about the work you want the system to do.
- Does the model lack knowledge it needs at the moment of answering? Retrieval is the usual answer.
- Does it need to behave differently — follow a format, adopt a house style, handle domain language? Fine-tuning is one answer.
- Does it need to take action across other systems? That calls for tools, and often an agent.
A single application can need all three. The point is to decide each layer on its own merits.
The order matters. Knowledge problems are usually cheapest to solve with retrieval and data work. Behavior problems may justify fine-tuning. Action problems are the most expensive, because they add integrations, permissions and failure modes. Start with the cheapest layer that could solve the actual problem, and move up only when the evidence says you must.
Retrieval-augmented generation: give the model the right documents
Retrieval-augmented generation, usually shortened to RAG, connects a model to a search layer. When a question arrives, the system retrieves relevant passages from your documents, database or knowledge base and places them in the prompt. The model answers from that context and can cite its sources.
Retrieval fits when:
- knowledge changes often and the model must stay current;
- the corpus is too large to fit in a prompt;
- answers must cite sources so people can verify them;
- different users should see different documents, based on permissions;
- you want to add knowledge without retraining a model.
Much of the engineering lives in the search layer, not the generation step. Parsing documents, chunking them at sensible boundaries, keeping metadata, combining keyword and vector search, reranking results and enforcing permissions all determine answer quality. A model given the wrong three paragraphs will produce a confident wrong answer, and no prompt will fix that.
Retrieval also does not teach a model a new skill or a reliable output format. It gives the model better material to work with. If the problem is behavior rather than knowledge, retrieval is the wrong tool.
Fine-tuning: teach the model a pattern
Fine-tuning continues training a model on examples of the behavior you want. It adjusts how the model responds — its format, tone, terminology or reasoning style — rather than what facts it can look up.
Fine-tuning fits when:
- the task is stable and repetitive, with many consistent examples;
- output must follow a strict structure or house style;
- domain vocabulary is unusual and prompt instructions are not enough;
- a smaller, cheaper model can match a larger one on a narrow task;
- latency or prompt length matters, and a tuned model shortens the instructions.
Fine-tuning is a poor database. Facts change, training data ages, and the model can still invent details. It also requires a maintained dataset, a repeatable training pipeline and an evaluation set that catches regressions. A fine-tuned model is another artifact to version, test and retire.
The common mistake is to fine-tune a model on policy documents and expect it to answer policy questions accurately. Use retrieval for facts that change. Use fine-tuning for the shape of the response, or for a narrow task where a smaller model needs to perform like a larger one.
Agents: let the model take actions
An agent uses a model to decide which tool to call next. It can search, query a database, create a ticket, send a message or update a record, then observe the result and continue. The model is the reasoning layer; the tools are the hands.
Agents fit when:
- the workflow spans several systems and the path is not fixed;
- decisions require reading context and choosing among options;
- a person should approve consequential steps;
- the work is high-volume and repetitive enough to justify the integration effort.
Errors compound across steps. An agent that is usually right at each step can still be wrong often enough across a long chain to need supervision. Prompt injection, over-broad permissions and unclear tool contracts create security risk. Testing is harder because the same input can produce different paths. Agents also cost more, because each step is another model call.
Design the escalation path first. The agent should know when it is uncertain, when a request exceeds its authority, and who receives the handoff. A clear escalation rule is more valuable than another tool.
If the steps are known in advance, a conventional workflow engine is usually the better choice. Reserve agency for the parts that genuinely require judgment.
How the approaches compare
| Decision | Retrieval (RAG) | Fine-tuning | Agents |
|---|---|---|---|
| What it changes | The context the model sees | The model’s behavior | What the system can do |
| Best fit | Changing knowledge, citations, permissions | Stable format, tone or domain language | Multi-step work across systems |
| Main cost | Search quality and data preparation | Training data, pipelines and re-evaluation | Integration, testing and supervision |
| Main risk | Wrong or missing retrieval | Stale examples and false confidence | Compounding errors and excess permissions |
| Typical first step | Index a representative document set and measure retrieval | Collect consistent examples and define a baseline | Map the workflow and tools, then automate one step |
They combine more often than they compete
That separation is the practical advantage of treating these as components rather than as a single choice. When an answer is wrong, the team can tell whether the search failed, the model misread the context, or the tool call did something unexpected.
Common mistakes
- Fine-tuning to add facts. Facts change; training data ages. Use retrieval and keep the model’s behavior stable.
- Building an agent for a process that a workflow engine should own. If the steps are known, code them.
- Adding retrieval without measuring retrieval. Answer quality starts with search quality.
- Choosing a technique from a vendor demo. Demos are curated. Your data is not.
- Skipping permissions. A retrieval system that ignores access control is a data leak with a chat interface.
- No baseline. If you cannot describe current performance, you cannot prove improvement.
- Treating a prototype prompt as production configuration. Version it, test it and review it.
How to evaluate the choice
Run a small, honest comparison on your own cases.
- Assemble a test set of real questions with agreed good answers.
- Try the simplest approach first: a strong prompt with good context, then retrieval, then fine-tuning, then agency.
- Measure quality, latency, cost per successful task and the human effort required.
- Test failure modes deliberately: missing documents, ambiguous questions and requests that should be refused.
- Decide what must be true for the approach to be worth its complexity.
Complexity is a cost that recurs every month. The right choice is the simplest approach that meets the quality bar on your evaluation set, not the most sophisticated one a vendor can demonstrate.
Summary
Key takeaways
- Retrieval changes what the model knows at the moment of answering; fine-tuning changes how it behaves; agents change what it can do.
- Start with the simplest layer that could work, and add complexity only when the evaluation shows it is needed.
- Fine-tuning is not a knowledge base, retrieval is not a writing style, and agents are not a substitute for a defined process.
- Evaluate each layer on real cases, including the failures you expect.
- Combine the approaches deliberately, with permissions and human review where the consequences are real.
If you are weighing these options for a specific workflow, we can help you frame the decision and test it before you commit. Our generative AI practice builds retrieval and fine-tuned systems; AI agents and automation covers tool-using workflows with human control; and AI strategy connects the choice to the business case. Tell us what you are trying to build and we will tell you which approach we would start with.
Continue reading
More field notes.
Start a conversation
Let’s turn AI ambition into systems you can rely on.
Tell us where you are today. We reply within one business day with a candid view of what is worth building — and what is not.
Or call +1 (714) 900-0052