RAG, fine-tuning, or prompting? An honest guide to building with LLMs

Almost every team we talk to opens the same way: "We want to fine-tune a model on our data." Most of the time, fine-tuning is the wrong first move — and an expensive one. The three techniques you hear thrown around (prompting, retrieval-augmented generation, and fine-tuning) solve genuinely different problems, and reaching for the heaviest tool first usually buys you cost and complexity without buying you quality.

The decision is less mystical than it sounds. It comes down to one question: what is actually missing from the model's answer? Is it instruction, knowledge, or behaviour? Get that diagnosis right and the technique chooses itself.

Start with prompting — it's cheaper than you think

Prompting is the default, and you should exhaust it before anything else. A well-structured prompt with clear instructions, the right examples, and an explicit output format will solve a surprising share of use cases on its own. It also has the shortest feedback loop in the business: you change a sentence, you see the result in seconds. No pipeline, no retraining, no infrastructure.

Prompting is the right tool when the gap is about how the model should behave, not what it knows. Reach for it first when:

• The task is general reasoning, rewriting, summarising, or classification on text the model already understands.
• You need a consistent tone, structure, or output schema (JSON, a fixed template, a rubric).
• The knowledge required is either common or small enough to paste straight into the context window.
• You're still discovering what "good" looks like — prompting lets you iterate before committing to any infrastructure.

Its limit is equally clear: prompting cannot teach the model facts it was never trained on, and stuffing huge documents into every request gets slow and costly. When the prompt starts ballooning with reference material, that's your signal to move on.

Use RAG when the gap is knowledge

Retrieval-augmented generation (RAG) is the answer when the model needs facts it doesn't have — your internal docs, a product catalogue, support tickets, policies, or anything that changes over time. Instead of baking knowledge into the model, you store it in a search index, retrieve the most relevant chunks at query time, and hand them to the model as context. The model reasons; your data supplies the truth.

RAG is the right call when:

• Answers must be grounded in a specific, private, or frequently updated body of knowledge.
• You need citations — RAG can point users back to the source passage it relied on.
• The knowledge base changes often; you update an index, not a model, which is far cheaper.
• Hallucination is a real risk and you need to constrain the model to documented facts.

The catch is that RAG lives or dies on retrieval quality. If the search step returns the wrong chunks, even the best model will confidently answer from bad context. Most "the AI gave a wrong answer" problems we debug turn out to be retrieval problems — chunking, embeddings, or ranking — not the language model at all. RAG is plumbing as much as it is AI, and the plumbing is where the engineering effort goes.

Fine-tune only when the gap is behaviour

Fine-tuning genuinely earns its place, but for a narrower set of reasons than people expect. It teaches the model a consistent style or skill — a specialised tone of voice, a niche output format, a domain-specific way of reasoning that prompting can't reliably enforce. It is not primarily a way to add knowledge; for facts, RAG is almost always the better, cheaper, more maintainable choice.

Consider fine-tuning when:

• You need a very specific behaviour or format repeated reliably at scale, and prompts get fragile or enormous trying to enforce it.
• You have a high-quality, labelled dataset of input/output examples — quality and consistency matter far more than volume.
• Latency and cost matter and you want to shrink long instruction prompts into the model's weights.
• You've already proven the use case with prompting and RAG, and fine-tuning is an optimisation, not a first attempt.

Fine-tuning also carries ongoing cost: every time a new base model ships, your tuned version risks falling behind, and you own a retraining pipeline forever. That maintenance burden is real and worth budgeting for honestly before you start.

These aren't either/or — and evaluation is the real work

In production, the strongest systems combine all three: a sharp prompt, RAG for grounding, and occasionally a fine-tuned model for a specialised step. A good rule of thumb is to climb the ladder in order — prompt first, add retrieval when knowledge is missing, fine-tune only when behaviour still won't hold — and stop the moment quality is good enough.

Whatever you choose, the technique is the easy part. The hard, unglamorous work is evaluation: defining what "correct" means for your task, building a test set of real examples, and measuring accuracy, cost, latency, and safety every time you change something. Without that, you're guessing — and guessing is how AI features quietly degrade in production. If you'd like a sober second opinion on where your LLM project really sits on that ladder, our AI & Machine Learning team is built for exactly that conversation.