The architecture choice starts with the job
A growing service business wants an internal assistant. Staff expect it to answer questions about current policies, use an approved tone and classify each request for the right team. Those requirements sound like one AI project. They're three different jobs.
Current policies must come from an approved source. Answers may need citations. Tone and classification are repeatable behaviours. A short instruction can shape some of that behaviour without changing a model at all.
The useful first question isn't “Should we use RAG or fine tuning?” It's “What must the system know at the moment of the request, and what must it do consistently?” If the use case, owner or decision boundary remains unclear, complete an AI readiness assessment before selecting an architecture.
This distinction matters because every layer creates work after launch. Retrieval needs governed sources, access rules and monitoring. Fine tuning needs suitable examples, training and regression tests. Combining them creates both sets of obligations plus an integration boundary. The architecture should earn that burden through a clear requirement.
What each approach actually changes
Prompt context supplies bounded material
Prompt context puts instructions, examples or source material into the request sent to the model. Google Cloud's guidance on using business data with language models describes this as the simplest way to supply static information without adapting the model.
It suits a bounded task when the required material is small enough to include, stable enough to maintain and safe to send on every relevant request. A team could provide a response template, three approved examples and the details of one case. As the material grows or changes frequently, manually choosing and maintaining that context becomes a system problem of its own.
RAG finds context when the request arrives
Retrieval-augmented generation, usually shortened to RAG, searches an external collection for material relevant to the request and places the retrieved passages into the model's context. The model itself hasn't learned the latest policy. The application supplies evidence when it asks the model to answer.
AWS's comparison of RAG and fine tuning favours retrieval for question answering over changing documents and notes that a RAG response can reference its information source. That makes retrieval a strong starting point for current, attributable business knowledge. It doesn't make every response accurate. The source, retrieval result and generated answer still need evaluation.
Fine tuning changes learned behaviour
Supervised fine tuning trains a model on examples of desired inputs and outputs. Google Cloud gives classification and structured output as suitable examples. AWS also notes that tuning can help outputs follow an organisation's style, while requiring training expertise and taking time when documents change.
That makes fine tuning a behavioural option, not a dependable filing cabinet for changing facts. It can be useful when a well-defined task repeatedly resists clear instructions and examples, and when the business has enough representative training and evaluation data. It can also be combined with retrieval, though a combined design should give each layer a separate, testable job.
Choose by knowledge freshness and required behaviour
The decision becomes clearer when two requirements are considered together. One axis asks how much the answer depends on fresh, attributable knowledge. The other asks how much learned, repeatable behaviour matters beyond what a prompt and examples can reliably produce.
Knowledge and behaviour determine the starting route
Learned behaviour increases
Fine tuning
Repeatable task behaviour dominates while source freshness stays low
Combine approaches
Current approved facts and repeatable behaviour are both material
Prompt context
Small stable context and limited behaviour change can stay in the request
RAG
Current attributable knowledge matters more than learned behaviour
Fresh attributable knowledge increases
This is a qualitative Off Piste framework informed by the AWS and Google Cloud guidance. It isn't a measured score or an exhaustive architecture model. Context size, permissions, model capability, latency, cost and the evaluation result can move a use case across a boundary.
Start in the bottom-left when a short instruction and stable context can do the job. Move right when the system must find current evidence or show where an answer came from. Move up when repeatable task behaviour remains inadequate after good instructions and examples. Use the top-right only when the facts and behaviour are independently material.
Changing policies usually point to retrieval
Consider an assistant answering staff questions about service eligibility. The policy changes several times a year, exceptions differ by service and staff must be able to open the approved source behind an answer. Retrieval gives the application a way to find the current policy at request time and show its provenance.
The work sits around the model. The team must decide which policy is authoritative, remove or label superseded versions, carry permissions into retrieval and test questions where similar rules conflict. If retrieval is the starting route, the AI-ready knowledge base guide covers that preparation without assuming that uploading files creates authority.
Security belongs in the design decision too. The OWASP RAG Security Cheat Sheet identifies risks across document ingestion, poisoning, embedding and index integrity, inherited access controls, retrieval, source attribution and output validation. RAG redistributes risk across a pipeline. It isn't a risk-free factual fix.
An evaluation should therefore include routine questions, conflicting sources, expired policies, missing evidence and users with different access rights. Check whether the right evidence was retrieved before judging the prose of the answer. A citation only helps when it opens the source that actually supports the nearby claim.
Consistent behaviour can justify fine tuning
Now consider a team that classifies incoming service requests into a stable taxonomy and produces one compact JSON record for a downstream system. The categories are settled, the desired outputs can be demonstrated and the task runs often enough for small inconsistencies to create real review work.
That's closer to the supervised fine-tuning use cases in Google's provider guidance. The team can assemble representative input-output pairs, hold back a test set and compare a tuned model with its prompt-only baseline. Approval should depend on task accuracy, valid output structure, performance on difficult cases and safe handling of requests outside the taxonomy.
Updates still have a cost. A category change can require new examples, retraining, regression testing and a rollback path. The team needs to retain the training data, configuration, model version and evaluation result so it can explain what changed.
Brand voice is less tidy than classification. Before training, build an AI-ready brand voice system with explicit principles, approved examples and review criteria. Prompting may prove sufficient. If it doesn't, those assets become the evidence needed to evaluate whether tuning improves behaviour rather than merely making it different.
Mixed requirements need a reason for every layer
Return to the internal assistant from the opening. It must answer from current approved policies, write in a consistent service style and route each request. Retrieval has a clear job in supplying current evidence. Prompt instructions and examples can shape the answer and routing first. Fine tuning becomes a candidate only if a defined behaviour repeatedly fails and the expected improvement justifies its training and operating burden.
Provider guidance from AWS and Google Cloud confirms that retrieval and tuning can be combined. It doesn't mean a combination is automatically better. The team must test retrieval quality, answer support and behavioural consistency separately, then test the integrated system.
Before approval, name the requirement served by each layer. Remove any layer whose contribution cannot be evaluated. Record which source wins when evidence conflicts, which users may retrieve it, which behaviours must remain stable and who owns changes to each part. The AI governance policy guide provides a wider home for approvals, data boundaries and accountable ownership.
What the decision costs after launch
Retrieval creates a source operation. Someone must approve additions and removals, preserve access rules, detect ingestion failures and retest questions affected by a policy change. The AI knowledge base monitoring guide separates source, retrieval, response and access failures so the right owner can repair them.
Fine tuning creates a model-change operation. Someone must curate examples, manage training data, record model and configuration versions, compare releases against a stable test set and decide when a behavioural change warrants retraining. A supplier changing model availability or tuning support can also force a new evaluation.
A combined system adds interaction risk. Better retrieval can still produce a poor answer if tuned behaviour overrules an evidence boundary. A better classifier can still route a request incorrectly when the input falls outside its training examples. Incidents need enough retained evidence to tell whether the source, retrieval, prompt, tuned behaviour or integration failed.
The NIST Generative AI Profile supports documenting data origins, system changes, evaluation, monitoring and accountable risk responses across the lifecycle. It doesn't endorse one architecture. It does make a useful operator point visible. Architecture approval is also approval of an ongoing evidence and ownership model.
Our default is to start with the least complex pattern that can pass a defined evaluation. That's an Off Piste synthesis of the guidance, not a universal technical rule. Simplicity is valuable because it reduces change paths and makes failures easier to isolate. It stops being valuable when the simpler design cannot meet a material freshness, attribution, permission or behaviour requirement.
Write the pilot brief before choosing the stack
A useful pilot brief makes the decision inspectable before products and implementation detail take over.
- Name one bounded use case, its users and the decision or action the output supports.
- List the approved knowledge sources, their owners, update rhythm and required attribution.
- Define the response or task behaviour that must remain consistent, using examples.
- Record access boundaries, sensitive material, refusal rules and human escalation.
- Build an evaluation set with routine, difficult, outdated, conflicting and out-of-scope cases.
- Set pass criteria and hard failure limits for retrieval, supported answers, behaviour and access.
- Choose the simplest starting pattern that can meet those criteria, then name the job of every component.
- Assign owners for source changes, model changes, incidents and the stop or rollback decision.
Once the pattern is clear, the AI delivery model guide can help decide whether to buy, configure, integrate or custom-build the capability. That's a sourcing choice, separate from deciding how knowledge and behaviour should enter the system.
An AI systems discovery or architecture workshop is useful when the brief exposes unresolved boundaries. Its output should be one bounded use case, knowledge and behaviour requirements, a justified pattern decision, an evidence plan, controls and a pilot brief the business can approve or decline.