RAG Architecture for Enterprise: What Works and What Does Not

RAG is now the standard approach when a company wants to bring its own content into a language model. The architecture is well understood, the tools are available, and first prototypes usually run within weeks.

Yet many of these projects stall after the prototype. Not on the technology.

What RAG delivers

Retrieval Augmented Generation means, simplified: before the model answers, matching document sections are retrieved from your sources and placed into the prompt. The model then answers on the basis of those sections rather than from training knowledge.

The effect is documented. A 2024 meta-analysis cites a reduction in hallucinations of 40 to 71 percent through RAG with validated context. That qualifier matters: it applies to RAG with validated context, not to RAG alone.

Inside that qualifier sits the reason projects get stuck.

Why the prototype works and the rollout does not

In the prototype somebody picks 20 good documents. The answers are convincing, the demo runs, the budget arrives.

In the rollout you point the system at the real estate: SharePoint, file shares, Confluence, a few ticket systems. And three things sit there at once that were not in the test set.

1 Stale content

The work instruction from 2021 sits next to the one from 2025. Both are formally valid documents. Retrieval has no basis for choosing the right one.

2 Contradictions

Two departments described the same process differently. Both versions feed into the answer, and the model formulates a plausible blend from them that applies nowhere. Why a wiki does not correct this state on its own is set out in the piece on the Confluence alternative.

3 Gaps

What is not documented cannot be found. At 42 percent undocumented job-related knowledge (Panopto/McKinsey), the missing part is exactly what critical questions need. The model fills that gap, because that is what it is built for. And then it gets expensive.

The consequences are measured: 95 percent of AI pilots show no measurable ROI (MIT NANDA, 2025). 47 percent of enterprise AI users made at least one important business decision based on hallucinated content. And 4.3 hours per employee per week go on verification, roughly 14,200 dollars a year.

The layer missing from most architecture diagrams

Typical RAG architectures consist of four building blocks: sources, indexing with embeddings, retrieval, generation. A fifth layer is missing in between.

Call it the trust layer. Its job is to turn documents into reliable context before they are indexed. Every knowledge object in this layer needs four properties:

• Owner. A named person accountable for the statement.
• Validity date. When it was reviewed, until when it applies.
• Verification status. Verified, unverified, in conflict.
• Provenance. Where the statement comes from, which interview or document.

With these four attributes retrieval can prioritise rather than guess. And the answer can state its basis, which is the difference between a usable and a risky answer.

How the gap gets filled

The trust layer solves the contradiction and currency problem. It does not solve the gap problem, because a validation layer can only validate what exists.

For the gap a different mechanism is needed: active capture. AI-supported expert interviews get decision logic, exceptions and heuristics out of people's heads and introduce them as structured knowledge objects into the same layer. Consolidation runs along five dimensions: correctness, currency, relevance, freedom from conflict, completeness.

Only then is the precondition met under which the 40 to 71 percent apply: validated context.

Connecting to your existing landscape

Practically relevant: this layer does not replace your front end. It delivers context to what you already have, via RAG API, REST API and MCP server. SharePoint delivery is possible, Microsoft Copilot can be connected as a context consumer, SAP HR or Workday can serve as a trigger source, for example when a departure is pending. When building your own pays off and when it does not is set out in the comparison of Microsoft Copilot against custom enterprise AI.

The MCP server is the piece that counts for agent scenarios. An agent acting on its own needs context with provenance, not textual similarity. Without context, AI scales errors rather than competence.

What European organisations have to settle additionally

Two regulatory points have been in force since 2025 and belong in every architecture decision: EU AI Act Article 4 on AI literacy obligations since February 2025, and NIS-2 since December 2025.

In practice that means requirements on hosting, traceability and governance. Relevant characteristics in the assessment: EU hosting on Azure with data centres in Germany, GDPR compliance, TISAX assessment, no use of customer data for model training, complete and demonstrable deletion, role-based access control with an audit trail.

And a side topic with real risk: over 50 percent of employees use private GenAI for work without oversight (Gartner, February 2026). A working internal route is therefore also a measure against shadow AI.

The sequence that proves itself in practice

Not: pick a model, then look for data. The other way round.

Pick a concrete use case with measurable benefit, for example support answers for one product line or decision support in an operational process. Determine which knowledge that requires. Check how much of it exists documented, current and free of contradiction. Capture the gap actively. Only then build the retrieval.

That sequence is slower to start and faster to result. The reverse produces convincing demos and unusable rollouts.