Agentic RAG

A fixed retrieval pipeline runs the same lookup no matter what the query looks like; agentic RAG is the layer where that decision — retrieve or not, from which index, and whether the result was good enough — moves from the pipeline author to the model itself. That handoff only pays off once the stages underneath it already work, because an agent that decides badly compounds every weakness in the retrieval it inherits rather than fixing one. It sits at the frontier tier of the retrieval-augmented generation stack, the point where teams stop tuning a fixed chain and start letting the model choose.

How to read agentic RAG: decision loop first, market last

Start with how LLM agents decide what to retrieve — it lays out the core shift: an agent that chooses whether to fetch documents, where to look, how to phrase the query, and whether the answer is good enough before it stops looping. Read the prerequisites and hard limits next, in the same sitting: it names what has to already work in your retrieval stack before an agent loop is worth adding, and where the reasoning loop hits its own physics.

When you are ready to build, the LangGraph, LlamaIndex, and Haystack guide decomposes the system into bounded layers — hybrid retrieval, rerank, and a loop with an explicit iteration budget — instead of one framework call. For the framework landscape behind that build, the agentic RAG framework race tracks how orchestration, retrieval, and managed platforms split into three separate bets. Close with the accountability question — once an agent starts choosing sources on its own, that choice needs an owner before it reaches real users.

Where agentic RAG stops being RAG with extra steps

Three confusions send teams debugging the wrong layer.

• Agentic RAG is not query transformation with a nicer name. Query transformation is one technique a pipeline runs once per query. Agentic RAG is the layer that decides whether to invoke a rewrite at all, and re-decides mid-loop if the result still looks weak — the technique is a tool the agent can call, not the thing that makes a system agentic.
• The confidence gate is not the evaluation harness. An agent’s stopping decision — captured as a structured confidence score plus a done flag — is a runtime judgment made on a single query. RAG evaluation frameworks score a pipeline offline, across many queries, against a metric contract. Treating the agent’s self-reported confidence as your quality measurement means nobody ever checks whether its stopping decisions were actually correct.
• The framework layers are not interchangeable. LangGraph orchestrates the loop, LlamaIndex Workflows and Haystack compose the retrieval pipeline underneath it, and managed platforms like Vectara bundle retrieval and governance together. Picking “a framework” without naming which layer you are buying is how teams end up with two orchestrators fighting each other.

Common questions about agentic RAG

Q: How do I stop an agentic RAG loop from calling the retriever forever? A: Set a hard iteration cap and a structured confidence gate — a JSON contract with fields like confidence, missing_info, next_action, and done — so the loop can only continue when it explicitly flags what is missing. The build guide walks the exact contract.

Q: Should I build on LangGraph, LlamaIndex, or a managed platform like Vectara? A: They are not competing for the same job — orchestration, retrieval-pipeline composition, and managed governance are three different layers. Picking one is an architectural commitment, not a library swap; the framework race maps who owns which layer.

Q: Who is accountable when an agentic RAG system picks a biased source? A: The agent decides which index to query, how to rephrase the question, and when the evidence counts as sufficient — decisions a fixed pipeline never made silently. The accountability question traces where that shifts responsibility away from a reviewable step.

Q: Is agentic RAG just a retry loop bolted onto a RAG pipeline? A: No — a retry loop reruns the same fixed step; an agent evaluates what came back and decides whether to change the query, the index, or stop entirely. That judgment call is why it needs its own prerequisites, not a while-loop around retrieval. The prerequisites and hard limits names what has to work first.

Part of the retrieval-augmented generation theme · closest neighbour: RAG evaluation — the discipline that checks whether the agent’s own stopping decisions hold up. Coming to this from a software background? Start with the story: Debugging RAG Failures: Why Developers Need a New Diagnostic Model.