Agentic RAG adds a reasoning loop to standard RAG. The model retrieves chunks, decides if they're sufficient, and retrieves again if not. The loop continues until it has enough information or hits a configured hop limit.

When is agentic RAG worth the cost?

Only when your questions require chaining multiple retrievals across documents, or when the answer requires checking live data not in your index. For everything else, one-shot RAG is more cost-effective.

How much more expensive is agentic RAG?

A 3-hop loop costs roughly 3x the inference of one-shot RAG, with latency stacking sequentially. A two-second response becomes five or six seconds, which is a product problem in customer-facing systems.

Does agentic RAG fix all RAG problems?

No. If your failures are poor retrieval ranking or coverage gaps, agentic RAG just iterates on those problems more expensively. Diagnose the failure mode first before committing to the architecture.

What should I build instead of agentic RAG?

Query expansion, HyDE (retrieving on hypothetical answers), or cross-encoder re-ranking all improve one-shot RAG quality without the latency and cost penalty of an agentic loop.

Agentic RAG: When the Retrieve-Reason Loop Pays

Agentic RAG is one-shot RAG with a planning loop added: the model retrieves, reasons about what came back, decides if it is enough, and retrieves again if not. It earns that cost in two specific situations. Everywhere else it is over-engineering with a higher invoice.

That framing is missing from every explainer currently ranking for this term. They describe the loop. None of them tell you what it costs or when to skip it.

What the Retrieve-Reason Loop Actually Does

Standard RAG is a single pass. Embed the query, pull top-k chunks, inject them into context, generate. The model gets one shot at retrieval and works with whatever came back.

Agentic RAG makes retrieval a conditional branch inside the generation process. The model sees the retrieved chunks, reasons about whether they answer the question, and if not, reformulates the query and tries again. This continues until the model decides it has enough, or a configured hop limit stops it.

The most widely cited implementation pattern is ReAct (Yao et al., 2022): the model alternates between reasoning traces and grounded actions, with "search" as the primary action type. FLARE (Jiang et al., 2023) is a tighter variant where the model generates forward-looking text and triggers retrieval only when it lands on a low-confidence span. Self-RAG (Asai et al., 2023) adds a critic mechanism that scores each generated segment for retrieval necessity and factual groundedness. Different architectures, same core mechanic: retrieval becomes a decision point, not a fixed pipeline step.

LangGraph and LlamaIndex both ship agentic RAG primitives. The wiring is not the hard part. Knowing when to wire it up is.

What One-Shot RAG Consistently Fails At

The case for agentic RAG rests on two real failure modes, not theoretical ones.

Multi-hop questions. A question like "What compliance framework does the vendor our payments team chose last quarter follow?" requires chaining at least two retrievals: first to identify which vendor was selected, then to retrieve that vendor's compliance posture. A single retrieval pass returns one or the other, rarely both in the right relationship. The model fills the gap with a hallucination or a hedge. Multi-hop benchmarks like HotpotQA and MuSiQue exist precisely because this failure pattern is consistent and measurable across model families and retrieval setups.

Live-data environments with no staleness signal. If your index is two weeks behind and someone asks about current inventory, a price, or a status that changed last Tuesday, one-shot RAG returns a confident wrong answer. There is no "I don't know" signal. The chunk is present, it looks valid, the model uses it. Agentic RAG does not fix stale data, but a retrieval loop that can call a live API as one of its action types can get a correct answer where a static index cannot. That is only possible when retrieval is a runtime decision, not a pipeline step.

Everything else, question answering over a well-scoped document set, summarization, entity extraction, single-fact lookups, is one-shot RAG territory. Not because the loop would fail at those tasks, but because it would solve them at 2-4x the cost for no measurable improvement in output quality.

What the Extra Round-Trips Actually Cost

Each hop in an agentic RAG loop adds, at minimum, two extra LLM calls: one to reason about the prior retrieval and reformulate the next query, one to process the new chunks. Plus the embedding and retrieval call itself. A 3-hop loop runs at roughly 3x the inference cost of one-shot RAG, not counting the planning overhead on top.

Latency stacks sequentially. Round-trips cannot run in parallel because each hop depends on the result of the last. A one-shot response that completes in two seconds can easily take five or six over three hops at the same model size and chunk count. For anything customer-facing, that is a product problem before it is a cost problem.

Token cost is also non-linear. Each hop accumulates context from prior retrievals, so the context window grows across hops. If hop 1 pulls 1,500 tokens of chunks and hop 2 pulls another 1,500, the second LLM call is processing 3,000 tokens of retrieval context plus the conversation history plus the planning prompt. The marginal cost per hop is not flat; it climbs.

I have seen teams reach for agentic RAG because it felt like the more capable choice, then discover mid-scale that retrieval costs were eroding margins they had not modeled. Running the math before committing to the architecture is not optional.

When Agentic RAG Actually Earns Its Bill

Two scenarios. Both specific.

Scenario 1: Multi-hop questions are the dominant query type. Enterprise knowledge bases where answers require bridging facts across documents. Research assistants. Internal compliance tools that cross-reference policy documents and vendor records. Due diligence workflows. If your real eval set is dominated by questions that require chaining more than one retrieval to answer correctly, build the loop. Run evals on your own data first (shaped like HotpotQA) to confirm the failure mode is actually there before committing to the architecture.

Scenario 2: The answer requires a live action, not just a chunk. Customer support agents that need to check a live order status before responding. Inventory queries. Booking and scheduling systems where availability changes mid-session. Here the agentic loop is not a nice-to-have; it is the only architecture that gets a correct answer, because the correct answer does not exist in any static index.

These are the cases. If your use case does not map to one of them, you are paying for retrieval complexity that will not move your eval metrics.

What to Build Instead When It Doesn't Apply

The right answer is usually a better one-shot pipeline, not a lighter version of agentic RAG.

Query expansion before retrieval (generating multiple reformulations of the user query and fusing the ranked results) captures most of the coverage benefit of multi-hop retrieval without the latency penalty. Hypothetical Document Embeddings (HyDE) generates a hypothetical answer and retrieves on that instead of the raw query, which improves recall for questions phrased differently from the indexed content. Re-ranking with a cross-encoder model after retrieval improves precision without additional LLM calls in the generation path.

The [Context Engineering: What Comes After RAG] piece covers the precision side of this in more depth. The [RAG vs Fine-Tuning] piece covers when retrieval is the wrong tool entirely. Neither piece makes the case for agentic RAG as a default architecture, and neither should.

Diagnose the Failure Mode Before You Choose the Architecture

Write your evals first. Take 50 real queries from your actual use case, run one-shot RAG, and look at where it fails. If the failures cluster around multi-hop bridging gaps or live-data staleness, build the loop. If they cluster around precision (wrong chunks ranked too high) or coverage (chunks exist but never surface), fix the retrieval layer before adding a control loop on top of it. An agentic retrieval loop on a broken index just iterates on bad results more expensively.

Agentic RAG is a legitimate architectural pattern solving a real class of problems. It is not a default upgrade from standard RAG. The teams getting the most out of it built it for a measured failure mode, not because the retrieve-reason loop sounded like the right level of sophistication.