RAG and Agents

Key concept · Why RAG isn't killed by long context

Even as context windows balloon, RAG stays useful for two reasons. First, data outgrows any window — data accumulates faster than context limits rise. Second, a model that can process long context doesn't use it well: the longer the input, the more likely the model focuses on the wrong part (the "lost in the middle" problem), and every extra token adds cost and latency. RAG lets the model use only the most relevant information, and it's a natural way to scope per-user data.

Key concept · The retriever: indexing and querying

A RAG system's quality lives or dies by its retriever, which has two jobs: indexing (processing data so it can be retrieved quickly later) and querying (sending a query to retrieve relevant data). Documents are split into chunks because retrieving whole documents makes context arbitrarily long. In the original RAG paper the retriever and generator were trained together; today they're usually separate, off-the-shelf components — though finetuning the whole system end-to-end can help. (The book uses "document" to mean a document or a chunk.)

Key concept · Term-based (lexical) retrieval — TF-IDF and BM25

Match documents by keywords. Two signals combine: term frequency (TF) — the more a term appears in a document, the more relevant it is — and inverse document frequency (IDF) — a term appearing in many documents (like "the", "at") is less informative. TF-IDF multiplies them; BM25 refines TF-IDF by normalizing for document length and remains a formidable baseline. Systems like Elasticsearch/BM25 use an inverted index (term → documents containing it) for fast lookup. Term-based retrieval is fast, works out of the box, but computes relevance at the surface level, not the meaning.

Key concept · Embedding-based (semantic) retrieval and vector search

Rank documents by how closely their meaning aligns with the query. Indexing gains an extra step — convert chunks to embeddings stored in a vector database; querying embeds the query and retrieves the k nearest chunk embeddings (by e.g. cosine similarity). The hard part is vector search: exact k-nearest-neighbors (k-NN) is precise but slow, so production uses approximate nearest neighbor (ANN) algorithms that organize vectors into buckets, trees, or graphs — HNSW (navigable graph), IVF (cluster then search), product quantization (compress vectors), LSH (hash similar vectors together), and Annoy (random-split trees) — trading a little accuracy for large speed gains (FAISS, ScaNN implement these).

Key concept · Comparing — and combining — the two

Term-based retrieval is faster and works well immediately, but has few knobs to improve and can't capture meaning. Embedding-based retrieval can be finetuned to keep improving and beat lexical search, but embeddings can obscure exact keywords like error codes (EADDRNOTAVAIL) or product names. The practical answer is a hybrid: combine both (often lexical first to narrow, embeddings to rerank), getting keyword precision and semantic recall together.

Key concept · Chunking, reranking, query rewriting, contextual retrieval

Chunking strategy matters most: chunk by length, sentence, paragraph, or recursively — too small loses context, too large dilutes relevance and wastes the window (overlap helps avoid splitting an idea). Reranking: retrieve a broad candidate set cheaply, then reorder with a stronger (cross-encoder) model, keeping the top few — better precision without embedding everything expensively. Query rewriting (query reformulation): rewrite an ambiguous or context-dependent query ("her" → the named person) before retrieving. Contextual retrieval: augment each chunk with metadata or a short summary of the document it came from, so an isolated chunk is still findable and interpretable. RAG also extends beyond text — multimodal RAG (images) and retrieval over tabular data (text-to-SQL).

Key concept · Evaluating the retriever — context precision and recall

Retrieval quality is judged on what it returns. Context precision (a.k.a. context relevance): of the documents retrieved, what fraction are relevant? Context recall: of all the relevant documents, what fraction did you retrieve? In production, precision is easier — you only compare the retrieved docs to the query (an AI judge can do it) — while recall needs relevance labels for the whole database. If ranking order matters, use NDCG, MAP, or MRR. Evaluate the retriever separately from the generator so you know which half to fix.

Key concept · What tools do — read vs. write actions

Tools fall into rough categories. Knowledge augmentation (read-only): retrievers, web search, SQL queries, APIs — they bring information in without changing state (a RAG retriever is one such tool). Capability extension: a calculator, a code interpreter, a translator — patching the model's weaknesses (models are bad at arithmetic; a calculator fixes it). Write actions: tools that change the world — send an email, execute a transaction, update a database. Write actions unlock automation but raise the stakes enormously, so they need the strongest safeguards (approval, sandboxing, least privilege). A bigger tool inventory adds capability but makes correct tool selection harder.

Key concept · Function calling — the mechanism

Tools are wired to the model through function calling: you declare each tool with a name, description, and parameter schema; the model, when it decides a tool is needed, emits a structured call (which tool, which arguments); your system executes it and returns the result to the model, which continues. This turns the model's free-text output into reliable, machine-executable actions — and is why instruction-following and structured-output quality (Chapter 4) matter so much for agents.

Key concept · Plan, validate, execute — and reflect

Decouple planning from execution: have the model generate a plan (a sequence of steps/tool calls), validate it (is every tool real, are the arguments sane, does it plausibly reach the goal — a human or another model can check) before executing, so a bad plan is caught cheaply. Interleaving reasoning and acting (ReAct: reason → act → observe → repeat) and adding reflection / self-correction (Reflexion: examine failures and retry) makes agents far more robust — the agent notices a tool returned an error or the result doesn't fit the goal, and adjusts. Complex tasks can also be split across multiple specialized agents.

Key concept · Agent failure modes and evaluation

Agents fail in ways single responses don't, and each compounds over steps. Planning failures: calling a nonexistent tool, passing wrong/invalid parameters, or producing a plan that never reaches the goal. Tool failures: a tool returns wrong output or errors, and the agent trusts it. Efficiency failures: it reaches the goal but with too many steps, too much latency, or too much cost. Evaluate agents on all of these — task success and the number of steps/tool calls/cost it took — not just the final answer.

Key concept · Memory — internal, short-term, long-term

Agents need memory to carry information across steps and sessions, in three forms: internal knowledge baked into the model's weights; short-term memory — the current context window, fast but bounded and wiped between sessions; and long-term memory — an external store (often a RAG database) the agent reads from and writes to, giving persistence beyond the window. A memory system lets an agent handle information overflow (offload to long-term, retrieve when needed), remember user preferences across sessions, and stay consistent — closing the loop between this chapter's two patterns, since long-term memory is retrieval.