AI
RAG implementation: why naive RAG falls short
Embeddings plus a vector database is an afternoon of work. A RAG pipeline that reliably finds the right documents is engineering. The difference lives mostly in what happens after the search.
Where naive RAG runs aground
The standard recipe, chunk documents, embed them, fetch the nearest chunks for a question and feed them to an LLM, works surprisingly well in a demo and disappointingly often almost-but-not-quite in production. Semantic similarity is not the same thing as relevance. The question "what does a permit cost in Amsterdam" retrieves chunks about costs, chunks about permits, and chunks about Amsterdam, and the chunk combining all three is by no means always on top.
In a demo nobody notices, because the builder asks the questions. In production, users ask questions the pipeline was never tuned on, and every miss erodes trust in every answer after it.
What a reranker adds
A reranker is a second judgement. The first retrieval pass can be broad and fast and fetch fifty candidates; the reranker then puts each candidate next to the question and scores how well they really fit. That is compute you do not want to spend on your whole corpus, but across fifty candidates it is cheap. The resulting order is demonstrably better than what vector distance alone produces.
In our own product TenderScan.nl, a search engine and index for public tenders across Europe, exactly this is the difference between "looks similar" and "is relevant". Users save searches and receive newly found tenders weekly; RAG with reranking is why that list is more relevant than standard search would make it. It runs in production, not in a notebook.
The trade-offs that actually matter
Quality versus cost. Every stage of the pipeline, chunking, embeddings, retrieval, reranking, generation, has a price per question. The craft is measuring where the quality comes from and paying where it pays. Better reranking often turns out to be worth more than a larger generation model.
Operations. A RAG pipeline is a living system. Documents change, the index must follow, and quality has to be watched with a fixed evaluation set rather than gut feeling. Skip that, and you learn about quality decay from complaining users.
Model independence. We set pipelines up model-independent: switching embedding model, reranker or LLM is a configuration choice, not a rebuild. Models get better and cheaper every quarter; a pipeline welded to one provider cannot take that gain. Vendor lock-in on your LLM provider is a choice, and in our view the wrong one.
When RAG is the wrong answer
Honesty obliges: sometimes RAG is overkill. If the knowledge fits in one prompt's context, a good prompt is cheaper and better. If the corpus is small and stable, classic search with some metadata goes a long way. RAG pays when the corpus is large, changing or too diverse to fit a prompt, like thousands of new tenders a week.
Read on, or talk
How we keep systems like this reliable is on agentic workflows; what an LLM integration involves is on integrating an LLM into your application. The production cases are on what we build.
Considering RAG for your own data? We will gladly show you what it is worth in your case, on your own documents.