Stop Evaluating RAG With Vibes

I keep seeing teams ship RAG systems after a demo that answers three handpicked questions. Two months later, production queries miss, users complain, and everyone blames the model. The failure was not the model. It was retrieval, and they never tested it properly.

A demo is not an evaluation. Cherry-picked questions leak the context you already know works. You cannot tell if your retrieval stack generalizes until you throw real user-shaped queries at it, some of which should fail, and measure whether the right chunks came back before the model ever saw them.

If you are shipping a RAG product and you do not have a golden set of test queries with known-good source documents for each one, you are evaluating with vibes. Here's the small harness I use before I trust a retrieval stack.

Why vibes fail

The classic mistake is testing answer quality without isolating retrieval. Someone asks, "What's our return policy?" The model responds with something fluent. The team calls it good. Nobody checked whether the chunks that made it into the prompt were the return policy doc, a customer support FAQ from 2022, or three totally unrelated paragraphs about shipping.

When retrieval is wrong, the model can still generate a confident answer. That answer might sound fine if you do not know the domain. It might even be mostly right if the model memorized general knowledge during training. But if the retrieval step missed the source doc you actually wanted, you are not doing RAG. You are doing conditional hallucination with extra steps.

The other failure mode is testing on questions you wrote while building the system. Of course they work. You tuned the chunking, the embeddings, and the query rewrite on those exact examples. The real test is: can it handle the questions your users will actually ask, phrased the way they will phrase them, including the weird edge cases, the acronyms, the typos, and the requests that should return nothing because the answer is not in your corpus?

What a small golden set looks like

I start with 20 to 50 real user queries. Not synthetic. Not the ones I invented while prototyping. Actual questions from support tickets, Slack messages, or recorded demos where someone asked something and I knew what document should have been retrieved.

For each query, I record:

• The query text exactly as the user would phrase it.
• The source document IDs or chunk IDs that must appear in the top-k results. These are the gold chunks.
• Optional: distractor document IDs that should not appear. These are the hard negatives, the docs that look relevant but are not.
• Optional: a note on what an acceptable answer should cover, for the answer-quality pass later.

The golden set is not about coverage. It's about failure detection. If retrieval misses the gold chunk on a query I know should work, the system is not ready.

I judge retrieval first. Did the gold chunk appear in the top-5 results? Top-10? If not, the retrieval step failed, and there's no point scoring the generated answer. Fix retrieval before you tune the prompt.

Once retrieval passes, I score answer faithfulness. Did the model use the chunks it was given? Did it invent facts? Did it cite the wrong source? That's a separate concern. You cannot fix answer problems by tweaking embeddings.

Failure modes I keep seeing

I have debugged enough RAG systems to recognize the patterns. These are the ones that show up repeatedly when someone skips the eval harness and goes straight to production.

Wrong chunk boundaries. The document gets split mid-sentence or mid-paragraph, and the chunk that lands in the context window is missing the key claim. The model has half the answer and guesses the rest. Retrieval technically worked—it found the right doc—but the chunking ruined it.

Stale index. The corpus was updated, the embeddings were not rebuilt, and now the retrieval step is searching a snapshot from three weeks ago. Production users are asking about the new product docs. The system does not know they exist.

Embedding mismatch. The user asks a question in plain English. The corpus is full of jargon, abbreviations, or structured data. The embedding model was trained on Wikipedia and Stack Overflow. It does not know that "MTD rev" and "month-to-date revenue" are the same thing. Semantic search fails because the vector space does not align with how your users actually talk.

Wrong tool for the job. Not everything belongs in a vector database. If the user is asking for a specific customer's order status, you do not need embeddings. You need a SQL query with the customer ID. I see teams dump structured data into a RAG pipeline because it feels like the modern thing to do, and then they are surprised when the retrieval is slower and less accurate than a JOIN.

The minimum harness

You do not need a research-grade eval framework. You need a script that runs the golden set, scores retrieval, and prints the failures.

Mine is a JSON file with the test cases, a Python script that calls the retrieval API for each query, and a simple scoring function: did the gold chunk ID appear in the top-k results? If yes, mark it pass. If no, print the query, the gold chunk, and the chunks that were returned instead.

I run it in CI every time the chunking logic changes, the embedding model changes, or the index gets rebuilt. If the pass rate drops, something regressed. I do not deploy until I know why.

You can wrap this in a FastAPI service if you want a web UI for the team to review failures. You can add LLM-as-judge scoring if you want to automate the answer-quality pass. But start simple. The goal is to catch retrieval failures before they reach production, not to build an MLOps platform.

If you cannot name 20 questions that should fail when retrieval is wrong, you are not ready to demo

The eval harness forces you to be specific about what your system should do. It forces you to test the queries users will actually ask, not the ones you wish they would ask. It forces you to separate retrieval failures from model failures, so you are not tuning the wrong thing.

A three-question demo that works on handpicked examples is not evidence your RAG system works. It's evidence you found three examples that work. The production traffic will find the ones that don't.

Build the harness. Run it before every deploy. Ship when it passes, not when the demo looks good.

If you are working on RAG systems, retrieval pipelines, or anything in this space and you want to talk through your eval strategy, reach out: bilaltehseen@gmail.com.