Most RAG systems fail quietly: answers sound confident while retrieval returns the wrong paragraphs. Before we tune prompts on any engagement, we build a golden question set with clients — 50 to 150 real questions with reference answers and the documents that should ground them.
From there, evaluation runs in three layers. Retrieval metrics (recall@k against tagged source passages) tell us whether the right context is even reaching the model. Answer metrics — graded by a judge model against the references — tell us whether responses are faithful and complete. Finally, deflection and thumbs-up rates in production tell us whether users actually trust it.
The discipline pays off when stakeholders ask "is it getting better?" — you answer with a chart, not an anecdote. Every index change, chunking experiment or model upgrade runs the same suite before it ships.