Cohere's RCP-nDCG says benchmarks mislabel relevance

Cohere introduced RCP-nDCG@10, a retrieval metric that grades every retrieved document against the same explicit relevance rubric using a calibrated AI judge, instead of scoring only against pre-existing pooled relevance labels. It is published as an arXiv paper with code and data on GitHub, and the same methodology was used to evaluate Cohere’s Embed 5.

The empirical argument is that existing labels are wrong in a specific, measurable direction. Human validation used 46 contracted annotators with degrees in the relevant field, working blind over NanoBEIR, BRIGHT and ViDoRe v3, across 289 contests on 273 queries with three annotators each.

• Conventional labels predict human relevance with an AUC of 0.65. The calibrated judge reaches 0.91.
• 28% of documents the benchmark called irrelevant were judged useful by humans. Only 15% of documents labelled relevant failed basic topicality.
• Benchmarks label 29% of documents relevant where humans suggest 60%.
• Inter-annotator exact agreement on a 0 to 4 scale is just 42%, and the post cites re-annotation studies as low as 17%.

Where the two metrics disagree, reviewers sided with RCP-nDCG 70% of the time. Above roughly a 0.02 margin that rises to 82%, and at the widest margins 97%. Conventional nDCG at its widest margins manages only 53%. Overall RCP-nDCG picked the reviewers’ preferred system 77% of the time against 52% for conventional nDCG.

The attribution matters for anyone tuning against these numbers: crediting relevant documents the answer key missed added 19 percentage points, and grading how relevant each document is added a further 6 points. Grading relevance alone, without the credit for missing documents, contributed only 3.3 points. Most of the gain is about fixing incomplete labels, not about a better rubric.

The method is five yes-or-no questions per document, identical for every query, combined with groupwise pairwise comparison. Calibration uses both, and Cohere’s description is that the comparisons set the order while the rubric places it on a scale shared by every query. Kenneth Enevoldsen, the primary maintainer of MTEB, is quoted as intending to bring RCP-nDCG into MTEB.

The honest limits: the headline 77% against 52% comes from a pool that deliberately oversamples disagreements, where the two agree both match reviewers 87% of the time. Sample sizes are modest at 46 annotators and 289 contests. And Cohere warns that its own fifth generation Embed and Rerank models may not always look strongest judged only by legacy nDCG, while conceding the objective is not to optimise for any given metric for its own sake.

(Original source: Cohere)