For now, we are keeping BM25

At the end of our M1 diagnostic for Discovery Front, we did not select the five-way retrieval blend we had defined in advance. We kept BM25, along with several promising individual retrieval paths.

It was not an obvious choice. The blend retrieved more relevant candidates in the first hundred results. On mean recall alone, it beat BM25. But its top ten ranked worse, and the recall improvement was more sensitive to the mix of cases than we were prepared to accept as a reason to switch.

The original idea was to combine complementary strengths: keywords for explicit terms, vectors for different ways of saying the same thing, and cross-model representations for connecting another coordinate system. The results made us reconsider where those paths should converge.

The experiment used requests and candidate profiles assembled from public materials, with completed diagnostic relevance judgments. There were 44 cases: 5 for selection and 39 for evaluation. Two evaluation cases had no labeled positives, so the recall and nDCG means use the remaining 37. The mean number of negatives in the top ten uses all 39. These judgments concern relevance between materials; they do not establish anyone’s willingness to collaborate.

Start with three versions: the BM25 keyword baseline, a fixed blend of five retrieval paths, and the blend with an additional ranking stage.

Metrics are computed per case and then averaged. Recall@100 measures how many relevant candidates enter the first hundred; nDCG@10 measures whether relevant items appear nearer the top of the first ten. Negative counts refer to the labels, not invitations actually sent. Values come from the original M1 evaluation report.

The blend’s recall point estimate was about 3.67 percentage points above BM25. At the same time, top-ten ranking quality fell and the mean negative count rose. The extra ranking stage recovered some quality near the front without overtaking BM25 on these means.

These columns answer different questions. Candidate generation must leave enough opportunities for what follows. Ranking determines where limited attention goes first. Enlarging a set and ordering that set well are distinct problems.

The five paths used keywords, hybrid retrieval, same-model vectors, shared references, and a linear mapping. They were combined using reciprocal rank fusion, or RRF: each candidate accumulates contributions from its rank in each list, rather than from a sum of raw scores measured on incompatible scales. RRF is an established method, with research published in 2009. Original paper

This gives different retrievers a relatively simple meeting point. It does not tell the system which retriever is best suited to the request in front of it.

Consider a teaching example. Keyword retrieval puts a candidate that satisfies an exact constraint in second place. Several other paths put a thematically similar candidate, missing that constraint, in fifth place. Agreement across paths may promote the latter. The fusion rule sees ranks; some input or judgment still has to express the consequential difference between the candidates.

That example illustrates a possible failure. The aggregate report does not establish which case types caused most of the observed decline. What we can directly observe is that adding more paths through a fixed blend did not deliver both better recall and a better top ten. Before adding another path, we should ask which requests benefit from each existing one.

A point estimate describes the average change in this batch. It does not show how sensitive that change is to the composition of the cases.

Using the prespecified method—10,000 paired bootstrap resamples at the case level—the Recall@100 difference between the blend and BM25 was 0.0367, with a 95% interval of −0.0332 to 0.1150. The interval crossed zero, so the diagnostic did not meet the selection requirement that the lower bound of the recall improvement be positive.

Two shortcuts would lose information here. “The mean is higher, so switch” ignores uncertainty. “It missed the selection criterion, so all its components are useless” erases differences between the methods. For example, the standalone same-model B path had a recall point estimate of 0.6066 on these evaluation cases, above the fixed blend. That is worth following up, but seeing the result does not entitle us to declare it the final winner after the fact.

We retained BM25 and the promising candidate paths, and did not select the fixed blend for the larger evaluation. Research stopped on September 1. Another batch of scores had been generated, but it had no complete quality result. This article revisits the existing records; it does not resume those runs.

The second stage combined the fusion score and an interpretable ranking score with fixed weights of 0.7/0.3. Compared with the blend, it reduced the mean number of top-ten negatives by 0.1795; the paired interval was −0.3333 to −0.0256. That improvement passed the criterion used in this diagnostic.

However, the lower bound for recall retention was −0.02325, just below the prespecified −0.02. We still lacked sufficient evidence that recall loss stayed within the permitted range.

The structural reason is straightforward. A second stage that rescores a larger pool and then takes the first hundred can change the membership of the result set. What is called “reranking” is also making a second selection.

That suggests a concrete implementation change: freeze the generator’s first hundred candidates, then reorder only those members. The second stage can affect the top ten without pushing already-retrieved relevant items out of the first hundred. The tradeoff is equally clear: it cannot rescue anything the first stage left outside that set. This structure has entered subsequent design work; its actual ranking benefit still needs a new evaluation.

The experiment separated the broad idea of “combined retrieval” into three questions: which requests need extra retrieval, which new members that retrieval contributes, and how to order the members already selected.

When keyword results are already good enough, an extra path may add computation and competition. When keywords miss an alternative expression, a vector path may supply something essential. Charging every request the same combination cost needs a stronger reason than having more methods available.

Our next question therefore became whether retrieval behavior, observed before looking at labels, can identify requests worth extra computation; whether that computation actually recovers more relevant items; and whether ranking a fixed membership reduces noise near the front. Evaluating those separately tells us where any gain comes from.

There is an interface readers can inspect immediately in their own systems: does the second stage change only order, or membership too? Making that distinction explicit in implementation and measurement often explains a system’s gains and losses better than adding another retriever.

This article uses the M1 evaluation-v3/report.json, decision-v1/M1_DECISION.json, and combination manifest saved on September 1, 2026. The three methods’ macro averages can be recomputed from the compact per-case dataset. Paired intervals come from the historical decision record; see the method guide. This was a diagnostic on public materials. It did not measure real collaboration acceptance or change production ranking.