Commit success after checking identity

An item returned by a batch review changed its case_id from gold-pair-078 to gold-pair-079. Its task identifier still pointed to the original case. Reconstructed from the failure record, the identity fields looked like this:

task_id:      gold-pair-078::gold-pair-078-candidate-127
candidate_id: gold-pair-078-candidate-127
case_id:      gold-pair-079

The return belonged to a batch of 20 candidates. Its JSON parsed and its structural checks passed. Per-item validation subsequently found the error, but the batch receipt already said status: ok. Original failure record and code locations

The result did not become a complete formal review. It did leave a recovery hazard: “success” in an old file contradicted the later failure record. A recovery process importing only successful receipts would accept the batch again.

The repair was specific: success must be recorded after every item’s identity has been checked. The caller already knows which case it requested. Software should check that fact against the request instead of asking the model to guarantee it.

A schema that defines only types and structure can admit two different strings equally well. A dynamic schema can further constrain identity values. This system also checks the actual requests after receiving the return, completing batch acceptance.

The implementation checks the top-level case_id, task coverage, item structure, and identity in order. Coverage compares both count and set to detect missing, duplicated, or additional tasks. Identity checks then compare task_id, case_id, candidate_id, and candidate type.

Here is a teaching version of the essential order. The production implementation also has phase-specific schemas, receipts, and retry handling.

expected = {task["task_id"]: task for task in requests}
if len(expected) != len(requests):
    raise InvalidBatch("The request itself contains duplicate tasks")

ids = [item["task_id"] for item in response["items"]]
if len(ids) != len(expected) or set(ids) != set(expected):
    raise InvalidBatch("Missing, duplicated, or additional tasks")

for item in response["items"]:
    task = expected[item["task_id"]]
    for key in ("task_id", "case_id", "candidate_id"):
        if item[key] != task[key]:
            raise InvalidBatch(f"Identity field mismatch: {key}")

commit_success(response)

Joining by identifier allows items to arrive in a different order. Comparing redundant fields prevents the same object from belonging to different tasks in different downstream modules: one module finding 078 through task_id, another grouping it under 079 through case_id.

The program does not silently change 079 back to 078. The mismatch alone cannot establish whether the model copied an identifier incorrectly or answered another question. Failing the whole batch and retaining the raw return preserves the opportunity to investigate both explanations.

Whole-batch acceptance has a cost: the other 19 apparently normal items cannot be reused either. That buys a simple commit boundary. Partial acceptance would require independent item validation, missing-item completion, and duplicate-commit handling—a different recovery design.

Repairing new calls does not remove historical receipts. Import code therefore rechecks each item too. If it finds the identity error associated with the old failure log, it keeps the batch as diagnostic material and admits none of its items into a reusable checkpoint.

The cost of calls already made remains in the account. A restart may use a different run directory; it cannot erase earlier attempts. Otherwise, the reported “cost per usable result” includes only the final successful stretch.

Batch size should be chosen against this total cost. Large batches amortize shared input, but whole-batch retries amplify the effect of an individual error. The project later allowed deterministic splitting only for designated batches that had exhausted retries, while retaining the ability for an individual failure to end the run. Splitting is a bounded recovery measure, not a reason to retry indefinitely.

The regression test deliberately corrupts an item’s identity in the first return and checks three things: that attempt writes a failure receipt, the successful-checkpoint path stays empty, and only the subsequent correct return writes success. It tests the precise gap through which this failure passed.

A batch’s completion count should increase here: the return matches the requests, the whole batch has passed acceptance, and success is then committed.