MULTI-AGENT EVALUATION · FIELD GUIDE
A collaboration trace answers “what happened”. Acceptance must also answer whether the original goal was met, whether the result entered the real system, and whether failure can be reproduced and repaired.
CORE JUDGMENT
Agent count, message count, tool calls, and runtime can show that a system was busy. None of them alone proves that the work is complete.
01 · WHAT RESEARCH ACTUALLY SUPPORTS
The claims below stay within what the primary papers or formal conference pages support. They are not capability claims about Flowness.
AgentBench evaluates reasoning and decision making across eight interactive environments, treating long-horizon reasoning, decision making, and instruction following as observable failure surfaces. The object of evaluation is behavior in an environment, not a model report.
SWE-bench combines real GitHub issues, repositories, and test environments into tasks. A candidate must change observable test state in an execution environment, showing why generating a patch and solving the problem are separate judgments.
SWE-bench Verified improves evaluation reliability through human checks of task statements, test correctness, and solvability. Automated tests matter, while whether a test actually corresponds to the target remains an independent question.
Research finds that strong model judges can approximate human preferences while also showing position, verbosity, self-preference, and limited-reasoning biases. A single judge pass therefore needs reproducible evidence and, where needed, human judgment.
One study observes gains from debate on selected reasoning and factuality tasks. Another systematic study finds that existing methods do not reliably beat self-consistency or ensembles and can be sensitive to tuning. More agents cannot replace an acceptance design.
02 · THE VERIFICATION LADDER
The higher the risk and the less reversible the change, the more layers are needed. Small, reversible tasks do not need the full ceremony.
Who claimed what, when, and against which version? This is a signal awaiting verification. It shows only that an execution occurred.
Does the candidate exist and bind to its goal, source, version, commit, or hash? It must not be silently replaced during review.
Do structure checks, tests, probes, queries, or policy gates show the target state changed? Does the verifier cover the real requirement?
Reviewers separated from the producer actively look for counterexamples. After a Finding is repaired, evidence is collected again on the successor candidate.
Did the artifact enter the repository, service, database, documentation system, or decision flow that actually consumes it? Readback evidence must come from that consumer.
Did a real or representative task use the new path? Could an old entry point, cache, or silent bypass make the feature look active?
Does target-native readback show the authoritative postcondition, and has the responsible party made the scoped acceptance decision? Are remaining risks and uncovered scope explicit?
03 · FALSE SIGNALS
That is an executor report, not independent evidence.
It shows protocol success, not that the business postcondition holds.
It covers the checks that ran in the local environment, not integration, deployment, or real use.
An LLM judge can scale evaluation, while bias and task fit still need control.
Built, integrated, activated, and accepted are different states.
The check may have been too narrow, the evidence missing, or the reviewers may share one blind spot.
04 · FAILURE ATLAS
Flowness public Failure Atlas organizes structural failure around seven questions. It is a diagnostic lens for replay and counterexample search, not statistical proof.
Why did an event fail to form executable work?
Why did work quietly stop without a conclusion?
Did identity, version, evidence, or interpretation drift?
After facts changed, was stale context or planning reused?
Did the artifact enter the system that actually consumes it?
Did “done” correspond to a real effect and acceptance?
Will the same structural failure be harder to repeat next time?
05 · MINIMUM ACCEPTANCE PACKET
What counts as success, and which outcomes remain unacceptable even when they look useful.
Version, commit, hash, environment, or another identity that cannot be confused with a neighbor.
An observation from the target system, database, test environment, or real consumer.
Who executed, who reviewed, which context was shared, and which common blind spots may remain.
Which candidate held the issue, which successor carried the repair, and why the evidence was not silently dropped.
Recheck the repaired successor candidate, then state that its verdict is not target-domain Effect or responsible-party acceptance, along with the environments, users, or conditions that remain uncovered.
EVIDENCE BOUNDARY
AgentBench, SWE-bench, LLM-as-a-Judge, and multi-agent debate research support evaluation in task environments, observable results, useful but bounded judges, and multi-perspective evaluation worth studying.
Those papers do not validate Flowness and do not prove that any harness automatically guarantees correctness. Flowness public Alpha evidence currently supports a narrow path: execution, independent review, retained Finding, targeted rework, and a fresh successor-candidate verdict. It does not establish target-domain Effect or responsible-party acceptance.
Full Flow runtime coverage, cross-domain effects, and long-term reliability remain claims that need further verification.
06 · QUESTIONS AT THE BOUNDARY
The answers below are deliberately narrower than a product promise.
No. A trace shows that an execution occurred. A fresh successor-candidate verdict still needs precise artifact identity, checks tied to the target, independent review, commitment, activation, and target-native readback. It is not target-domain Effect or responsible-party acceptance by itself.
No. AgentBench, SWE-bench, and LLM judge research provide bounded evaluation methods. A benchmark score, judge score, debate result, CI result, or same-session review does not by itself prove a real-world acceptance condition or a fresh successor-candidate verdict.
The public Alpha evidence supports a narrow assurance-kernel path with execution, independent review, retained Findings, targeted rework, and a fresh successor-candidate verdict. That verdict is not target-domain Effect or responsible-party acceptance. Full Flow runtime coverage, cross-domain effects, and long-term reliability remain unverified.
07 · PRIMARY SOURCES
Evidence checked: 2026-08-18. Research scope and Flowness implementation evidence remain separate on this page.
CONTINUE