Bridging Evaluation Science and Deployment Practice
AAAI-27 Bridge Program · February 16–17, 2027 · Palais des Congrès, Montréal, Canada
Organizations increasingly decide whether to deploy agents on the strength of benchmark scores, yet there is no shared practice for connecting a benchmark result to the decision it is meant to inform. Task success, the headline metric of most agent benchmarks, records whether an objective was met. It does not record how it was met, whether the task matches the setting the agent will work in, or whether the agent leaked private data, violated policy, took unacceptable actions, or needed so much correction that it saved the operator no time.
The knowledge needed to close this gap sits in two communities. The AI evaluation community (benchmark builders, agent developers, and measurement researchers) knows how scores are produced and what they can mean. The deployment practice community (workflow owners, domain experts, applied AI engineers, and reliability, safety, and human factors specialists) knows which failures are unacceptable, how people check and correct an agent's work, and what must hold before use is authorized. This Bridge brings the two together around one question:
Under what conditions do agent-benchmark results justify claims about reliability, cost, and usefulness in a specified operational setting?
Participants start from an operational setting and derive its deployment criteria before seeing any benchmark. They then examine which criteria the benchmark supports, which consequential failures it misses, and what a supervised pilot must still establish. This edition covers two application domains: industrial asset operations and maintenance, and software engineering.
The Bridge will produce two worked case audits and a versioned protocol draft that builds on PREP-Eval and maps to the Agentic Benchmark Checklist, BetterBench, and Evaluation Cards.
We invite participants from both communities, including:
To take part, submit a short case statement or interest statement by December 1, 2026. Reports and posters will be presented in a lightning session on Day 2. Practitioners who cannot travel can contribute remotely by completing the audit template and reviewing draft audits.
Submission instructions will be posted here. Attendance requires AAAI-27 registration (Bridge add-on or Bridge-only).
Day 1 is focused on education and Day 2 on collaboration. Participants work in mixed teams of six to eight.
| Session | Activity |
|---|---|
| Day 1, morning | Deriving the criteria. Reciprocal tutorials by practitioners, measurement researchers, and agent developers. Teams receive the operational setting of the maintenance case, derive deployment criteria, and rank failures by consequence before the benchmark is revealed. |
| Day 1, afternoon | Inspecting the evidence. A trace subgroup scores saved traces against the automatic judge and separates scorer defects from agent failures. A setting subgroup checks the criteria against a practitioner-completed fidelity worksheet. |
| Day 2, morning | Challenging the interpretation. Lightning talks and posters, each with a methodological and a practitioner response. Teams then work on the developer-productivity case and design one additional evaluation. |
| Day 2, afternoon | Cross-review and synthesis. Teams review each other's audits. A joint session merges them into the two worked audits and the protocol draft. |
A detailed schedule with times will be posted closer to the event. No coding is needed for the setting subgroup; starter notebooks are provided for the trace subgroup.
A configured agent system in the simulated maintenance environment of AssetOpsBench. Teams ask whether a successful score supports the proposed operational claim and where an operator would need to check, correct, or stop the system. Maintenance practitioners review the workflow description and complete a fidelity worksheet for the simulated environment.
A human–agent workflow, using METR's developer-productivity study and its released task-level data, paired with a configured agent on SWE-bench Verified. Teams examine what self-reported completion time captures, how a small number of developers affects uncertainty, and what evidence would support extending the findings.
Neither case requires production access or private traces. Part A of each case packet (the operational setting and audit template), along with a worked example on τ²-bench, will be published before the event.
| December 1, 2026 | Case or interest statements due |
| December 13, 2026 | Participation decisions |
| January 6, 2027 | Case packets (Part A) and worked example released |
| February 16–17, 2027 | Bridge at AAAI-27, Montréal |
| March 19, 2027 | Reviewed audits and protocol v0.1 released |
| May 2027 | Follow-up meeting |
Questions can be sent to Vedant Khandelwal at vedantk@ibm.com.