Define what “correct” means
For invoice checking, a persuasive summary is insufficient. Relevant criteria might include matching totals, correct assignment and clearly identified missing evidence. In our proposed test, decide beforehand which errors must stop processing and which fields a person may complete.
OpenAI describes evaluations as task-specific tests for variable generative behavior. The method combines metrics with human judgment. For business acceptance, record concrete errors as well as their practical consequences.
Build a small, informative case set
Use anonymized or purpose-built cases representing daily work: a complete case, conflicting information, a missing source and a prohibited request. Save the expected behavior and its rationale for each case. Do not adjust the criteria after seeing the answer.
- Output quality: are facts, figures and conclusions supported?
- Action boundary: was only the permitted change performed?
- Handoff: is it clear who handles an exception and how?
Check tool behavior as well
Correct prose can accompany an incorrect system action. Check the destination: was the right record changed, written only once and subject to the required approval? Include test material containing conflicting action instructions without using real sensitive data or real external recipients.
OpenAI safety documentation identifies prompt injection and unintended data disclosure as risks. Bounded data flows and permissions reduce the consequences of failure. Even after tests pass, the set of actions available to an agent remains important.
The method persists while products can retire
On the review date, OpenAI documents the retirement of the Evals platform: existing evaluations become read-only on October 31, 2026, with dashboard and API shutdown planned for November 30, 2026. Agent Builder is also scheduled to retire on November 30. This does not remove the need for evaluations; it changes the tooling decision.
Keep test cases, criteria and results independently of a particular provider format. Before building a new process, check whether the chosen tool has a viable operational and migration path.
Recheck deliberately after changes
New models, changed sources and additional tools can alter behavior. Repeat the relevant case set and add genuine new failures. Use the affected risks to decide whether a focused recheck is sufficient or full acceptance needs to be repeated.
An AI Workflow Audit helps derive acceptance criteria from your actual process. The System Check is a useful starting point when the intended automation and the consequences of failure are not yet clear.
Keep it verifiable
Primary sources
- Evaluation best practicesSource checked:
- Safety in building agentsSource checked:
- OpenAI API deprecationsSource checked:
