How to build a failure map for an AI agent before you pilot it

A full-workday healthcare task is a better pilot gate than a vendor demo. A DAYJOB: Healthcare assignment has an average estimated human completion time of 13.6 hours. Before you pilot a long-horizon agent, put it on a released full-workday task, grade every rubric criterion, and record each miss by fact, file, decision, or deadline. The map, not the headline score, tells you whether the agent can survive your workflow.

Demos are good at making a narrow task look finished. Real professional work is harder: it has source files, ambiguous instructions, deadlines, and consequences. A pre-pilot evaluation needs a workload that resembles that work, not a clean prompt.

Use a full-workday benchmark as the pilot gate

Start with tasks that require professional judgment over a long stretch of materials. DAYJOB is a released benchmark set for professional work, with two benchmarks: healthcare and finance, and 130 expert assignments in total. The healthcare subset contains 50 expert-designed assignments. The scale is useful before a pilot because it forces the agent to sit with materials long enough to make the mistakes you would otherwise find in production.

Finance tasks are a different stress test. DAYJOB: Finance prompts average 81 words, while GDPval prompts average 337 words. Finance tasks also carry 25.7 input files on average, compared with 1.2 input files on average in GDPval. The average estimated human time for a finance task is 21.6 hours, while the full GDPval set has a 4-hour median estimated human time. If your workflow involves many documents, financial tasks are the better pressure test.

Do not run one task and call it a pilot. A single pass can be luck. A single failure can be noise. You need a set of tasks that exposes recurring weaknesses. A benchmark set also gives you a shared vocabulary for misses, which makes the later pilot conversation easier.

Grade every criterion, then map the misses

The pass rule matters more than the score. In the healthcare part of DAYJOB, a task counts as passed only when every rubric criterion is satisfied. A near-complete answer can still miss the detail that matters. In regulated work, the missing part is often the part that gets you in trouble.

Use a simple protocol. The steps below are the minimum for a pre-pilot failure map.

• Run the candidate agent on a released full-workday task set, using the same files and prompt the benchmark provides. Done looks like a complete output package: drafts, tables, notes, decisions, and any required artifacts.
• Grade each rubric criterion separately, not the final answer as a whole. The target is one row per criterion, with pass, fail, and the exact reason for the fail.
• Log every miss by fact, file, decision, or deadline. Each failure needs a category, the evidence that caused it, and the step where it first appeared.
• Compare misses across tasks to find patterns. Recurring failure modes should surface, such as missed source citations, wrong calculations, skipped files, or late deliverables.

The usual mistake is treating the headline pass rate as a product review. A low pass rate can hide a safe agent for one narrow job and a dangerous one for another. The per-criterion verdicts are the actual agent failure analysis.

Turn the map into a pilot decision

Read the map before you choose a pilot scope. Repeated fact misses should keep it out of work where that fact matters. File-tracking misses call for a smaller file set or a human checkpoint after file review. Deadline misses need a test of timing, tooling, or task design.

Some failures are fixable with prompts, tools, or guardrails. Some are not. The map tells you which failure is controllable. A pilot should start there. If the fix is unclear, treat the failure as a scope limit.

Keep the map visible during the pilot. When the agent fails in production, compare the new failure to the pre-pilot map. If it matches a known pattern, you already know where to look. If it is new, add it to the map and tighten the scope until the pattern disappears.

When the map shows repeated misses in the same category, stop expanding the pilot and fix that category first.