Score Your AI Agent on What It Learned, Not What It Knows

The demo is not the workflow

You watch an agent answer a question from a public dataset, then it stalls when your process changes. A demo is a party; a harness is a lab. A useful agent evaluation harness should test what the agent learned under declared access, not what it already knows. The test should be repeatable enough to run after every process change, before you trust it with real work.

Brackett launched the Agent Effectiveness Index, or AEI, as a free open-source benchmark for evaluating AI agents on process comprehension, proactive action, and continued learning as processes change. Use it as the frame for your own harness.

The harness comes before the agent

Start with the AEI specification. It combines business understanding, operational execution, and learning persistence into a 0-100 score, assigning 40 percent to the first two components and 20 percent to learning persistence, and applies a geometric mean. A geometric mean makes a weak component hard to hide.

Declare the access ladder. AEI defines four declared access levels: K0 is the starting system, K1 adds documents, K2 adds a graph with a base navigator, and K3 uses that graph with a trained navigator. Record which knowledge the agent can see and which it cannot.

Write the prohibited actions before the first run. Under the AEI reference safety gate, a critical prohibited action makes an episode's execution score zero and causes the gate to fail. A failed gate should stop the comparison, even if the other numbers look strong.

Choose tasks with a difficulty spread. Show & Tell uses TCI as a task-selection heuristic to spread benchmark difficulty and organizes tasks by rules, evidence, steps, and demonstration. Use that template for the task card: what the agent must infer, what it must not do, and how success is shown.

Repeatable order makes the eval useful

Freeze the workflow first. Write down the task, the inputs, the expected output, and the single process change you will test. The result should be a versioned task file that another person could run without guessing.

Run a baseline episode. Give the agent the declared access profile and let it complete the task without the process change. Save the component scores, the gate result, and the task metadata so the next run becomes an agent regression test.

Run the changed episode. Apply the process change, keep the access profile the same, and record what the agent did differently. The rows should differ only in the rule, the evidence, or the step order.

Keep the access profile in the filename. A K0 run and a K3 run are not comparable unless the task, rubric, and gate are identical. A folder where the process change is the only variable is easier to defend.

The lab note comes before the total

Read learning persistence first, then operational execution, then business understanding. If the agent improves after a process change, the learning score should move. If it does not, the total may still look respectable because the other parts are strong.

Use the access profile as the comparison key. An agent that scores well at K3 is not the same result as an agent that scores well at K0. The access level tells you how much scaffolding the agent needed.

Treat public snapshots with care. The preliminary Show & Tell snapshot contained 191 selected attempts across 39 tested cases. The snapshot did not report a confidence interval and was not a verified canonical leaderboard. That is enough to see a pattern, not enough to crown a winner.

Finish by asking what the agent learned. Did it notice the new rule? Did it avoid the dangerous move? Did it keep the useful path after the change? If yes, you have a candidate for real work. If no, put the next candidate through the same harness.