Runtime evaluations for validating deployed agents against datasets of inputs and expected outputs, on demand or on a recurring schedule.

An agent that passed its tests at design time can still fail in production. Models get updated, inputs drift, and the tools your agent calls change underneath it. If you only evaluate before you ship, you find out about these failures from your users.

Runtime evaluations run against the deployed agent itself, so the results describe what is in production, not a design-time copy. Running them after every change and on a schedule surfaces regressions early, the way a failing test does in CI.

To evaluate an agent while you're still building it, see Evaluations.

An agent can return the right answer by the wrong route: it calls a tool it shouldn't have, skips a step it was supposed to take, or loops three times before it gets there. A check on the final output won't catch any of that, and in production those are the failures that cost you.

Each data point is scored by the evaluators linked to its dataset. Use more than one kind:

Every evaluator result comes with a justification and a full execution trace, which show why a score dropped.

A dataset is a set of data points. Each one holds the inputs you send to the agent and the output you expect back. You can write data points by hand, but you don't have to. There are three other sources, and production runs are the most representative because they include inputs you didn't think to test:

Before you rely on a data point, review its expected output. The data point form matches your agent's input and expected output schemas, and it warns you when the expected output doesn't match the keys in the agent's output.

An evaluation run sends each data point to your deployed agent. If the agent's tools are live, the run triggers them: it sends the email, writes the record, and creates the escalation for a human to handle. If you don't want those downstream effects in production, simulate every tool and escalation that changes something outside the agent. This includes human-in-the-loop (HITL) steps, so an evaluation doesn't put work in a person's queue.

Each simulated component starts with its output schema filled in. You can simulate it in two ways:

You can start an evaluation run on demand or from a schedule.

A schedule runs a whole dataset. To create one:

Each scheduled run uses the dataset as it is when the run starts. If you add, edit, or remove data points, the next run picks up the changes automatically, so you don't need to reschedule.

After you create a schedule, you can pause and resume it, change how often it runs, or delete it. The Schedules view lists only the schedules in the deployed agent's folder.

Open a run in the Eval runs view to see:

An overall score tells you whether something changed. It doesn't tell you what. Selecting a data point shows:

When a data point's score drops, either the agent is wrong or the expected output is. If the agent is wrong, that's a regression. If the expected output is wrong, the data point needs fixing so the dataset stays trustworthy.

Open Agent Registry > Agents and select a deployed agent. Its Evaluations tab has four views:

What you can do on the Evaluations tab depends on your permissions in the deployed agent's folder. Without them, editing and run actions are hidden or unavailable, and the tab is read-only.

Runtime evaluations are stored separately from the agent package. You can change datasets, data points, and evaluators without republishing the agent. Files attached to data points live in Orchestrator storage buckets, and you manage them from the Evaluations tab, not the Orchestrator Buckets page.

You can turn a completed run of your deployed agent into a data point. The run's output becomes the data point's expected output, so check that it's correct before you add it. A wrong answer saved here becomes the answer future runs are scored against.

A confirmation message names the dataset, and the data point appears in it on the Evaluations tab.