UiPath CLI - uip eval

uipath-cli

latest

false

UiPath CLI user guide

uip eval

Syntax and options for `uip eval`, which runs runtime evaluations against Orchestrator packages and manages evaluators, eval sets, evaluations, and schedules.

uip eval runs runtime evaluations against a deployed Orchestrator package: submit real inputs through a process, score the outputs with one or more evaluators, and inspect the results. It also manages the reusable building blocks behind that workflow — evaluators (scoring logic), eval sets (named collections that group evaluations), evaluations (individual saved test-case definitions — name, inputs, expected output/behavior), and recurring schedules that trigger runs automatically.

This resource spans four pages, split by sub-domain:

This page — concepts, execute-and-evaluate / run (submit a run, then list/inspect its results), and simulatable-components (list the components an evaluation can simulate).
schedule — create/list/get/update/pause/resume/delete recurring eval runs.
evaluator — manage the scoring logic (e.g. LLM-judge evaluators) a run applies to its outputs.
eval-set and evaluation — manage curated item collections and individual evaluation definitions.

uip eval ships as its own standalone package (@uipath/eval-tool, command prefix eval) — it is not a sub-command of uip or, even though every verb here operates against an Orchestrator process/folder. For the shared Orchestrator conventions (auth, folder scoping, key types, output envelope) that this tool's --process-key/--folder-key/--tenant options follow, see the uip or overview.

Concepts

Eval item — one test case passed inline to execute-and-evaluate: an id, a name, and an inputs object matching the process's expected input schema. In inline mode, a run submits an array of these directly on the command line with --items.
Evaluation — the saved counterpart to an eval item: a persisted test-case definition (name, inputs, and optionally an expected output/behavior) that belongs to an eval set. Manage these with eval evaluation.
Evaluator — scoring logic applied to each item's output, identified by an evaluatorTypeId (for example uipath-llm-judge-output-semantic-similarity) plus an evaluatorConfig object. Manage saved evaluators with eval evaluator; a run can also pass evaluator configs inline via --evaluators.
Eval set — a named, reusable collection of evaluations, managed with eval eval-set. execute-and-evaluate runs in one of two modes: in eval-set mode, --eval-set-id alone fetches the items and evaluators from the saved eval set; in inline mode, --items and --evaluators supply them on the command line.
Eval set run vs. eval run — submitting execute-and-evaluate creates one eval set run (the overall batch, identified by EvalSetRunId), which fans out into one eval run per item (run results lists these individually, each with its own Status/Result).
Simulation — simulates a component the process calls (a tool or an escalation) when an evaluation runs. Attach simulations to a saved evaluation with --simulations. List the components a deployed process exposes with simulatable-components list.
Folder scoping — execute-and-evaluate resolves a folder from --folder-key. When --folder-key is omitted, it uses your personal workspace in inline mode (the CLI looks it up automatically) and the eval set's folder in eval-set mode. This differs from most uip or verbs, which require an explicit folder selector. Refer to the overview for the general convention.

Synopsis

uip eval execute-and-evaluate --process-key <guid> --eval-set-id <id> [--batch-size <n>] [--folder-key <key>] [--tenant <name>]
uip eval execute-and-evaluate --process-key <guid> --workload-id <guid> --items <json> --evaluators <json> [--eval-set-id <id>] [--batch-size <n>] [--folder-key <key>] [--tenant <name>]
uip eval run list    --process-key <guid> [--limit <n>] [--offset <n>] [--tenant <name>]
uip eval run get     <evalSetRunId> --process-key <guid> [--tenant <name>]
uip eval run results <evalSetRunId> --process-key <guid> [--tenant <name>]
uip eval simulatable-components list --process-key <guid> [--process-version <version>] [--tenant <name>]
uip eval execute-and-evaluate --process-key <guid> --eval-set-id <id> [--batch-size <n>] [--folder-key <key>] [--tenant <name>]
uip eval execute-and-evaluate --process-key <guid> --workload-id <guid> --items <json> --evaluators <json> [--eval-set-id <id>] [--batch-size <n>] [--folder-key <key>] [--tenant <name>]
uip eval run list    --process-key <guid> [--limit <n>] [--offset <n>] [--tenant <name>]
uip eval run get     <evalSetRunId> --process-key <guid> [--tenant <name>]
uip eval run results <evalSetRunId> --process-key <guid> [--tenant <name>]
uip eval simulatable-components list --process-key <guid> [--process-version <version>] [--tenant <name>]

execute-and-evaluate has an alias: run-offline-evals.

uip eval execute-and-evaluate

Submit a runtime eval run for a process: send a batch of items through the process identified by --process-key, score each output with the evaluators, and return the EvalSetRunId to poll with run get/run results.

The command runs in one of two modes:

Mode	Pass	Items and evaluators come from
Eval-set mode	`--eval-set-id` without `--items` and `--evaluators`	The saved eval set: its evaluations become the items, and its evaluator refs select the saved evaluators. The workload ID, folder, and batch size default to the eval set's values.
Inline mode	`--items`, `--evaluators`, and `--workload-id`	The command line. `--eval-set-id` is optional; when omitted, the run uses the zero GUID as its eval set ID.

The command fails when you pass neither --eval-set-id nor --items/--evaluators, or only one of --items and --evaluators. In eval-set mode, it also fails when the eval set has no evaluations or no evaluator refs, or when an evaluator ref doesn't match a saved evaluator of the process.

Options

Short	Long	Value	Default	Description
—	`--process-key`	GUID	—	Required. Process key. Use `uip or processes list` to find available keys.
—	`--eval-set-id`	ID	—	Eval set ID. Without `--items`/`--evaluators`, the run fetches its items and evaluators from this eval set.
—	`--workload-id`	GUID	—	Workload ID. Required in inline mode.
—	`--items`	JSON array	—	Inline mode. Eval items, each with `id` (string), `name` (string), and `inputs` (object). Validated client-side before the call — a missing/malformed field on any item fails fast, naming the item index.
—	`--evaluators`	JSON array	—	Inline mode. Evaluator configs, each with an `id`, `evaluatorTypeId`, and `evaluatorConfig`.
—	`--batch-size`	number	`5` in inline mode; the eval set's value in eval-set mode	Max concurrent evaluation pipelines.
—	`--folder-key`	GUID	personal workspace in inline mode; the eval set's folder in eval-set mode	Folder to run in. Use `uip or folders list` to find available keys.
—	`--tenant`	name	session default	UiPath tenant name. Plain option — see Concepts.

Examples

# Run from a saved eval set (fetches items and evaluators automatically)
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04
# Run from a saved eval set (fetches items and evaluators automatically)
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04

# Submit a run with an ad hoc item and an inline evaluator
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]'
# Submit a run with an ad hoc item and an inline evaluator
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --items '[{"id":"1","name":"greeting","inputs":{"input":"hello"}}]' \
  --evaluators '[{"id":"ev-1","evaluatorTypeId":"uipath-llm-judge-output-semantic-similarity","evaluatorConfig":{}}]'

# Run from a saved eval set in a specific folder, with a larger batch
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --folder-key a9f3b2c1-7d4e-4a8b-9c2f-5e1d3b6a8f7e \
  --batch-size 10
# Run from a saved eval set in a specific folder, with a larger batch
uip eval execute-and-evaluate --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --folder-key a9f3b2c1-7d4e-4a8b-9c2f-5e1d3b6a8f7e \
  --batch-size 10

Data shape (--output json)

Inline mode, with --folder-key omitted:

{
  "Code": "EvalRunSubmitted",
  "Data": {
    "ProcessKey": "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09",
    "Folder": "[email protected]'s workspace",
    "EvalSetId": "00000000-0000-0000-0000-000000000000",
    "EvalSetRunId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04"
  }
}
{
  "Code": "EvalRunSubmitted",
  "Data": {
    "ProcessKey": "9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09",
    "Folder": "[email protected]'s workspace",
    "EvalSetId": "00000000-0000-0000-0000-000000000000",
    "EvalSetRunId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04"
  }
}

Folder is the folder key the run uses: the key you passed with --folder-key, or the eval set's folder key in eval-set mode. In inline mode with --folder-key omitted, it is the resolved personal-workspace name.

uip eval run

Inspect eval set runs created by execute-and-evaluate, scoped by --process-key.

uip eval run list

List eval set runs for a process.

Options

Short	Long	Value	Default	Description
—	`--process-key`	GUID	—	Required. Process key (deployment GUID).
—	`--limit`	number	`50`	Max runs to return.
—	`--offset`	number	`0`	Number of runs to skip.
—	`--tenant`	name	session default	UiPath tenant name.

Example

uip eval run list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
uip eval run list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalSetRunList",
  "Data": [
    {
      "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Status": "completed",
      "Score": 0.86,
      "EvalsExecuted": 5,
      "Duration": "42.5s",
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "Returned": 1, "Limit": 50, "Offset": 0, "HasMore": false }
}
{
  "Code": "EvalSetRunList",
  "Data": [
    {
      "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Status": "completed",
      "Score": 0.86,
      "EvalsExecuted": 5,
      "Duration": "42.5s",
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "Returned": 1, "Limit": 50, "Offset": 0, "HasMore": false }
}

Score and Duration are "-" when not yet available (run still in progress). Duration is formatted as Nms under one second, N.Ns otherwise.

uip eval run get

Get details of one eval set run.

Arguments

Name	Required	Purpose
`<evalSetRunId>`	yes	Eval set run ID (GUID), from `run list` or `execute-and-evaluate`'s output.

Options

Short	Long	Value	Default	Description
—	`--process-key`	GUID	—	Required. Process key (deployment GUID).
—	`--tenant`	name	session default	UiPath tenant name.

Example

uip eval run get a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
uip eval run get a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalSetRunDetails",
  "Data": {
    "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
    "Status": "completed",
    "Score": 0.86,
    "EvalsExecuted": 5,
    "Duration": "42.5s"
  }
}
{
  "Code": "EvalSetRunDetails",
  "Data": {
    "EvalSetRunId": "a1b2c3d4-0000-0000-0000-000000000101",
    "Status": "completed",
    "Score": 0.86,
    "EvalsExecuted": 5,
    "Duration": "42.5s"
  }
}

Same shape as one run list entry (minus EvalSetId/CreatedAt), fetched by ID instead of listed.

uip eval run results

View per-item eval run results for an eval set run — one entry per item submitted in the original execute-and-evaluate call.

Arguments

Name	Required	Purpose
`<evalSetRunId>`	yes	Eval set run ID (GUID).

Options

Short	Long	Value	Default	Description
—	`--process-key`	GUID	—	Required. Process key (deployment GUID).
—	`--tenant`	name	session default	UiPath tenant name.

Example

uip eval run results a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
uip eval run results a1b2c3d4-0000-0000-0000-000000000101 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalRunResults",
  "Data": [
    {
      "EvalRunId": "e1e2e3e4-0000-0000-0000-000000000001",
      "DataPoint": "Test Case 1",
      "Status": "completed",
      "Result": {}
    }
  ]
}
{
  "Code": "EvalRunResults",
  "Data": [
    {
      "EvalRunId": "e1e2e3e4-0000-0000-0000-000000000001",
      "DataPoint": "Test Case 1",
      "Status": "completed",
      "Result": {}
    }
  ]
}

DataPoint is the originating item's name (falls back to the eval run's own ID if the snapshot has none). Result is the evaluator's raw scoring payload — shape depends on which evaluator ran.

uip eval simulatable-components

List the components (tools, escalations) of a deployed process that an evaluation can simulate. Use the Id and ComponentType values to build the --simulations value of eval evaluation create and eval evaluation update.

uip eval simulatable-components list

List simulatable components for a deployed process.

Options

Short	Long	Value	Default	Description
—	`--process-key`	GUID	—	Required. Process key. Use `uip or processes list` to find keys.
—	`--process-version`	semantic version	latest deployed version	Process version to read the components from.
—	`--tenant`	name	session default	UiPath tenant name.

Example

uip eval simulatable-components list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
uip eval simulatable-components list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "SimulatableComponentList",
  "Data": [
    {
      "Id": "web-search",
      "Name": "Web Search",
      "Description": "Searches the web",
      "ComponentType": "tool",
      "SubType": "Api",
      "IsEnabled": true
    }
  ]
}
{
  "Code": "SimulatableComponentList",
  "Data": [
    {
      "Id": "web-search",
      "Name": "Web Search",
      "Description": "Searches the web",
      "ComponentType": "tool",
      "SubType": "Api",
      "IsEnabled": true
    }
  ]
}

OutputSchema is included on a component only when the process defines one for it.

schedule — automate execute-and-evaluate on a recurring schedule.
evaluator — manage saved scoring logic.
eval-set and evaluation — manage saved item collections and evaluation definitions.

uip eval

Concepts

Synopsis

uip eval execute-and-evaluate

Options

Examples

Data shape (--output json)

uip eval run

uip eval run list

Options

Example

Data shape (--output json)

uip eval run get

Arguments

Options

Example

Data shape (--output json)

uip eval run results

Arguments

Options

Example

Data shape (--output json)

uip eval simulatable-components

uip eval simulatable-components list

Options

Example

Data shape (--output json)

See also

Was this page helpful?

Concepts​

Synopsis​

uip eval execute-and-evaluate​

Options​

Examples​

Data shape (--output json)​

uip eval run​

uip eval run list​

Options​

Example​

Data shape (--output json)​

uip eval run get​

Arguments​

Options​

Example​

Data shape (--output json)​

uip eval run results​

Arguments​

Options​

Example​

Data shape (--output json)​

uip eval simulatable-components​

uip eval simulatable-components list​

Options​

Example​

Data shape (--output json)​

Related​

See also​

Was this page helpful?

Concepts

Synopsis

uip eval execute-and-evaluate

Options

Examples

Data shape (--output json)

uip eval run

uip eval run list

Options

Example

Data shape (--output json)

uip eval run get

Arguments

Options

Example

Data shape (--output json)

uip eval run results

Arguments

Options

Example

Data shape (--output json)

uip eval simulatable-components

uip eval simulatable-components list

Options

Example

Data shape (--output json)

Related

See also