UiPath CLI - uip eval eval-set & evaluation

uipath-cli

latest

false

UiPath CLI user guide

uip eval eval-set & evaluation

Syntax and options for managing Orchestrator eval sets and the evaluations (data points) inside them with `uip eval eval-set` and `uip eval evaluation`.

Part of the uip eval subsystem. This page covers the two resources that define what gets evaluated:

eval eval-set — a named, process-scoped collection of evaluation data points, linked to one or more evaluators via --evaluator-refs. An eval set also carries batching/timeout settings used when it's run.
eval evaluation — one data point inside an eval set: an input payload, an optional expected output/behavior, and optional per-evaluator criteria overrides. execute-and-evaluate and schedule runs work through an eval set's evaluations, scoring each one with the eval set's linked evaluators.

Synopsis

uip eval eval-set list --process-key <guid> [--limit <n>] [--offset <n>] [--tenant <tenant>]
uip eval eval-set get <evalSetId> --process-key <guid> [--tenant <tenant>]
uip eval eval-set create --process-key <guid> --workload-id <guid> --folder-key <guid> --name <name> [options...] [--tenant <tenant>]
uip eval eval-set update <evalSetId> --process-key <guid> [options...] [--tenant <tenant>]
uip eval eval-set delete <evalSetId> --process-key <guid> [--tenant <tenant>]

uip eval evaluation list --process-key <guid> --eval-set-id <guid> [--limit <n>] [--offset <n>] [--tenant <tenant>]
uip eval evaluation get <evaluationId> --process-key <guid> --eval-set-id <guid> [--tenant <tenant>]
uip eval evaluation create --process-key <guid> --eval-set-id <guid> --folder-key <guid> --name <name> --inputs <json> [options...] [--tenant <tenant>]
uip eval evaluation update <evaluationId> --process-key <guid> --eval-set-id <guid> [options...] [--tenant <tenant>]
uip eval evaluation delete <evaluationId> --process-key <guid> --eval-set-id <guid> [--tenant <tenant>]
uip eval eval-set list --process-key <guid> [--limit <n>] [--offset <n>] [--tenant <tenant>]
uip eval eval-set get <evalSetId> --process-key <guid> [--tenant <tenant>]
uip eval eval-set create --process-key <guid> --workload-id <guid> --folder-key <guid> --name <name> [options...] [--tenant <tenant>]
uip eval eval-set update <evalSetId> --process-key <guid> [options...] [--tenant <tenant>]
uip eval eval-set delete <evalSetId> --process-key <guid> [--tenant <tenant>]

uip eval evaluation list --process-key <guid> --eval-set-id <guid> [--limit <n>] [--offset <n>] [--tenant <tenant>]
uip eval evaluation get <evaluationId> --process-key <guid> --eval-set-id <guid> [--tenant <tenant>]
uip eval evaluation create --process-key <guid> --eval-set-id <guid> --folder-key <guid> --name <name> --inputs <json> [options...] [--tenant <tenant>]
uip eval evaluation update <evaluationId> --process-key <guid> --eval-set-id <guid> [options...] [--tenant <tenant>]
uip eval evaluation delete <evaluationId> --process-key <guid> --eval-set-id <guid> [--tenant <tenant>]

uip eval eval-set list

List eval sets defined for a process.

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key. Use `uip or processes list` to find one.
`--limit <n>`	integer	no	Maximum items to return. Default `100`.
`--offset <n>`	integer	no	Items to skip. Default `0`.
`--tenant <tenant>`	name	no	UiPath tenant name.

Example

uip eval eval-set list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
uip eval eval-set list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalSetList",
  "Data": [
    {
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Name": "Smoke Tests",
      "Description": "Basic smoke tests",
      "BatchSize": 10,
      "EvaluatorRefs": ["ev-1", "ev-2"],
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "Returned": 1, "Limit": 100, "Offset": 0, "HasMore": false }
}
{
  "Code": "EvalSetList",
  "Data": [
    {
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Name": "Smoke Tests",
      "Description": "Basic smoke tests",
      "BatchSize": 10,
      "EvaluatorRefs": ["ev-1", "ev-2"],
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "Returned": 1, "Limit": 100, "Offset": 0, "HasMore": false }
}

BatchSize is "-" when unset. UpdatedAt is present only once the eval set has been updated at least once.

uip eval eval-set get

Get one eval set's details.

Arguments

Name	Required	Purpose
`<evalSetId>`	yes	Eval set ID (GUID).

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--tenant <tenant>`	name	no	UiPath tenant name.

Example

uip eval eval-set get f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
uip eval eval-set get f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{
  "Code": "EvalSetDetails",
  "Data": {
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Smoke Tests",
    "Description": "Basic smoke tests",
    "BatchSize": 10,
    "EvaluatorRefs": ["ev-1", "ev-2"],
    "CreatedAt": "2026-08-01T10:00:00Z"
  }
}
{
  "Code": "EvalSetDetails",
  "Data": {
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Smoke Tests",
    "Description": "Basic smoke tests",
    "BatchSize": 10,
    "EvaluatorRefs": ["ev-1", "ev-2"],
    "CreatedAt": "2026-08-01T10:00:00Z"
  }
}

uip eval eval-set create

Create an eval set for a process.

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--workload-id <guid>`	GUID	yes	Workload ID.
`--folder-key <guid>`	GUID	yes	Folder key.
`--name <name>`	string	yes	Eval set name.
`--description <text>`	string	no	Eval set description.
`--batch-size <n>`	integer ≥ 1	no	Max concurrent evaluations per run.
`--timeout-minutes <n>`	integer ≥ 1	no	Timeout in minutes per evaluation.
`--evaluator-refs <refs...>`	space-separated IDs	no	Evaluator IDs to link to this eval set. Variadic — pass multiple IDs separated by spaces, not commas.
`--tenant <tenant>`	name	no	UiPath tenant name.

An internal externalId (a random UUID) is generated automatically on every create — it isn't user-settable.

Example

uip eval eval-set create --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --folder-key f1f2f3f4-0000-0000-0000-000000000001 \
  --name "Smoke Tests" --evaluator-refs ev-1 ev-2
uip eval eval-set create --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --workload-id a1b2c3d4-0000-0000-0000-000000000001 \
  --folder-key f1f2f3f4-0000-0000-0000-000000000001 \
  --name "Smoke Tests" --evaluator-refs ev-1 ev-2

Data shape (--output json)

{
  "Code": "EvalSetCreated",
  "Data": { "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04", "Name": "Smoke Tests" }
}
{
  "Code": "EvalSetCreated",
  "Data": { "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04", "Name": "Smoke Tests" }
}

uip eval eval-set update

Update an existing eval set. At least one updatable field is required beyond --process-key; fields you omit keep their current value (the command reads the existing eval set first, then applies your changes on top).

Arguments

Name	Required	Purpose
`<evalSetId>`	yes	Eval set ID (GUID).

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--name <name>`	string	no*	New eval set name.
`--description <text>`	string	no*	New description.
`--batch-size <n>`	integer ≥ 1	no*	New max concurrent evaluations.
`--timeout-minutes <n>`	integer ≥ 1	no*	New timeout in minutes.
`--evaluator-refs <refs...>`	space-separated IDs	no*	Replaces the full evaluator-refs list (not additive).
`--tenant <tenant>`	name	no	UiPath tenant name.

* At least one of --name, --description, --batch-size, --timeout-minutes, or --evaluator-refs is required — the command fails fast if none are passed.

Example

uip eval eval-set update f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --name "Updated Name" --evaluator-refs ev-1 ev-2
uip eval eval-set update f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --name "Updated Name" --evaluator-refs ev-1 ev-2

Data shape (--output json)

{
  "Code": "EvalSetUpdated",
  "Data": {
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Updated Name",
    "EvaluatorRefs": ["ev-1", "ev-2"]
  }
}
{
  "Code": "EvalSetUpdated",
  "Data": {
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Updated Name",
    "EvaluatorRefs": ["ev-1", "ev-2"]
  }
}

uip eval eval-set delete

Delete an eval set.

Arguments

Name	Required	Purpose
`<evalSetId>`	yes	Eval set ID (GUID).

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--tenant <tenant>`	name	no	UiPath tenant name.

Example

uip eval eval-set delete f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09
uip eval eval-set delete f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09

Data shape (--output json)

{ "Code": "EvalSetDeleted", "Data": { "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04" } }
{ "Code": "EvalSetDeleted", "Data": { "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04" } }

Deleting an eval set does not cascade-delete or otherwise warn about schedules or evaluators that reference it — the CLI issues a plain DELETE with no dependency check.

uip eval evaluation list

List the evaluations (data points) inside an eval set.

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--eval-set-id <guid>`	GUID	yes	Eval set ID.
`--limit <n>`	integer	no	Maximum items to return. Default `100`.
`--offset <n>`	integer	no	Items to skip. Default `0`.
`--tenant <tenant>`	name	no	UiPath tenant name.

Example

uip eval evaluation list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04
uip eval evaluation list --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04

Data shape (--output json)

{
  "Code": "EvaluationList",
  "Data": [
    {
      "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001",
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Name": "Greeting test",
      "Inputs": { "input": "hello" },
      "ExpectedOutput": { "content": "Hi there!" },
      "ExpectedBehavior": "",
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "Returned": 1, "Limit": 100, "Offset": 0, "HasMore": false }
}
{
  "Code": "EvaluationList",
  "Data": [
    {
      "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001",
      "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
      "Name": "Greeting test",
      "Inputs": { "input": "hello" },
      "ExpectedOutput": { "content": "Hi there!" },
      "ExpectedBehavior": "",
      "CreatedAt": "2026-08-01T10:00:00Z"
    }
  ],
  "Pagination": { "Returned": 1, "Limit": 100, "Offset": 0, "HasMore": false }
}

Simulations appears on an evaluation only when it has simulations. Each entry carries ComponentId, ComponentType, and SimulationStrategy (Llm, Mockito, or Static), plus ComponentDescription, SimulationInstruction, OutputSchema, MockValue, Behaviors, and ChildSimulations when set. UpdatedAt appears once the evaluation has been updated.

uip eval evaluation get

Get one evaluation's details.

Arguments

Name	Required	Purpose
`<evaluationId>`	yes	Evaluation ID (GUID).

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--eval-set-id <guid>`	GUID	yes	Eval set ID.
`--tenant <tenant>`	name	no	UiPath tenant name.

Example

uip eval evaluation get c1d2e3f4-0000-0000-0000-000000000001 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04
uip eval evaluation get c1d2e3f4-0000-0000-0000-000000000001 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04

Data shape (--output json)

Same object shape as one entry in evaluation list's Data array.

uip eval evaluation create

Add an evaluation (data point) to an eval set.

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--eval-set-id <guid>`	GUID	yes	Eval set ID.
`--folder-key <guid>`	GUID	yes	Folder key.
`--name <name>`	string	yes	Evaluation name.
`--inputs <json>`	JSON	yes	Input values, e.g. `'{"input":"hello"}'`.
`--expected-output <json>`	JSON	no	Expected output for comparison-style evaluators.
`--expected-behavior <text>`	string	no	Expected agent behavior description, for trajectory evaluators.
`--evaluation-criterias <json>`	JSON	no	Per-evaluator criteria overrides.
`--simulations <json>`	JSON array	no	Simulations for the components the process calls, for example `'[{"componentId":"web-search","componentType":"tool"}]'`. Each object needs `componentId` and `componentType`. For the LLM simulation strategy (the default), the CLI fills in a missing `outputSchema` from the deployed agent. Refer to `uip eval simulatable-components list` for the available components.
`--tenant <tenant>`	name	no	UiPath tenant name.

Malformed JSON on --inputs, --expected-output, --evaluation-criterias, or --simulations fails fast client-side, with an example of the expected shape in the error for --inputs and --simulations. A simulation without a componentId or componentType fails with the index of the invalid entry.

Example

uip eval evaluation create --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --folder-key f1f2f3f4-0000-0000-0000-000000000001 \
  --name "Greeting test" --inputs '{"input":"hello"}' --expected-output '{"content":"Hi there!"}'
uip eval evaluation create --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 \
  --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --folder-key f1f2f3f4-0000-0000-0000-000000000001 \
  --name "Greeting test" --inputs '{"input":"hello"}' --expected-output '{"content":"Hi there!"}'

Data shape (--output json)

{
  "Code": "EvaluationCreated",
  "Data": {
    "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001",
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Greeting test"
  }
}
{
  "Code": "EvaluationCreated",
  "Data": {
    "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001",
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Greeting test"
  }
}

uip eval evaluation update

Update an existing evaluation. At least one updatable field is required beyond --process-key/--eval-set-id; omitted fields keep their current value.

Arguments

Name	Required	Purpose
`<evaluationId>`	yes	Evaluation ID (GUID).

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--eval-set-id <guid>`	GUID	yes	Eval set ID.
`--name <name>`	string	no*	New evaluation name.
`--inputs <json>`	JSON	no*	New input values.
`--expected-output <json>`	JSON	no*	New expected output.
`--expected-behavior <text>`	string	no*	New expected behavior description.
`--evaluation-criterias <json>`	JSON	no*	New per-evaluator criteria overrides.
`--simulations <json>`	JSON array	no*	New simulations, replacing the current set. Same shape and validation as on `evaluation create`.
`--tenant <tenant>`	name	no	UiPath tenant name.

* At least one of --name, --inputs, --expected-output, --expected-behavior, --evaluation-criterias, or --simulations is required.

Example

uip eval evaluation update c1d2e3f4-0000-0000-0000-000000000001 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --inputs '{"input":"updated"}' --expected-output '{"content":"new response"}'
uip eval evaluation update c1d2e3f4-0000-0000-0000-000000000001 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04 \
  --inputs '{"input":"updated"}' --expected-output '{"content":"new response"}'

Data shape (--output json)

{
  "Code": "EvaluationUpdated",
  "Data": {
    "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001",
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Greeting test"
  }
}
{
  "Code": "EvaluationUpdated",
  "Data": {
    "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001",
    "EvalSetId": "f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04",
    "Name": "Greeting test"
  }
}

uip eval evaluation delete

Remove an evaluation from an eval set.

Arguments

Name	Required	Purpose
`<evaluationId>`	yes	Evaluation ID (GUID).

Options

Long	Value	Required	Description
`--process-key <guid>`	GUID	yes	Process key.
`--eval-set-id <guid>`	GUID	yes	Eval set ID.
`--tenant <tenant>`	name	no	UiPath tenant name.

Example

uip eval evaluation delete c1d2e3f4-0000-0000-0000-000000000001 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04
uip eval evaluation delete c1d2e3f4-0000-0000-0000-000000000001 \
  --process-key 9e4b2f17-7c3a-4d81-b592-3f6e8a1d5c09 --eval-set-id f3a7d219-8b4c-4e62-a951-7d3f6e2c8b04

Data shape (--output json)

{ "Code": "EvaluationDeleted", "Data": { "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001" } }
{ "Code": "EvaluationDeleted", "Data": { "EvaluationId": "c1d2e3f4-0000-0000-0000-000000000001" } }

uip eval — overview, execute-and-evaluate, and run.
uip eval schedule — recurring evaluation runs against an eval set.
uip eval evaluator — the scoring logic an eval set's --evaluator-refs point to.
uip or processes — resolve a --process-key.

Synopsis​

uip eval eval-set list​

Options​

Example​

Data shape (--output json)​

uip eval eval-set get​

Arguments​

Options​

Example​

Data shape (--output json)​

uip eval eval-set create​

Options​

Example​

Data shape (--output json)​

uip eval eval-set update​

Arguments​

Options​

Example​

Data shape (--output json)​

uip eval eval-set delete​

Arguments​

Options​

Example​

Data shape (--output json)​

uip eval evaluation list​

Options​

Example​

Data shape (--output json)​

uip eval evaluation get​

Arguments​

Options​

Example​

Data shape (--output json)​

uip eval evaluation create​

Options​

Example​

Data shape (--output json)​

uip eval evaluation update​

Arguments​

Options​

Example​

Data shape (--output json)​

uip eval evaluation delete​

Arguments​

Options​

Example​

Data shape (--output json)​

Related​

See also​

Was this page helpful?

Synopsis

uip eval eval-set list

Options

Example

Data shape (--output json)

uip eval eval-set get

Arguments

Options

Example

Data shape (--output json)

uip eval eval-set create

Options

Example

Data shape (--output json)

uip eval eval-set update

Arguments

Options

Example

Data shape (--output json)

uip eval eval-set delete

Arguments

Options

Example

Data shape (--output json)

uip eval evaluation list

Options

Example

Data shape (--output json)

uip eval evaluation get

Arguments

Options

Example

Data shape (--output json)

uip eval evaluation create

Options

Example

Data shape (--output json)

uip eval evaluation update

Arguments

Options

Example

Data shape (--output json)

uip eval evaluation delete

Arguments

Options

Example

Data shape (--output json)

Related

See also