UiPath Documentation
uipath-cli
latest
false
UiPath CLI user guide

How-to: run tests from the CLI

Run a Test Manager test set from UiPath CLI as part of a CI pipeline and correctly interpret the execution verdict.

This page shows how to run a UiPath Test Manager test set from the CLI as part of a CI pipeline — and, just as importantly, how to read the verdict correctly. The uip tm tool splits a test set run into three verbs for a reason; using only one of them can produce a passing build even when tests have failed.

The golden rule: launch → wait → verify​

For a test set run, no single uip tm verb exits non-zero because tests failed. Getting a pipeline to fail on a red test set run takes three commands (a package run takes one, see Run every test in an uploaded package):

  1. Launch — uip tm testsets run. Queues the run and exits 0 as soon as Orchestrator accepts the request. Status: "Running" in the response reflects the state at launch, not the final outcome.
  2. Block — uip tm wait. Polls until the execution reaches a terminal state (Finished or Cancelled). Exits 0 when it reaches any terminal state — because "finished" is the success signal for wait. Exits 2 on --timeout (a domain-specific reuse of the authentication-error slot, so scripts can branch on "took too long" without parsing text).
  3. Verify — uip tm report get. Reads Data.Failed (and Passed, None, PassRate). Exits 0 whenever it successfully fetches the summary, regardless of pass/fail — the script is responsible for turning Data.Failed > 0 into a non-zero exit.

A launch+wait shortcut does exist for test sets — uip tm testsets run --wait launches and blocks in one call (--timeout/--poll-interval tune the poll) — but it still isn't launch+wait+verify: a Finished run exits 0 even when tests inside it failed (only a Cancelled run exits 1). You still need the separate verify step to catch a red run. Always write the three steps (or the two-step run --wait plus the same report get verify at the end).

Run every test in an uploaded package​

When the tests ship in a package already uploaded to Orchestrator, uip tm run --type package launches, waits, and verifies in one call. It runs every test case in the package, with no test set to author, writes a JUnit or UiPath report to --result-path, and exits 1 when a test fails or reports no result, no test runs, or the run is cancelled, so the pipeline fails on a red run without a separate verify step.

uip tm run --type package \
  --project-key DEMO \
  --package-name MyTests \
  --package-version 1.0 \
  --folder-path Shared \
  --result-path ./results/junit.xml
uip tm run --type package \
  --project-key DEMO \
  --package-name MyTests \
  --package-version 1.0 \
  --folder-path Shared \
  --result-path ./results/junit.xml

Leave out --project-key to run the package in the tenant default project; Data.ProjectKey in the output names the project the tests ran in.

Exit codeMeaning
0Every test passed.
1A test failed or reported no result, the run was cancelled, no test ran, or a service call failed. A missing login or no access to a project you passed also exits 1, before the run starts.
2Login or permission problem after the run starts, or no permission to open or create the tenant default project.
3Bad option or input.
4The run exceeded --timeout.

Minimum viable snippet​

#!/usr/bin/env bash
set -euo pipefail

# 1. Launch
EXECUTION_ID=$(uip tm testsets run \
  --test-set-key DEMO:10 \
  --output-filter "ExecutionId" \
  --output plain)

# 2. Block (default timeout: 30 min; adjust with --timeout)
if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --timeout 1800; then
  code=$?
  if [ "$code" -eq 2 ]; then
    echo "test run did not finish within 30 minutes" >&2
    exit 2
  fi
  echo "wait failed (exit $code)" >&2
  exit "$code"
fi

# 3. Verify
FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"
#!/usr/bin/env bash
set -euo pipefail

# 1. Launch
EXECUTION_ID=$(uip tm testsets run \
  --test-set-key DEMO:10 \
  --output-filter "ExecutionId" \
  --output plain)

# 2. Block (default timeout: 30 min; adjust with --timeout)
if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --timeout 1800; then
  code=$?
  if [ "$code" -eq 2 ]; then
    echo "test run did not finish within 30 minutes" >&2
    exit 2
  fi
  echo "wait failed (exit $code)" >&2
  exit "$code"
fi

# 3. Verify
FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"

--output plain returns the bare value for a scalar filter result; this is the safest way to capture a field into a shell variable without parsing JSON. See Scripting patterns — reading values out of the envelope.

Picking inputs to the commands​

uip tm testsets run needs the test set key, not its UUID. The key has the form <ProjectKey>:<Number> — for example, DEMO:10. Two ways to find it:

# From the Test Manager UI — it is the "Key" column in the test set list.

# Or from the CLI:
uip tm testsets list --project-key DEMO --filter smoke \
  --output-filter "[*].{key:TestSetKey, name:Name}"
# From the Test Manager UI — it is the "Key" column in the test set list.

# Or from the CLI:
uip tm testsets list --project-key DEMO --filter smoke \
  --output-filter "[*].{key:TestSetKey, name:Name}"

list takes the project as --project-key or --project-id. run derives the project from the test set key's prefix (DEMO:10 → project DEMO).

If you need the numeric UUID for uip tm executions list --test-set-id, that is the Id field on the same list output — not the TestSetKey.

Picking which cases run​

uip tm testsets run --execution-type controls which test cases inside the set run:

  • automated (default) — only automated test cases.
  • manual — only manual test cases.
  • mixed — both.
  • none — no type filter.

For CI, keep the default (automated) — manual cases require a human to mark pass/fail and will sit in InProgress until --timeout.

Passing parameter overrides​

If the test set exposes parameters (for example, a target URL or an account ID), override them at launch with --input-path:

cat > ./params.json <<'EOF'
[
  {"name": "TargetUrl", "type": "String", "value": "https://staging.example.com"},
  {"name": "AccountId", "type": "String", "value": "acme-staging"}
]
EOF

uip tm testsets run --test-set-key DEMO:10 --input-path ./params.json
cat > ./params.json <<'EOF'
[
  {"name": "TargetUrl", "type": "String", "value": "https://staging.example.com"},
  {"name": "AccountId", "type": "String", "value": "acme-staging"}
]
EOF

uip tm testsets run --test-set-key DEMO:10 --input-path ./params.json

Overrides are matched against the test set's current parameter definitions by name (case-insensitive) and, when present, type. If the server reports no parameter definitions, the inputs are sent as-is. The overrides are saved on the test set, so later runs without --input-path reuse them.

Reading the verdict in detail​

uip tm report get is the summary reader. Default output gives you the scorecard; pass --query (a jq-style filter) to extract a subset in one call:

# Pretty scorecard
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO

# Script-friendly — JSON with just the counts
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO \
  --query '{total: .TotalTests, passed: .Passed, failed: .Failed}'
# Pretty scorecard
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO

# Script-friendly — JSON with just the counts
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO \
  --query '{total: .TotalTests, passed: .Passed, failed: .Failed}'

The Data envelope has these fields (full schema in the report get reference):

  • TotalTests, Passed, Failed, None (tests with no result), PassRate (a number such as 80, computed over the tests that passed or failed), Duration (HH:MM:SS).
  • FailedTests — one entry per failed test case, with TestCaseName and an Error string (either the log's info field, or a concatenated list of failed assertion messages).

When you need a JUnit-XML file that your CI's test dashboard can ingest, use uip tm result download instead of (or alongside) report get.

Surfacing failed cases in CI logs​

Turn the FailedTests array into a human-readable line per failure before exiting the pipeline:

if [ "$FAILED" -gt 0 ]; then
  uip tm report get \
    --execution-id "$EXECUTION_ID" \
    --project-key DEMO \
    --output-filter "FailedTests[*].[TestCaseName, Error]" \
    --output plain >&2
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi
if [ "$FAILED" -gt 0 ]; then
  uip tm report get \
    --execution-id "$EXECUTION_ID" \
    --project-key DEMO \
    --output-filter "FailedTests[*].[TestCaseName, Error]" \
    --output plain >&2
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

For deeper failure analysis — assertion-by-assertion detail on a single failed case — use uip tm testcaselog list-assertions on the log IDs returned by uip tm executions testcaselogs list --only-failed.

Retrying flaky cases​

If a run has failures that might be flaky (timing, environment), retry only the failed cases in place:

uip tm executions retry \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO
uip tm executions retry \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO

uip tm executions retry reuses the same execution ID — no new one is created. If there are no failures to retry, it prints an informational message and exits 0 (intentionally not an error).

After retry, loop back through wait + report get on the same execution ID.

Common pitfalls​

  • Parsing testsets run output to decide pass/fail. The Status field at launch is typically Running — it means the run was queued, not that every test passed. Always call wait and then report get.
  • Forgetting the --timeout on wait. Default is 1800 seconds (30 minutes). Pass 0 to wait indefinitely (rarely what you want in CI); pass a larger number for long suites.
  • Treating exit 2 from wait as an auth failure. On wait specifically, 2 means timeout, not AuthenticationError — see uip tm wait exit codes.
  • Using testsets verbs to create tests. Test cases and their links to Orchestrator automations come from uip tm testcases; testsets only groups them.

Full CI-ready example​

A bash snippet for a pipeline step — fails the build when any test fails, times out, or errors:

#!/usr/bin/env bash
set -euo pipefail

TEST_SET_KEY="${TEST_SET_KEY:?TEST_SET_KEY is required}"
PROJECT_KEY="${PROJECT_KEY:?PROJECT_KEY is required}"
TIMEOUT="${TEST_TIMEOUT:-1800}"

EXECUTION_ID=$(uip tm testsets run \
  --test-set-key "$TEST_SET_KEY" \
  --output-filter "ExecutionId" \
  --output plain)

echo "started execution $EXECUTION_ID" >&2

if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --timeout "$TIMEOUT"; then
  code=$?
  case "$code" in
    2) echo "execution $EXECUTION_ID did not finish within ${TIMEOUT}s" >&2; exit 2 ;;
    *) echo "wait failed with exit $code" >&2; exit "$code" ;;
  esac
fi

# Get the scorecard and the names of any failed cases in one shot
uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY"

FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"
#!/usr/bin/env bash
set -euo pipefail

TEST_SET_KEY="${TEST_SET_KEY:?TEST_SET_KEY is required}"
PROJECT_KEY="${PROJECT_KEY:?PROJECT_KEY is required}"
TIMEOUT="${TEST_TIMEOUT:-1800}"

EXECUTION_ID=$(uip tm testsets run \
  --test-set-key "$TEST_SET_KEY" \
  --output-filter "ExecutionId" \
  --output plain)

echo "started execution $EXECUTION_ID" >&2

if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --timeout "$TIMEOUT"; then
  code=$?
  case "$code" in
    2) echo "execution $EXECUTION_ID did not finish within ${TIMEOUT}s" >&2; exit 2 ;;
    *) echo "wait failed with exit $code" >&2; exit "$code" ;;
  esac
fi

# Get the scorecard and the names of any failed cases in one shot
uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY"

FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"

Run this script after your Solution deploy step in any of the CI recipes. The only env-specific config is TEST_SET_KEY / PROJECT_KEY — set them per environment in the pipeline, and the same script promotes cleanly across dev/stage/prod.

See also​

Was this page helpful?

Connect

Need help? Support(opens in new window)

Want to learn? UiPath Academy(opens in new window)

Have questions? UiPath Forum(opens in new window)

Stay updated