- Getting started
- Best practices
- Organization Modeling in Orchestrator
- Managing Large Deployments
- Automation Best Practices
- Optimizing Unattended Infrastructure Using Machine Templates
- Organizing Resources With Tags
- Orchestrator Read-only Replica
- Exporting grids in the background
- Enforcing user-level Integration Service connection governance
- Tenant
- About the Tenant Context
- Searching for Resources in a Tenant
- Managing Robots
- Connecting Robots to Orchestrator
- Storing Robot Credentials in CyberArk
- Storing Unattended Robot Passwords in Azure Key Vault (read only)
- Storing Unattended Robot Credentials in HashiCorp Vault (read only)
- Storing Unattended Robot Credentials in AWS Secrets Manager (read only)
- Deleting Disconnected and Unresponsive Unattended Sessions
- Robot Authentication
- Robot Authentication With Client Credentials
- Configuring automation capabilities
- Solutions
- Audit
- Resource Catalog Service
- Automation Suite Robots
- Folders Context
- Automations
- Processes
- Jobs
- Apps
- Triggers
- Logs
- Monitoring
- Queues
- Assets
- Storage Buckets
- Indexes
- Orchestrator testing
- Integrations
- Troubleshooting
Orchestrator user guide
Test Executions
Overview
Test management capabilities have been moved to Test Manager. As a result, for any new tenant, you cannot execute test cases directly from the Orchestrator user interface anymore, except through Orchestrator APIs for Test automation, or using CI/CD integrations. To execute test cases, use Test Manager instead.
In this topic, you can learn how to manage your test executions. The test executions serve as an immutable record of the execution of any respective test set, at a specific point in time. By comparison, the test set is always subject to change, as it can target a different application version or its test case selection can be modified.
Prerequisites: To perform test executions, you need to have the Robot role assigned.
Executing test cases that contain Integration Service connections
For test cases and projects that contain test cases created using a specific Integration Service connection, you cannot assign a specific connection at runtime, other than the one used during the design phase in Studio,
For example, if you create a test case containing an Integration Service connection, then publish it to Orchestrator and execute it independently, there will be no option available to assign a specific connection at runtime.
If you publish a Studio project containing test cases using Integration Service connections to Orchestrator as a Process, the option to configure the connection at runtime will be available under the Package Requirements of the process.
When configuring the connection in the Package Requirements of the test cases will continue to use the design-time connection previously selected in Studio.
Re-execute Test Set
For any new tenant, you can re-execute test sets only using Orchestrator APIs for Test automation.
You can re-execute test sets after a previous run. The new test case result will override the previous one shown on the Test Executions page.
Be aware that you cannot re-execute test cases from a Test Set with enabled coverage calculation.
- Log in to Orchestrator.
- Open your folder.
- Navigate to Testing > Test Executions.
- Select a test run and go to More actions > Details.
- Select a test case and go to More actions > Re-execute.
- Configure the execution target:
- User: Select the user from the dropdown list that is assigned to that specific folder. The default is set to Any User.
- Machine: Select the machine resource from the dropdown list.
- (Optional) Define custom Arguments for your test cases to override default values at the test set level. To define arguments, you need to have published a package with arguments.
- Click Execute to confirm.
Known limitation
The results from re-executing a test set in Orchestrator are not synced to Test Manager. We recommend you to execute and re-execute tests directly from Test Manager.
Test Execution Coverage
For any new tenant, test execution coverage is not available. You can view the activity coverage of your tests in Test Manager. For more information, visit Viewing activity coverage.
You can view information on the activities that have been covered during a test execution. This option is also available in Studio, as Activity Coverage.
Activity coverage calculator is available when you select this option for existing or new test sets. For more information, see Create test set.
-
Open your folder in Orchestrator.
-
Navigate to Testing > Test Executions.
-
Select a test run and go to More actions > Details.
-
Go to the Coverage tab to view coverage details.
The coverage files are uploaded to the Attachments tab.
Figure 1. Coverage tab
Analyze Test Executions
You can analyze test executions to understand what happened in instances where the execution failed or got stuck.
You can view screenshots of failed test executions by navigating to Automations > Job > Job Details. To benefit from this feature, you need to enable execution media at the application level. For more information, see Execution Media Permissions.
Figure 2. Execution details
You can choose from the following actions to examine your test runs.
| Actions | Description | Procedure |
|---|---|---|
| Execution details | View Robot details, including execution attachments, data variation and screenshots. Make sure that you have Execution Media Permissions . |
|
| View assertions | View failed or passed assertions. |
|
| View logs | View and export a detailed log of execution events. |
|
Test Executions API
To query for item related activities, you can use the Background Tasks, internally available through API. For more information on Orchestrator Permissions, see. For more information on test automation API, see Test Automation API Permissions.