UI Automation Activities

Last updated Mar 13, 2025

Enabling Computer Vision for mobile automation

Computer Vision enables you to use UiPath's neural network in order to identify UI elements and interact with them via Robots, simulating human interaction. In the context of Mobile Automation, the Computer Vision feature enhances the number of target elements that you can identify, allowing you to automate UI elements that are otherwise harder to identify and automate.

See the activities that can use Computer Vision in the table below:

Activity	Description
Get Text	Extracts the text from a specified UI element.
Set Text	Types the specified text into a target UI element.
Tap	Returns a boolean variable that states whether a specified UI element exists.
Swipe	Swipes the screen in a specified direction and for a specified amount of times.
Element Exists	Returns a boolean variable that states whether a specified UI element exists.

To start discovering mobile UI elements using Computer Vision, you need to set up your Computer Vision server, for either cloud or on-premises servers. You can set up the Computer Vision server in the Project Settings menu of the project that you are working in.

The Computer Vision configuration section is split into three other sub-sections:

Server - the URL for the type of Computer Vision server that you want to connect to: cloud or on-premises. The default value for the Run value and Debug value server fields is the cloud instance of Computer Vision: https://cv.uipath.com.
Api Key - the Computer Vision's API key, specific for your organization.
Use Local Server - use a local Computer Vision server.

Known limitation

Computer Vision for mobile automation doesn't support CJK (Chinese, Japanese, Korean) languages.

Cloud Computer Vision server

To connect to a cloud Computer Vision server, you can authenticate using an API key, or a User token.

API key

In the Project panel, go to Project Settings, Mobile Automation, and then focus on the Computer Vision section.
Under Server in the Run value and Debug value fields, input the URL of a Computer Vision cloud server. https://cv.uipath.com is the default value.
For changing the endpoint, visit Public endpoints.
Input your organization's Computer Vision API key. Visit API keys to learn how to get your Computer Vision API key.

Authenticating to the Computer Vision server

Visit Authentication to learn how to authenticate to the Computer Vision Server, so you can use the corresponding capabilities inside your project.

To learn how to configure your cloud Computer Vision server, visit the resources below:

Enabling local Computer Vision server

Follow these steps to use a Computer Vision local server:

Install the UiPath.ComputerVision.LocalServer package.
In the Project panel, go to Project Settings, Mobile Automation, and then focus on the Computer Vision section.
In the Server field:
1. If you are using a MobileAutomation.Activities version up to and including 24.12-preview: Input a Computer Vision server URL in the Run value and Debug value fields.
  Warning: Usually for a Computer Vision local server, this step is not required, but due to a current known issue, a Computer Vision server URL is needed for the local Computer Vision server for mobile automation to work as expected.
  
  The default value of the Computer Vision server is https://cv.uipath.com.
  
  For more information about the available Computer Vision public endpoints that you can use, visit Public endpoints.
2. If you are using MobileAutomation.Activities version 25.2.0-preview or higher: You can skip this step.
Under Use Local Server, set the Run value and Debug value to True.