- Overview
- UI Automation
- Applications and technologies automated with UI Automation
- Project compatibility
- UI-ANA-016 - Pull Open Browser URL
- UI-ANA-017 - ContinueOnError True
- UI-ANA-018 - List OCR/Image Activities
- UI-DBP-006 - Container Usage
- UI-DBP-013 - Excel Automation Misuse
- UI-DBP-030 - Forbidden Variables Usage In Selectors
- UI-PRR-001 - Simulate Click
- UI-PRR-002 - Simulate Type
- UI-PRR-003 - Open Application Misuse
- UI-PRR-004 - Hardcoded Delays
- UI-REL-001 - Large Idx in Selectors
- UI-SEC-004 - Selector Email Data
- UI-SEC-010 - App/Url Restrictions
- UI-USG-011 - Non Allowed Attributes
- UX-SEC-010 - App/Url Restrictions
- UX-DBP-029 - Insecure Password Use
- UI-PST-001 - Audit Log Level in Project Settings
- UiPath Browser Migration Tool
- Clipping region
- Computer Vision Recorder
- Activate
- Anchor Base
- Attach Browser
- Attach Window
- Block User Input
- Callout
- Check
- Click
- Click Image
- Click Image Trigger
- Click OCR Text
- Click Text
- Click Trigger
- Close Application
- Close Tab
- Close Window
- Context Aware Anchor
- Copy Selected Text
- Element Attribute Change Trigger
- Element Exists
- Element Scope
- Element State Change Trigger
- Export UI Tree
- Extract Structured Data
- Find Children
- Find Element
- Find Image
- Find Image Matches
- Find OCR Text Position
- Find Relative Element
- Find Text Position
- Get Active Window
- Get Ancestor
- Get Attribute
- Get Event Info
- Get From Clipboard
- Get Full Text
- Get OCR Text
- Get Password
- Get Position
- Get Source Element
- Get Text
- Get Visible Text
- Go Back
- Go Forward
- Go Home
- Google Cloud Vision OCR
- Hide Window
- Highlight
- Hotkey Trigger
- Hover
- Hover Image
- Hover OCR Text
- Hover Text
- Image Exists
- Indicate On Screen
- Inject .NET Code
- Inject Js Script
- Invoke ActiveX Method
- Key Press Trigger
- Load Image
- Maximize Window
- Microsoft Azure Computer Vision OCR
- Microsoft OCR
- Microsoft Project Oxford Online OCR
- Minimize Window
- Monitor Events
- Mouse Trigger
- Move Window
- Navigate To
- OCR Text Exists
- On Element Appear
- On Element Vanish
- On Image Appear
- On Image Vanish
- Open Application
- Open Browser
- Refresh Browser
- Replay User Event
- Restore Window
- Save Image
- Select Item
- Select Multiple Items
- Send Hotkey
- Set Clipping Region
- Set Focus
- Set Text
- Set To Clipboard
- Set Web Attribute
- Show Window
- Start Process
- System Trigger
- Take Screenshot
- Tesseract OCR
- Text Exists
- Tooltip
- Type Into
- Type Secure Text
- Use Foreground
- Wait Attribute
- Wait Element Vanish
- Wait Image Vanish
- Application Event Trigger
- Check/Uncheck
- Check App State
- Check Element
- Click
- Click Event Trigger
- Drag and Drop
- Extract Table Data
- For Each UI Element
- Get Attribute
- Get Browser Data
- Get Text
- Get URL
- Go to URL
- Highlight
- Hover
- Inject Js Script
- Keyboard Shortcuts
- Keypress Event Trigger
- Mouse Scroll
- Navigate Browser
- Select Item
- Set Browser Data
- Set Runtime Browser
- Set Text
- Take Screenshot
- Type Into
- Use Application/Browser
- Perform browser search and retrieve results using UI Automation APIs
- Web Browsing
- Find Images
- Click Images
- Trigger and Monitor Events
- Create and Override Files
- HTML Pages: Extract and Manipulate Information
- Window Manipulation
- Automated List Selection
- Find and Manipulate Window Elements
- Manage Text Automation
- Load and Process Images
- Manage Mouse Activated Actions
- Automate Application Runtime
- Automated Run of a Local Application
- Browser Navigation
- Web Automation
- Trigger Scope Example
- Enable UI Automation support in DevExpress
- Computer Vision Local Server
- Mobile Automation
- Release notes
- Project compatibility
- Get Log Types
- Get Logs
- Get Page Source
- Get Device Orientation
- Get Session Identifier
- Install App
- Manage Current App
- Manage Other App
- Open DeepLink
- Open URL
- Mobile Device Connection
- Directional Swipe
- Draw Pattern
- Positional Swipe
- Press Hardware Button
- Set Device Orientation
- Take Screenshot
- Take Screenshot Part
- Element Exists
- Execute Command
- Get Attribute
- Get Selected Item
- Get Text
- Set Selected Item
- Set Text
- Swipe
- Tap
- Type Text
- Terminal
Computer Vision activities
This section includes Computer Vision related activities found in the UiPath.UIAutomation.Activities package.
- CV Check
- CV Click
- CV Dropdown Select
- CV Element Exists
- CV Extract Table
- CV Get Text
- CV Highlight
- CV Hover
- CV Refresh
- CV Screen Scope
- CV Type Into
The Computer Vision activities contain refactored fundamental UI Automation activities such as Click, Type Into, or Get Text. The main difference between the Computer Vision activities and their classic counterparts is their usage of the Computer Vision neural network developed in-house by our Machine Learning department. The neural network is able to identify UI elements such as buttons, text input fields, or check boxes without the use of selectors.
Created mainly for automation in virtual desktop environments, such as Citrix machines, these activities bypass the issue of non-existent or unreliable selectors, as they send images of the window you are automating to the neural network, where it is analyzed and all UI elements are identified and labeled according to what they are. Smart anchors are used to pinpoint the exact location of the UI element you are interacting with, ensuring the action you intend to perform is successful.
All the Computer Vision activities function only when inside a CV Screen Scope activity, which establishes the actual connection to the neural network server, thus enabling you to analyze the UI of the applications you want to automate.
Any workflow using the Computer Vision activities must begin with dragging a CV Screen Scope activity to the designer panel.
Once this is done, the Indicate on screen button in the body of the scope activity can be used to select the area of the screen that you want to work in.
Once a CV Screen Scope activity is properly configured, you can start using all of the other activities in the pack to build your automation.
The activities that perform actions on UI elements can be configured at design time by using the Indicate on screen button found in the activity card. The activities that have this feature are:
- CV Check
- CV Click
- CV Dropdown Select
- CV Element Exists
- CV Extract Table
- CV Get Text
- CV Highlight
- CV Hover
- CV Screen Scope
- CV Type Into
Clicking the Indicate on screen (hotkey: I) button opens the helper wizard.
The Indicate field specifies what you are indicating at the moment. When the helper is opened for the first time, the Target needs to be indicated. For each possible target, the wizard automatically selects an anchor, if one is available.
The Show Elements (hotkey: s) button in the wizard highlights all UI elements that have been identified by the Computer Vision analysis, making it easier for you to choose what to interact with.
The Screen Refresh (hotkey: F5) button can be used at design time, in case something changes in the target app, enabling you to send a new picture to the Computer Vision server to be analyzed again.
The Delayed Screen Refresh (hotkey: F2) button performs a refresh of the target app after waiting 3 seconds. The run-time auto-scroll direction is also captured (ScrollDirection property).
The activities that have the Indicate wizard also include a Relative To feature, where the target is a stable element and you drag a distance relative to it. The main target, the stable one, acts as a proxy target for the second target, the relative one.
This is useful when you want to select an area that is not correspondent to the UI elements that the Computer Vision engine has identified, or when you want to select a general area in the UI without specifying a certain UI element.
The relative target can be either a single point or an area selection in the application you are automating.
Single point selection is normally placed in a spot where Computer Vision detection is not accurate and might not find certain elements. The activities that use single point selection are CV Click, CV Hover, and CV Type Into.
Area selection is used by CV Get Text and is automatically activated for this activity.
As a general rule, any anchor that you add must be found at runtime, meaning that if at least one anchor is not found, the entire selection fails to be found. So, anchors need to be stable elements that you know for certain Computer Vision can detect every time and that do not change drastically in the UI at runtime.
Let's say you want to click the "X" button in the image below. You can notice that the target is on the "X" button, but the click happens in the green circle; this is because you have chosen to click relative to the "X" button. So, unless Computer Vision cannot detect a certain UI element, there is no need to use the Relative To feature.
To select the "X" button and not an area relative to it, make sure the "X" button has a green outline on it.
If the "X" button is not stable, you can click relative to a stable element.
The Computer Vision activities also offer support for indicating tables. Targeting in tables can be done by selecting a cell you want to interact with, which prompts the neural network to automatically identify the column and the row that define the position of that cell, displaying them in a grid.
By default, the names of the column and row are used in the descriptor to pinpoint the location of the cell. Clicking the column and row indexes automatically adds them to your descriptor. This might be useful in situations where column and row names are changed, but you want to extract the same position of a cell.
After successfully indicating the Target, the wizard closes and the activity is configured with the target you selected.
