Overview of AI Computer Vision and its machine-learning approach to UI element identification in Automation Cloud.
AI Computer Vision is a machine-learning based method used to visually identify all the UI elements on a computer screen and interact with them via UiPath Robots, simulating human interaction. It doesn't require or use the underlying properties of applications, but only the aspect and relationship of various screen elements.
Rather than relying on selectors, AI Computer Vision uses AI (Object Detection, OCR, fuzzy text-matching, image-matching for icons) and an anchoring system to tie it all together.
To visually locate elements on the screen, AI Computer Vision performs an element detection (on the machine-learning server) and a text (OCR) detection, and combines these two into a full understanding of the UI. The relationship between elements detected with these two methods is then encoded into a multi-anchor descriptor, which uniquely identifies the targeted element.
AI Computer Vision has two parts:
- A set of activities, which are part of the UI Automation activity package.
- A server hosting the AI model that performs the actual analysis of the UI you're automating. The server can be cloud, on-premises, or local.
The following table describes the three server types and what to consider for each one.
| Server type | How it works | What to consider |
|---|---|---|
| Cloud (default and recommended) | The UiPath cloud server is used by default for all AI Computer Vision and UI Automation activities. | You can use AI Computer Vision cloud regardless of your deployment type. Whether Orchestrator is on-premises or in the cloud, Computer Vision cloud runs with no special configuration required. |
| On-premises | You host and manage your own AI Computer Vision server, and use it to run the AI Computer Vision activities. | You need your own hardware infrastructure (GPUs) or cloud environment, which you deploy, update, and maintain locally. Compared to the UiPath cloud server, you might also run into backwards compatibility issues when upgrading the AI model. |
| Local | The local server runs on local CPU and is the most portable version. | It is slower and has a slightly lower detection accuracy. |
Key benefits
Here are some features of AI Computer Vision you can benefit from:
- Automation beyond selectors - Enable robots to recognize and interact with more on-screen fields and components - even Flash, Silverlight, PDFs, and images.
- Reliable on VDIs and desktops - Relieves issues with failure-prone image automation techniques and with selector-based targeting on desktops. Start by creating automations within Citrix, VMware or Microsoft’s Remote Desktop.
- Broad range of interface types - Includes VDI environments (Citrix, VMWare, Microsoft RDP, VNC, and others) for desktop and web applications. Save your time by getting UI elements identified and added to object repository for you.
- Intelligent, intuitive capabilities - Provides details, validation, and notifications about on-screen selections via an on-screen wizard. Uses the recorder to easily generate full vision-based automations.
- Run-time auto-scroll support - Easily automate scrollable content in webpages or apps using AI Computer Vision activities.
- Cross-platform capabilities - Automate for Windows, Linux, Android and other operating systems through remote desktops.
- Automation between VDI & non-VDI - Simplifies VDI-to-desktop automation by reducing necessary modifications.
- Multiple deployment options - Deploys via SaaS; available on-premises for Linux and Windows, or right from your desktop.
- Dynamic UI elements - Enables automations that include tables, drop-down lists, and checkbox elements. This increases the resilience of your automations, enabling them to adapt to small changes to the UI and interact with these dynamic elements.
- Available in UI Automation as part of Unified Target - Reduces the complexity of building UI-based automations when you need both selectors and AI Computer Vision descriptors.
Deployment options
For a parallel comparison of our existing AI Computer Vision deployment options, check the AI Computer Vision differences section in the Overview guide.