UI Automation Activities

Image recognition activities can also simulate human behaviour, using images as means of identifying UI elements. These activities enable you to make decisions based on whether or not a given image is displayed, or they can be used to perform certain actions in a loop, by using them as Conditions in the Retry Scope activity. They can also scan the screen of the machine for UI elements which appear at random positions and return UiElement variables that have the clipping region set to the found element. They also enable the upload and download of images. Image recognition activities have an Accuracy parameter, which states whether the images must match 100% or less to register as found which can compensate for possible changes. This feature is useful if the graphical elements you are searching for may be slightly different.

Click Image and Hover Image are activities used to identify UI elements based on their image. After an image is specified, the activity scans the screen for a given element and either clicks or hovers it. These activities are fast and reliable, but sensitive to graphical variations, as they can fail if colors or background details change. These activities are useful in automating processes that imply simulating human behaviour, using UI elements such as buttons or check boxes. These activities are also important when automating processes in virtual machine environments, such as Citrix, as they make interaction with UI elements possible. As input, these activities receive an image variable which contains the image to be clicked, and a Target, which can be either a string variable, a Region variable, a UIElement variable or a selector, which indicate the coordinates where the action must be performed. The target can also be automatically generated by using the Indicate on Screen functionality, which tries to identify UI elements in the indicated region, and generates selectors for them. If this does not work for you, then manual intervention might be required.

Find Image is an activity that waits for a certain UI element to appear. To do this, an image of the UI element is provided by the user as a model of the image to be searched. Once the element appears, the activity returns a UiElement variable with the clipping region set to the found image. This activity can be a useful tool in identifying UI elements in virtual machines and performing different actions on them. Find Image also enables you to make decisions based on whether or not a given image is displayed, or it can be used to perform certain actions in a loop, by using it as a Condition in the Retry Scope activity. As input, this activity receives an image variable which contains the image to be searched for, and a Target, which can be either a Region variable, a UiElement variable or a selector, that helps you identify what you want to automate and where the actions must be performed. The target can also be automatically generated by using the Indicate on Screen functionality, which tries to identify UI elements in the indicated region, and generates selectors for them. If this does not work for you, then manual intervention might be required.

Image Exists is an activity that is used to verify if a certain image exists on the screen. It returns a boolean variable which states whether the image was found or not. This activity can be useful as it enables you to make decisions based on whether or not a given image is displayed, or it can be used to perform certain actions in a loop, by using it as a Condition in the Retry Scope activity. As input, this activity receives an image variable which contains the image to be searched for, and a Target, which can be either a Region variable, a UiElement variable or a selector, that helps you identify what you want to automate and where the actions must be performed. The target can also be automatically generated by using the Indicate on Screen functionality, which tries to identify UI elements in the indicated region, and generates selectors for them. If this does not work for you, then manual intervention might be required.

On Image Appear waits for an image to appear on screen for a set amount of time. This activity is a container, which means that multiple actions can be inserted in it and performed on the found image. This is a very useful activity in virtual machine environments, as it can monitor when a UI element appears and then perform a suite of actions. On Image Appear can also be used as a trigger for other activities. As input, this activity receives an image variable which contains the image to be searched for, and a Target, which can be either a Region variable, a UiElement variable or a selector, that helps you identify what you want to automate and where the actions must be performed. The target can also be automatically generated by using the Indicate on Screen functionality, which tries to identify UI elements in the indicated region, and generates selectors for them. If this does not work for you, then manual intervention might be required.

On Image Vanish waits for an image to vanish from the screen for a set amount of time. This activity is a container, which means multiple actions can be inserted in it and performed after the image disappears. This is a very useful activity in virtual machine environments, as it can monitor when a UI element disappears and then perform a suite of actions. On Image Vanish can also be used as a trigger for other activities. As input, this activity receives an image variable which contains the image to be searched for, and a Target, which can be either a Region variable, a UiElement variable or a selector, that helps you identify what you want to automate and where the actions must be performed. The target can also be automatically generated by using the Indicate on Screen functionality, which tries to identify UI elements in the indicated region, and generates selectors for them. If this does not work for you, then manual intervention might be required.