activities
latest
false
UiPath logo, featuring letters U and I in white

UI Automation Activities

Last updated Mar 13, 2025

Device Interaction

Device interaction consists of the actions that you take on a mobile device, within the Mobile Device Manager (MDM), that you can record and then import into your test case.

Device Interaction Conditions

When an action is executed, device interaction is locked until its completion.

Starting an Application

To interact with your device you must first start an application. In MDM, you have three options to choose from, as listed in the following table.

Method

Procedure

Start an application from a device
  1. Go to the Devices tab and select a device.
  2. Click Start an application on this device.
  3. Choose an application from the dropdown list and click Connect.

Start an application from a configured app
  1. Go to the Applications tab and select an application.
  2. Click Start application.
  3. Choose a device from the dropdown and click Connect.

Start an application using Quick Actions within the Welcome tab
  1. Go to the Welcome tab and select Start an application.
  2. Choose a device and an application from the dropdown.

  3. Click Connect.

    For more information about Quick Actions, refer to Quick Actions.

Interacting with the device

You can interact with your device in MDM as you would with an actual physical device, using your mouse or the action bar. The screen and the actions are locked when you perform an action.



On-Screen Interaction

The following table lists the on-screen actions that you can take to interact with your device.

Action

Description

Procedure

Tap

Access an on-screen element.

Click an element on the device screen.

Double Tap

Perform double tap action on-screen.

Double-click an element on the device screen.

Long Tap

Long press.

Click and hold on an element.

Pinch (Zoom In/Out)

Zoom in and out. This action is not recorded.

Right-click and hold, and then drag to the bottom of the screen to zoom in, or towards the upper part to zoom out.

Swipe

Swipe on-screen.

Click and drag towards any part of the screen.

Keyboard Input

Type in your text and send it to the mobile. For example, you can use the search bar to quickly access a website.
  1. Double-click on a text area to open the input window.
  2. Type in your text.
  3. Select Press Enter key after sending text to type in and enter your input directly.
  4. Click Send Text.

Action Bar

The following table lists the actions that you can take to interact with your device. The action bar is right next to the device.

Action

Description

Procedure

iOS Home

Return to the home screen.

Click the iOS Home button.

Android Back

Get back to the previous screen/page.

Click the Android Back button.

Android Home

Return to the home screen.

Click the Android Home button.

Android Switch Apps

Switch between apps.

Click the Android Switch Apps button, and choose your next action.

Change orientation to landscape/portrait

Toggle between portrait (vertical) and landscape (horizontal) mode for your device. Screen orientation is not supported by all applications.

Click the Change Orientation toggle to switch between modes.

Refresh

Refresh the device connection.

Click the Refresh button.

Alternatively, click the Reconnect button when the connection has been lost.

Save Screenshot

Save the current screen.

Click Save Screenshot and choose a location to save the file.

Send Text

Type in your text and send it to the mobile. For example, you can use the search bar to quickly access a website.
  1. Select a focusable element and then click Send Text.
  2. Type in your text.
  3. Select Press Enter key after sending to type in and enter your input directly.
  4. Deselect Clear the existing text to paste your text next to the existing text without clearing it.
  5. Click Send Text to confirm.

Draw Pattern

Draws whatever pattern on the device screen. Several points from the finger path are recorded and displayed on a X-Y axis.
  1. Select Draw Patterndocs image and draw whatever pattern you want inside a UI element.
  2. To import the action in Studio, click Import recorded actions. The pattern that you drew in MDM shows up as a Draw pattern activity, that prints out the points recorded from the finger path.
Enable CV recordingEnables recording for automating UI elements identified by the Computer Vision feature. This feature uses a UiPath® neural network that is able to identify UI elements without the use of selectors. This method, in turn, manages to identify elements that are otherwise impossible to automate.
  1. Select Enable CV recording and select whatever UI element you want to automate.
  2. To import the action in Studio, click Import recorded actions. The UI element that you recorded in MDM shows up as a corresponding activity in your workflow.

Selection by identified elements

Selects UI elements that are identified by MDM.

Click the Selection by identified elements to choose to select UI elements that are identified by MDM.

Selection by absolute position

Uses fixed element coordinates for selecting a UI element. The coordinates are recorded as a Positional Swipe activity.

Click Selection by absolute position and choose the default coordinates for selecting a UI element.

Record element action

Records a manually configured action completed on certain UI elements. It is recommended for recording more granular actions, such as adding an offset to an on-screen tap.

  1. Click Record element action docs image.
  2. Select a UI element on the device’s screen.
  3. Click Single tap docs image, and choose a type of on-screen interaction from the drop-down list. For example, choose Single tap.
  4. Click Configure Action docs image to further configure the action, for more granularity.
  5. Click Execute Action docs image to run the action that you’ve just recorded, in the Recorded Actions panel.

Toggle WebView selectors as native selectors

Tries to convert future selectors of type WebView, to selectors of type native, in accordance with the operating system used. This feature makes automating web apps easier.

  1. Click Toggle WebView selectors as native selectors docs image.
  2. Start recording your actions.

Install App

Install an application on your device.
  1. Click the horizontal ellipsis to open Advanced Settings.
  2. Click the Install App button and enter your app full location.
  3. Click Install to confirm.

Manage App

Manage the current application.
  1. Click the horizontal ellipsis to open Advanced Settings.
  2. Click the Manage App and select one of the following actions from the dropdown:
    1. Launch: Launch the default app page.
    2. Reset: Restart the app.
  3. Click Execute to confirm.

Manage Other App

Manage other applications on your device. You can use this when you test integrations between apps.
  1. Click the horizontal ellipsis to open Advanced Settings.
  2. Click the Manage Other Apps and enter your app's name.
  3. Choose on of the following commands from the dropdown:
    1. Remove: Uninstall the app.
    2. Activate: Open the app.
    3. Terminate: Close the app.
  4. Click Execute to confirm.

Open URL

Open URL in browser.
  1. Open a browser on your device.
  2. Click the horizontal ellipsis to open Advanced Settings.
  3. Click the Open URL and enter a full URL. For example: https://www.google.com/.
  4. Click Open to confirm.

Open Deep Link

Launch an app that opens a specific page.
  1. Click the horizontal ellipsis to open Advanced Settings.
  2. Click the Open Deep Link and configure the following:
    1. App: App path
    2. Deep Link: Specify the page that you want to open.
  3. Click Open to confirm.

Save Source

Save the XML source file of the device.
  1. Click the horizontal ellipsis to open Advanced Settings.
  2. Click the Save Source and select a location to save the file.

Set Geo Location

Set geographic coordinates.
  1. Click the horizontal ellipsis to open Advanced Settings.
  2. Click the Set Geo Location and use the Latitude, Longitude, Altitude finder on map to get coordinates. Alternatively, you can search for your location.
  3. Click Set to confirm.

Record element action

The Record element action feature can be found inside the Action bar. This allows you to record highly granular actions, that can't be captured through a regular on-screen interaction. The Record element action feature works by recording an action that is completed on one, or more UI elements, and configuring it. The configuration of the action depends on the type of on-screen interaction that you choose. For example, if you choose Single Tap as your on-screen interaction, you can configurate into Double Tap or Long Tap.



The Record element action feature is recommended for recording more granular actions, such as adding an offset to an on-screen tap.

Using Record Element Action

To use the Record element action feature:

  1. Click Record element action docs image.
  2. Select a UI element on the device’s screen.
  3. Click Single tap docs image, and choose a type of on-screen interaction from the drop-down list. For example, choose Single tap.
  4. Click Configure Action docs image to further configure the action, for more granularity.
  5. Click Execute Action docs image to run the action that you’ve just recorded, in the Recorded Actions panel.

WebView selectors as native selectors

The WebView selectors as native selectors feature can be found inside the Action bar. When toggled, it tries to convert all future selectors from WebView to Native. This allows a better, faster, and more familiar interaction with UI elements inside web views.

Note: Keep in mind that there are scenarios where this selection strategy may not work as expected, especially for complex web pages or web pages with elements that update frequently.

Using WebView Selectors as Native Selectors

To use WebView selectors as Native selectors:

  1. Click Toggle WebView selectors as native selectors docs image.
  2. Start recording your actions.

Recorded actions

You can interact with your device through the action bar or by indicating your actions on-screen to record the steps. For example, you can record and manage your actions to prepare mobile automation workflows in MDM. For more information, see Create Workflow Using Recorded Actions.



The following table lists the actions that you can take using Recorded Actions.

Action

Description

Procedure

Execute

Execute the action.

Click the Play button on an action to execute it.

Copy

Copy the action to your clipboard. You can use the Import Recorded Actions in Studio to paste it to your workflow.

Click the Copy button to copy the action to your clipboard.

Delete

Delete the action.

Click Delete to remove the action from the list. Alternatively, you can click Clear All to remove all actions.

Reorder

Reorder the list of actions in alternate order.

Drag and drop an action to different position within the list.

Wait for page update

Wait for the page source to retrieve a screenshot of each action.

Toggle the Wait for new screenshot after each action.

Was this page helpful?

Support and Services
Get The Help You Need
UiPath Academy
Learning RPA - Automation Courses
UiPath Forum
UiPath Community Forum
Uipath Logo White
Trust and Security
Terms of Use
Privacy Policy
Cookies Policy
© 2005-2025 UiPath. All rights reserved.