- Overview
- Requirements
- Installation
- Q&A: Deployment templates
- Configuring the machines
- Configuring the external objectstore
- Configuring an external Docker registry
- Configuring the load balancer
- Configuring the DNS
- Configuring Microsoft SQL Server
- Configuring the certificates
- Online multi-node HA-ready production installation
- Offline multi-node HA-ready production installation
- Disaster recovery - Installing the secondary cluster
- Downloading the installation packages
- install-uipath.sh parameters
- Enabling Redis High Availability Add-On for the cluster
- Document Understanding configuration file
- Adding a dedicated agent node with GPU support
- Adding a dedicated agent Node for Task Mining
- Connecting Task Mining application
- Adding a Dedicated Agent Node for Automation Suite Robots
- Post-installation
- Cluster administration
- Monitoring and alerting
- Migration and upgrade
- Migration options
- Step 1: Moving the Identity organization data from standalone to Automation Suite
- Step 2: Restoring the standalone product database
- Step 3: Backing up the platform database in Automation Suite
- Step 4: Merging organizations in Automation Suite
- Step 5: Updating the migrated product connection strings
- Step 6: Migrating standalone Insights
- Step 7: Deleting the default tenant
- B) Single tenant migration
- Product-specific configuration
- Best practices and maintenance
- Troubleshooting
- How to troubleshoot services during installation
- How to uninstall the cluster
- How to clean up offline artifacts to improve disk space
- How to clear Redis data
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bucket
- How to disable streaming logs for AI Center
- How to debug failed Automation Suite installations
- How to delete images from the old installer after upgrade
- How to automatically clean up Longhorn snapshots
- How to disable TX checksum offloading
- How to manually set the ArgoCD log level to Info
- How to generate the encoded pull_secret_value for external registries
- How to address weak ciphers in TLS 1.2
- Unable to run an offline installation on RHEL 8.4 OS
- Error in downloading the bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- First installation fails during Longhorn setup
- SQL connection string validation error
- Prerequisite check for selinux iscsid module fails
- Azure disk not marked as SSD
- Failure after certificate update
- Antivirus causes installation issues
- Automation Suite not working after OS upgrade
- Automation Suite requires backlog_wait_time to be set to 0
- GPU node affected by resource unavailability
- Volume unable to mount due to not being ready for workloads
- Support bundle log collection failure
- Failure to upload or download data in objectstore
- PVC resize does not heal Ceph
- Failure to resize PVC
- Failure to resize objectstore PVC
- Rook Ceph or Looker pod stuck in Init state
- StatefulSet volume attachment error
- Failure to create persistent volumes
- Storage reclamation patch
- Backup failed due to TooManySnapshots error
- All Longhorn replicas are faulted
- Setting a timeout interval for the management portals
- Update the underlying directory connections
- Authentication not working after migration
- Kinit: Cannot find KDC for realm <AD Domain> while getting initial credentials
- Kinit: Keytab contains no suitable keys for *** while getting initial credentials
- GSSAPI operation failed due to invalid status code
- Alarm received for failed Kerberos-tgt-update job
- SSPI provider: Server not found in Kerberos database
- Login failed for AD user due to disabled account
- ArgoCD login failed
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis probe failure
- RKE2 server fails to start
- Secret not found in UiPath namespace
- ArgoCD goes into progressing state after first installation
- Issues accessing the ArgoCD read-only account
- MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion
- Unhealthy services after cluster restore or rollback
- Pods stuck in Init:0/X
- Prometheus in CrashloopBackoff state with out-of-memory (OOM) error
- Missing Ceph-rook metrics from monitoring dashboards
- Pods cannot communicate with FQDN in a proxy environment
- Running High Availability with Process Mining
- Process Mining ingestion failed when logged in using Kerberos
- Unable to connect to AutomationSuite_ProcessMining_Warehouse database using a pyodbc format connection string
- Airflow installation fails with sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''
- How to add an IP table rule to use SQL Server port 1433
- Using the Automation Suite Diagnostics Tool
- Using the Automation Suite support bundle
- Exploring Logs
Basic architecture considerations
As with any multi-site deployment, the primary architecture considerations for Automation Suite account for infrastructure, latency, data source, management, Recovery Time Objective, Recovery Point Objective, etc.
We recommend using the same hardware for both clusters. However, the Automation Suite cluster will likely work with similar hardware configurations with little difference. Heterogeneous hardware may increase complexity and slow down troubleshooting.
The two Automation Suite clusters are independent and do not share any configuration. Therefore, any management or maintenance activity must be done individually on these clusters. For instance, you must update the SQL connection strings on both clusters, configure certificates separately, etc. In addition, you must monitor the two clusters independently, upgrade them individually, etc.
The objectstore, combined with the SQL database, forms the state of an installed product on Automation Suite.
SQL Server configuration plays a vital role in a multi-site deployment. Though SQL Server is a component external to Automation Suite, a few additional steps are required to ensure true HA when working with Automation Suite.
MultiSubnetFailover=True property in the connection string when the SQL server/databases are distributed across multiple subnets.
For more details, see Always On availability groups and Prerequisites, Restrictions, and Recommendations for Always On availability groups.
The external objectstore is immune to possible corruption due to node failure. Data replication and disaster recovery can be carried out independently of Automation Suite. Like SQL Server, the external objectstore must be configured in a highly available Disaster Recovery setup. The primary objectstore instance is physically located in the primary data center, and at least one secondary instance is located in the secondary data center with data sync enabled. You can configure a load balancer on the objectstore to ensure both Automation Suite clusters refer to the same endpoints. This makes the deployment independent of how the objectstore is configured internally.
For AWS S3, the multi-region access point does not support all the s3 APIs required by all the products running in Automation Suite. For details on the list of support APIs, see Using Multi-Region Access Points with supported API operations.
You can create two buckets per product/suite in both regions and enable synchronization. The Automation Suite cluster running in the same region will refer to the buckets in the same region.
Your organization’s policy around RTO is vital in designing your multi-site Automation Suite cluster. To achieve the desired RTO, take the following aspects into consideration:
- Design of the Traffic Manager;
- Availability of the nodes in the secondary/passive cluster;
- Dynamic workload availability on the secondary cluster; for example, MLSkill;
- Configuration Management.
You can reduce the recovery time by configuring the Traffic Manager to always route the traffic to the primary cluster when available. Redirection to the secondary cluster must be done only when the primary cluster is down. This ensures the traffic switch is automatic and reduces the time for a manual switchover. You can use the health enpoints of both clusters to achieve this.
If all the nodes in the secondary cluster are running, you can save time by turning on the nodes and waiting for the cluster to be active. However, this can increase your infrastructure's cost by almost twice.
A few products, such as AI Center, deploy the ML Skills dynamically at the runtime. The deployment of the skills in another cluster is always asynchronous. This cannot guarantee their availability. To ensure that your automation solution returns online within the desired time, you can periodically sync the skills in another cluster.
Since multi-site Automation Suite deployments consist of two distinct clusters, any operation performed on any cluster must be performed on the other cluster in time to reduce the drift. This ensures that both clusters possess similar configurations, and no additional effort is required during the recovery phase.
Your organization’s policy around Recovery Point Objective (RPO) is vital in designing your multi-site Automation Suite cluster. To achieve the desired RPO, you must take the following aspects into account:
- Data synchronization;
- Scheduled backup.
When written to the primary data source, data must also be synced to the secondary cluster. However, there is a risk of data loss when the data center is down, and data is not synced. Exemplary network configurations, such as high bandwidth and low latency between the two data centers, can speed up synchronization.
Not all disaster recovery provides complete immunity to data loss. However, you can deploy a regular and periodic backup strategy to minimize the impact of the disaster on data recovery. For details, see Backing up and restoring the cluster.
