- Overview
- Requirements
- Installation
- Post-installation
- Accessing Automation Suite
- Managing the Certificates
- Resizing PVC
- Updating the SQL connection strings
- Cluster administration
- Managing products
- Managing the cluster in ArgoCD
- Setting up the external NFS server
- Automated: Enabling the Backup on the Cluster
- Automated: Disabling the Backup on the Cluster
- Automated, Online: Restoring the Cluster
- Automated, Offline: Restoring the Cluster
- Manual: Enabling the Backup on the Cluster
- Manual: Disabling the Backup on the Cluster
- Manual, Online: Restoring the Cluster
- Manual, Offline: Restoring the Cluster
- Additional configuration
- Migrating objectstore from persistent volume to raw disks
- Monitoring and alerting
- Migration and upgrade
- Migration options
- Step 1: Moving the Identity organization data from standalone to Automation Suite
- Step 2: Restoring the standalone product database
- Step 3: Backing up the platform database in Automation Suite
- Step 4: Merging organizations in Automation Suite
- Step 5: Updating the migrated product connection strings
- Step 6: Migrating standalone Insights
- Step 7: Deleting the default tenant
- B) Single tenant migration
- Product-specific configuration
- Best practices and maintenance
- Troubleshooting
- How to Troubleshoot Services During Installation
- How to Uninstall the Cluster
- How to clean up offline artifacts to improve disk space
- How to clear Redis data
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bucket
- How to disable streaming logs for AI Center
- How to debug failed Automation Suite installations
- How to delete images from the old installer after upgrade
- How to automatically clean up Longhorn snapshots
- How to disable TX checksum offloading
- How to address weak ciphers in TLS 1.2
- Unable to run an offline installation on RHEL 8.4 OS
- Error in Downloading the Bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- First installation fails during Longhorn setup
- SQL connection string validation error
- Prerequisite check for selinux iscsid module fails
- Azure disk not marked as SSD
- Failure After Certificate Update
- Automation Suite not working after OS upgrade
- Automation Suite Requires Backlog_wait_time to Be Set 1
- Volume unable to mount due to not being ready for workloads
- RKE2 fails during installation and upgrade
- Failure to upload or download data in objectstore
- PVC resize does not heal Ceph
- Failure to Resize Objectstore PVC
- Rook Ceph or Looker pod stuck in Init state
- StatefulSet volume attachment error
- Failure to create persistent volumes
- Storage reclamation patch
- Backup failed due to TooManySnapshots error
- All Longhorn replicas are faulted
- Setting a timeout interval for the management portals
- Update the underlying directory connections
- Cannot Log in After Migration
- Kinit: Cannot Find KDC for Realm <AD Domain> While Getting Initial Credentials
- Kinit: Keytab Contains No Suitable Keys for *** While Getting Initial Credentials
- GSSAPI Operation Failed With Error: An Invalid Status Code Was Supplied (Client's Credentials Have Been Revoked).
- Alarm Received for Failed Kerberos-tgt-update Job
- SSPI Provider: Server Not Found in Kerberos Database
- Login Failed for User <ADDOMAIN><aduser>. Reason: The Account Is Disabled.
- ArgoCD login failed
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis Probe Failure
- RKE2 Server Fails to Start
- Secret Not Found in UiPath Namespace
- After the Initial Install, ArgoCD App Went Into Progressing State
- MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion
- Unexpected Inconsistency; Run Fsck Manually
- Degraded MongoDB or Business Applications After Cluster Restore
- Missing Self-heal-operator and Sf-k8-utils Repo
- Unhealthy Services After Cluster Restore or Rollback
- RabbitMQ pod stuck in CrashLoopBackOff
- Prometheus in CrashloopBackoff state with out-of-memory (OOM) error
- Missing Ceph-rook metrics from monitoring dashboards
- Pods cannot communicate with FQDN in a proxy environment
- Using the Automation Suite Diagnostics Tool
- Using the Automation Suite support bundle
- Exploring Logs
Managing the Certificates
The installation bundle provides a cluster management tool that enables you to update certificates post-installation. To access the tool, navigate to the location of the installer bundle:
cd /opt/UiPathAutomationSuite/
cd /opt/UiPathAutomationSuite/
To generate the CSR and the private key, run the following command:
# copy the machine openssl configuration locally
cp /etc/pki/tls/openssl.cnf ./openssl.tmp.cnf
# Replace the [AUTOMATION_SUITE_FQDN] value. For example, "automationsuite.corp.com"
AS_FQDN=[AUTOMATION_SUITE_FQDN]
cat >> ./openssl.tmp.cnf <<EOF
[SAN]
subjectAltName=DNS:$AS_FQDN,DNS:alm.$AS_FQDN,DNS:monitoring.$AS_FQDN,DNS:registry.$AS_FQDN,DNS:objectstore.$AS_FQDN,DNS:insights.$AS_FQDN
EOF
# create the certificate request
openssl req -new -sha256 -newkey rsa:2048 -nodes -keyout server.key -subj "/C=xx/ST=xx/O=xx/OU=xx/CN=$AS_FQDN" -reqexts SAN -config openssl.tmp.cnf -out ${AS_FQDN}.csr
# copy the machine openssl configuration locally
cp /etc/pki/tls/openssl.cnf ./openssl.tmp.cnf
# Replace the [AUTOMATION_SUITE_FQDN] value. For example, "automationsuite.corp.com"
AS_FQDN=[AUTOMATION_SUITE_FQDN]
cat >> ./openssl.tmp.cnf <<EOF
[SAN]
subjectAltName=DNS:$AS_FQDN,DNS:alm.$AS_FQDN,DNS:monitoring.$AS_FQDN,DNS:registry.$AS_FQDN,DNS:objectstore.$AS_FQDN,DNS:insights.$AS_FQDN
EOF
# create the certificate request
openssl req -new -sha256 -newkey rsa:2048 -nodes -keyout server.key -subj "/C=xx/ST=xx/O=xx/OU=xx/CN=$AS_FQDN" -reqexts SAN -config openssl.tmp.cnf -out ${AS_FQDN}.csr
Your IT team uses the obtained values to generate a signed certificate. The generated private key remains local.
To view more information about server certificates, run the following command:
sudo ./configureUiPathAS.sh tls-cert --help
sudo ./configureUiPathAS.sh tls-cert --help
Output:
************************************************************************************
Manage cluster tls and server certificate
Usage:
configureUiPathAS.sh tls-cert [command]
configureUiPathAS.sh tls-cert [flags]
Available Commands:
update Update the tls / server certificate
get Get the tls / server certificate
Flags:
-h|--help Display help
************************************************************************************
************************************************************************************
Manage cluster tls and server certificate
Usage:
configureUiPathAS.sh tls-cert [command]
configureUiPathAS.sh tls-cert [flags]
Available Commands:
update Update the tls / server certificate
get Get the tls / server certificate
Flags:
-h|--help Display help
************************************************************************************
./configureUiPathAS.sh tls-cert
command.
Online installation: How to find the server certificate
istio-ingressgateway-certs
name in the istio-system
namespace.
See the certificate files in the following list:
- Server TLS certificate is stored as
tls.crt
- Server TLS private key as
tls.key
- CA bundle is stored as
ca.crt
You can verify the secrets using the following command :
kubectl -n istio-system get secrets istio-ingressgateway-certs -o yaml
kubectl -n istio-system get secrets istio-ingressgateway-certs -o yaml
Certificates are also stored in the UiPath namespace. This is applicable to every UiPath® product that needs certificate information to trust incoming calls. For details, see Understanding the container architecture related to certificates.
Offline installation: How to find the server certificate
rootCA.crt
and tls.crt
: ArgoCD and Docker Registry. Certificates are then stored in both Docker and ArgoCD namespaces.
You can verify the secrets using the following command :
# For docker registry
kubectl -n docker-registry get secrets docker-registry-tls -o yaml
# For Argocd
argocd login alm.cluster_fqnd --username argocd_username --password argocd_password
argocd cert list --cert-type https
# For docker registry
kubectl -n docker-registry get secrets docker-registry-tls -o yaml
# For Argocd
argocd login alm.cluster_fqnd --username argocd_username --password argocd_password
argocd cert list --cert-type https
How to update server certificates
To decrypt the certificate key, run the following command:
# replace /path/to/encrypted/cert/key to absolute file path of key
# replace /path/to/decrypt/cert/key to store decrypt key
# Once prompted, please entry the passphrase or password to decrypt the key
openssl rsa -in /path/to/encrypted/cert/key -out /path/to/decrypt/cert/key
# replace /path/to/encrypted/cert/key to absolute file path of key
# replace /path/to/decrypt/cert/key to store decrypt key
# Once prompted, please entry the passphrase or password to decrypt the key
openssl rsa -in /path/to/encrypted/cert/key -out /path/to/decrypt/cert/key
configureUiPathAS.sh
script to update the certificate as shown below. You need the path to each of the three certificate files. All the certificate
file should be in PEM
format.
- Certificate Authority Bundle - This bundle should contain only the chain certificates used to sign the TLS server certificate. The chain limit is up to nine certificates.
- Server Certificate - Public server certificate
-
Private key - Private key for server certificate
sudo ./configureUiPathAS.sh tls-cert update --ca-cert-file /path/to/cacert --tls-cert-file /path/to/tlscert --tls-key-file /path/to/tlskey
sudo ./configureUiPathAS.sh tls-cert update --ca-cert-file /path/to/cacert --tls-cert-file /path/to/tlscert --tls-key-file /path/to/tlskey
/directory/path/to/store/certificate
location.
To print out the certificate files, run the following command, specifying the directory where certificates are stored.
sudo ./configureUiPathAS.sh tls-cert get --outpath /directory/path/to/store/certificate
sudo ./configureUiPathAS.sh tls-cert get --outpath /directory/path/to/store/certificate
You are responsible for making sure the generated certificates are trusted.
To add the certificate to the host VM trust store, run the following commands on all the nodes in the cluster:
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
To view more information about additional CA certificates, run the following command:
./configureUiPathAS.sh additional-ca-certs --help
./configureUiPathAS.sh additional-ca-certs --help
Output:
***************************************************************************************
Manage additional CA certificates, this can be used to add sql server CA
Usage:
configureUiPathAS.sh additional-ca-certs [command]
configureUiPathAS.sh additional-ca-certs [flags]
Available Commands:
update Update the additional trusted CA certificates.
get Get the additional trusted CA certificates
Flags:
-h|--help Display help
***************************************************************************************
***************************************************************************************
Manage additional CA certificates, this can be used to add sql server CA
Usage:
configureUiPathAS.sh additional-ca-certs [command]
configureUiPathAS.sh additional-ca-certs [flags]
Available Commands:
update Update the additional trusted CA certificates.
get Get the additional trusted CA certificates
Flags:
-h|--help Display help
***************************************************************************************
./configureUiPathAS.sh
additional-ca-certs
command.
This command helps you update or replace the existing configured CA certificates.
./configureUiPathAS.sh additional-ca-certs update --ca-cert-file /path/to/ca/certs
./configureUiPathAS.sh additional-ca-certs update --ca-cert-file /path/to/ca/certs
--replace
at the end.
.pem
format and can have more than one certificate present in it.
To download the already configured CA certificates, run the following command:
./configureUiPathAS.sh additional-ca-certs get --outpath /path/to/download/certs
./configureUiPathAS.sh additional-ca-certs get --outpath /path/to/download/certs
You are responsible for making sure the generated certificates are trusted.
To add the certificate to the host VM trust store, run the following commands on all the nodes in the cluster:
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/
# 2. Update the trust store configuration
update-ca-trust
To view more information about identity token-signing certificates, run the following command:
sudo ./configureUiPathAS.sh identity token-cert --help
sudo ./configureUiPathAS.sh identity token-cert --help
Output:
************************************************************************************
Manage Identity token signing certificate
Usage:
configureUiPathAS.sh identity token-cert [command]
configureUiPathAS.sh identity token-cert [flags]
Available Commands:
update Update secondary certificate to signing
the authentication token
rotate Switch secondary certificate as a primary
token signing certificate
get Get token signing certificate
Flags:
-h|--help Display help
************************************************************************************
************************************************************************************
Manage Identity token signing certificate
Usage:
configureUiPathAS.sh identity token-cert [command]
configureUiPathAS.sh identity token-cert [flags]
Available Commands:
update Update secondary certificate to signing
the authentication token
rotate Switch secondary certificate as a primary
token signing certificate
get Get token signing certificate
Flags:
-h|--help Display help
************************************************************************************
./configureUiPathAS.sh identity
token-cert
command.
.pfx
extension.
--password
flag is between simple quotation marks. For example: --password 'CertificatePassword!@#'
.
sudo ./configureUiPathAS.sh identity token-cert update --cert-file-path /path/to/cert --password '<cert_pass>'
sudo ./configureUiPathAS.sh identity token-cert update --cert-file-path /path/to/cert --password '<cert_pass>'
This will rotate or switch the old certificate with the new one uploaded using update certificate.
sudo ./configureUiPathAS.sh identity token-cert rotate
sudo ./configureUiPathAS.sh identity token-cert rotate
There should be the lead time of about 24 - 48 hours between certificate update and rotate.
We need this lead time to keep supporting the authentication for cached token signed by old certificate.
If you rotate the certificate too soon before the expiry of cache token can result in downtime. And you may have to restart all your robots.
To perform an emergency certificate update, take the following steps:
By default, RKE2 certificates expire in 12 months. In the 90 days prior to their expiration date, certificates are rotated when you restart RKE2.
For more details, refer to RKE2 - Advanced Options - Certificate rotation.
if [[ -d "/var/lib/rancher/rke2/server/tls" ]]; then
dir="/var/lib/rancher/rke2/server/tls"
elif [[ -d "/var/lib/rancher/rke2/agent/tls" ]]; then
dir="/var/lib/rancher/rke2/agent/tls"
else
dir="/var/lib/rancher/rke2/agent/"
fi
# Loop through each .crt file in the directory
for file in "$dir"/*.crt; do
# Extract the expiry date from the certificate
expiry=$(openssl x509 -enddate -noout -in "$file" | cut -d= -f 2-)
# Get the file name without the path
filename=$(basename "$file")
# Print the filename and expiry date in a pretty format
printf "%-30s %s\n" "$filename:" "$expiry"
done
if [[ -d "/var/lib/rancher/rke2/server/tls" ]]; then
dir="/var/lib/rancher/rke2/server/tls"
elif [[ -d "/var/lib/rancher/rke2/agent/tls" ]]; then
dir="/var/lib/rancher/rke2/agent/tls"
else
dir="/var/lib/rancher/rke2/agent/"
fi
# Loop through each .crt file in the directory
for file in "$dir"/*.crt; do
# Extract the expiry date from the certificate
expiry=$(openssl x509 -enddate -noout -in "$file" | cut -d= -f 2-)
# Get the file name without the path
filename=$(basename "$file")
# Print the filename and expiry date in a pretty format
printf "%-30s %s\n" "$filename:" "$expiry"
done
The output you get should be similar to the one shown in the following image:
By default, RKE2 certificates expire in 12 months. In the 90 days prior to their expiration date, certificates are rotated when you restart RKE2. However, if the validity of the certificates exceeds the 90-day period, you must manually rotate the certificates by following the steps mentioned in RKE2 - Advanced Options - Certificate rotation.
If you want to customize the expiration period of RKE2 certificates to meet particular requirements, you can do so before restarting the RKE2 services for both server and agent nodes.
To rotate the RKE2 certificates, you must first execute a series of action on the server nodes, then proceed with some steps on the agent nodes.
- Stop the RKE2 server:
systemctl stop rke2-server.service
systemctl stop rke2-server.service - Clear any remaining RKE2 processes:
rke2-killall.sh
rke2-killall.sh - Delete the
dynamic-cert.json
file located at/var/lib/rancher/rke2/server/tls/
. -
To customize the expiration period of the RKE2 certificates, use the following command. Be aware that this example sets the validity period to 1000 days, but you can change this value based on your requirements.
SERVICE_NAME="rke2-server.service" conf_file_path="/etc/systemd/system/${SERVICE_NAME}.d/cert.conf" mkdir -p /etc/systemd/system/"${SERVICE_NAME}".d/ cat > "$conf_file_path" <<EOF [Service] Environment="CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=1000" EOF systemctl daemon-reload
SERVICE_NAME="rke2-server.service" conf_file_path="/etc/systemd/system/${SERVICE_NAME}.d/cert.conf" mkdir -p /etc/systemd/system/"${SERVICE_NAME}".d/ cat > "$conf_file_path" <<EOF [Service] Environment="CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=1000" EOF systemctl daemon-reload - Restart the RKE2 server:
systemctl start rke2-server.service
systemctl start rke2-server.serviceNote: If the cluster has more than one server node, steps 1-4 might not wholly execute as etcd may be unable to complete the leader election. Should this happen, repeat steps from 1-4 on other server nodes. - Delete the
rke2-serving
secret from thekube-system
namespace:kubectl delete secret -n kube-system rke2-serving
kubectl delete secret -n kube-system rke2-servingNote:In a multi-node deployment, you may not be able to run thekubectl
commands until you have completed the first four operations on the necessary number of server nodes. This is to fulfill the etcd quorum requirement. You can remove therke2-serving
secret immediately after the RKE2 server starts up.
kubectl get nodes
command. When your server nodes are ready, you can proceed to the agent nodes to regenerate the certificates.
Take the following steps on the agent nodes:
- Stop the RKE2 server:
systemctl stop rke2-agent.service
systemctl stop rke2-agent.service - Clear any remaining RKE2 processes:
rke2-killall.sh
rke2-killall.sh -
To customize the expiration period of the RKE2 certificates, use the following command. Be aware that this example sets the validity period to 1000 days, but you can change this value based on your requirements.
SERVICE_NAME="rke2-agent.service" conf_file_path="/etc/systemd/system/${SERVICE_NAME}.d/cert.conf" mkdir -p /etc/systemd/system/"${SERVICE_NAME}".d/ cat > "$conf_file_path" <<EOF [Service] Environment="CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=1000" EOF systemctl daemon-reload
SERVICE_NAME="rke2-agent.service" conf_file_path="/etc/systemd/system/${SERVICE_NAME}.d/cert.conf" mkdir -p /etc/systemd/system/"${SERVICE_NAME}".d/ cat > "$conf_file_path" <<EOF [Service] Environment="CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=1000" EOF systemctl daemon-reload - Restart the RKE2 server:
systemctl start rke2-agent.service
systemctl start rke2-agent.service
- Generating a Certificate Signing Request (CSR) and a Private Key
- Managing server certificates
- Updating the server certificate
- Accessing the TLS certificate
- Adding the CA certificate to the host trust store
- Managing additional CA certificates
- Updating the CA certificates
- Accessing the CA certificates
- Adding the CA certificate to the host trust store
- Managing identity token-signing certificates
- Updating the Certificate
- Rotating the Certificate
- Emergency Certificate Rotation
- Accessing the certificate
- Managing RKE2 certificates
- Checking the RKE2 certificate expiration date
- Rotating the RKE2 certificate