RabbitMQ pod stuck in CrashLoopBackOff

Description

This issue causes the RabbitMQ pod to be stuck in CrashLoopBackOff, with the log of the failing pod showing wal_checksum_validation_failure as a reason.

To get a list of all pods, run the following command:

kubectl -n rabbitmq get podskubectl -n rabbitmq get pods

To get the logs of a pod, run the following command:

kubectl -n rabbitmq logs <CrashLoopBackOff-Pod-Name>kubectl -n rabbitmq logs <CrashLoopBackOff-Pod-Name>

Solution

To fix the issue, take the following steps:

Find the number of RabbitMQ replicas:

rabbitmqReplicas=$(kubectl -n rabbitmq get rabbitmqcluster rabbitmq -o json | jq -r '.spec.replicas')rabbitmqReplicas=$(kubectl -n rabbitmq get rabbitmqcluster rabbitmq -o json | jq -r '.spec.replicas')

Scale down the RabbitMQ replicas:

kubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": 0}}" --type=mergekubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": 0}}" --type=merge

kubectl -n rabbitmq scale sts rabbitmq-server --replicas=0kubectl -n rabbitmq scale sts rabbitmq-server --replicas=0

Wait until all RabbitMQ pods are terminated:

kubectl -n rabbitmq get podkubectl -n rabbitmq get pod

Find and delete the PVC of the RabbitMQ pod that is stuck in the CrashLoopBackOff state:

kubectl -n rabbitmq get pvckubectl -n rabbitmq get pvc

kubectl -n rabbitmq delete pvc <crashloopbackupoff_pod_pvc_name>kubectl -n rabbitmq delete pvc <crashloopbackupoff_pod_pvc_name>

Scale up the RabbitMQ replicas:

kubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": $rabbitmqReplicas}}" --type=mergekubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": $rabbitmqReplicas}}" --type=merge

Check if all RabbitMQ pods are healthy:

kubectl -n rabbitmq get podkubectl -n rabbitmq get pod

On this page

Description
Solution

Was this page helpful?

PREVIOUSUnhealthy Services After Cluster Restore or Rollback

NEXTPrometheus in CrashloopBackoff state with out-of-memory (OOM) error

Support and Services

Get The Help You Need

UiPath Academy

Learning RPA - Automation Courses

UiPath Forum

UiPath Community Forum

Trust and Security

Cookies Policy