How to manually clean up logs

Cleaning up Ceph logs

Moving Ceph out of read-only mode

If you installed AI Center and use Ceph storage, take the following steps to move Ceph out of read-only mode:

Check if Ceph is at full capacity:
```
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph statuskubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
```
If Ceph is at full capacity, you must adjust the read-only threshold to start up the rgw gateways.

Scale down the ML skills:

kubectl -n uipath scale deployment <skill> --replicas=0kubectl -n uipath scale deployment <skill> --replicas=0

Put the cluster in write mode:

ceph osd set-full-ratio 0.95 <95 is the default value so you could increase to 96 and go up 
incrementall>ceph osd set-full-ratio 0.95 <95 is the default value so you could increase to 96 and go up 
incrementall>

Run Garbage Collection:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- radosgw-admin gc process --include-allkubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- radosgw-admin gc process --include-all

When storage goes down, run the following commands:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph dfkubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df

At this point, the storage should be lower, and the cluster should be healthy.

Disabling streaming logs

To ensure everything is in a good state, disable streaming logs by taking the following steps.

Disable aut-sync on UiPath and AI Center.
Disable streaming logs for AI Center.

If you have ML skills that have already been deployed, run the following commands:

kubectl set env deployment [REPLICASET_NAME] LOGS_STREAMING_ENABLED=falsekubectl set env deployment [REPLICASET_NAME] LOGS_STREAMING_ENABLED=false

Find out which buckets use the most space:

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin bucket stats | jq -r '["BucketName","NoOfObjects","SizeInKB"], ["--------------------","------","------"], (.[] | [.bucket, .usage."rgw.main"."num_objects", .usage."rgw.main".size_kb_actual]) | @tsv' | column -ts $'\t'kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin bucket stats | jq -r '["BucketName","NoOfObjects","SizeInKB"], ["--------------------","------","------"], (.[] | [.bucket, .usage."rgw.main"."num_objects", .usage."rgw.main".size_kb_actual]) | @tsv' | column -ts $'\t'

Install s3cmd to prepare for cleaning up the sf-logs:

pip3 install awscli s3cmd
export PATH=/usr/local/bin:$PATHpip3 install awscli s3cmd
export PATH=/usr/local/bin:$PATH

Clean up the sf-logs logs. For details, see How to clean up old logs stored in the sf-logs bundle.

Complete the cleanup operation:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- radosgw-admin gc process --include-allkubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- radosgw-admin gc process --include-all

If the previous steps do not solve the issue, clean up the AI Center data.

Check if the storage was reduced:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph dfkubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df

Once storage is no longer full, reduce the backfill setting:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd set-full-ratio 0.95kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph osd set-full-ratio 0.95

Check if the ML skills are affected by the multipart upload issue:

echo $(kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin bucket list --max-entries 10000000 --bucket train-data | jq '[.[] | select (.name | contains("_multipart")) | .meta.size] | add') | numfmt --to=iec-iecho $(kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin bucket list --max-entries 10000000 --bucket train-data | jq '[.[] | select (.name | contains("_multipart")) | .meta.size] | add') | numfmt --to=iec-i