Recovering Loki in standalone mode if PVC is full

In standalone mode, Loki stores data on a single PVC. If the PVC is full, the Loki pod may go CrashLoopBackOff, and logs will trow errors like no space left on device.

Do not remove the PVC as your first recovery step. In standalone, this will result in the loss of local log storage.

Checking statusChecking status

1. Find a namespace with Loki. For the logging component, use stackland-logging, for bootstrap installation, you may use loki.

   kubectl get pods,pvc -A | grep -E 'loki|stackland-logging'
   

2. Check the pod and PVC:

   NS=stackland-logging
   kubectl -n "$NS" get pods,pvc
   kubectl -n "$NS" describe pod -l app.kubernetes.io/name=loki
   kubectl -n "$NS" describe pvc
   

3. Check free space on the PVC based on kubelet metrics or inside the pod if the pod is still starting:

   POD=$(kubectl -n "$NS" get pod \
     -l app.kubernetes.io/name=loki,app.kubernetes.io/component=single-binary \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl -n "$NS" exec -it "$POD" -- df -h /var/loki
   

Preferred method: expand PVCPreferred method: expand PVC

1. Find out the PVC name and StorageClass:

   NS=stackland-logging
   kubectl -n "$NS" get pvc
   PVC=<PVC_NAME>
   SC=$(kubectl -n "$NS" get pvc "$PVC" -o jsonpath='{.spec.storageClassName}')
   kubectl get storageclass "$SC" -o jsonpath='{.allowVolumeExpansion}{"\n"}'
   

2. If allowVolumeExpansion=true, increase the PVC. The new value must be greater than the current one:

   kubectl -n "$NS" patch pvc "$PVC" --type merge \
     -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
   

3. Wait for the resize to complete:

   kubectl -n "$NS" get pvc "$PVC" -w
   

4. Restart Loki if the pod does not recover by itself:

   STS=$(kubectl -n "$NS" get statefulset \
     -l app.kubernetes.io/name=loki,app.kubernetes.io/component=single-binary \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl -n "$NS" rollout restart statefulset/"$STS"
   kubectl -n "$NS" rollout status statefulset/"$STS"
   

5. Fix the new size in LoggingConfig or else the next reconcile may return the old value:

   apiVersion: stackland.yandex.cloud/v1alpha1
   kind: LoggingConfig
   metadata:
     name: main
   spec:
     settings:
       logStorage:
         storage:
           enabled: true
           size: 100Gi
   

If PVC cannot be expandedIf PVC cannot be expanded

If StorageClass does not support expansion or VolumeGroup has no free space left, begin by freeing or adding capacity at the disk subsystem level. With that done, repeat PVC expansion.

If you cannot add space quickly, you can delete some of Loki's old local data. This is an emergency method that involves losing some of the logs.

1. Stop Loki:

   NS=stackland-logging
   STS=$(kubectl -n "$NS" get statefulset \
     -l app.kubernetes.io/name=loki,app.kubernetes.io/component=single-binary \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl -n "$NS" scale statefulset/"$STS" --replicas=0
   

2. Attach the PVC to a temporary pod and delete the oldest data from /var/loki/chunks. Before deleting, check which directories are taking up space:

   kubectl -n "$NS" run loki-pvc-debug --image=busybox:1.37 --restart=Never \
     --overrides='{"spec":{"containers":[{"name":"debug","image":"busybox:1.37","command":["sleep","1d"],"volumeMounts":[{"name":"storage","mountPath":"/var/loki"}]}],"volumes":[{"name":"storage","persistentVolumeClaim":{"claimName":"<PVC_NAME>"}}]}}'
   
   kubectl -n "$NS" exec -it loki-pvc-debug -- sh
   du -h /var/loki | sort -h | tail -20
   find /var/loki/chunks -type f -mtime +30 -exec rm -f {} \;
   exit
   

3. Remove the temporary pod and start Loki:

   kubectl -n "$NS" delete pod loki-pvc-debug
   kubectl -n "$NS" scale statefulset/"$STS" --replicas=1
   kubectl -n "$NS" rollout status statefulset/"$STS"
   

After the restorationAfter the restoration

Make sure retention and compactor are enabled for standalone:

spec:
  settings:
    logStorage:
      loki:
        singleBinary:
          enabled: true
          storage:
            size: 50Gi
          limitsConfig:
            retentionPeriod: 30d
          compactor:
            retentionEnabled: true
            retentionDeleteDelay: 2h

If Loki fills up PVC regularly even with retention on, decrease loki.singleBinary.limitsConfig.retentionPeriod, increase loki.singleBinary.storage.size, check the incoming log volume, and consider migrating to loki.simpleScalable with S3 storage.