Cleaning up full instance disk space

A GitLab instance can run out of disk space. This is indicated by the 500 Internal Server Error HTTP status code. In which case you will not be able to access the GitLab instance.

You can increase the instance disk space yourself.

To reduce the probability of running out of disk space in future:

• Clean up the instance disk space manually at regular intervals.
• Periodically delete irrelevant pipelines.
• Configure disk space cleanup policies.

You can also set up alerts for the Data disk chart in monitoring.

Clean up the instance disk spaceClean up the instance disk space

1. Manually review all Docker images and their tags in GitLab Container Registry.
2. Delete obsolete images and tags.

Delete obsolete pipelinesDelete obsolete pipelines

Pipelines create jobs, which in turn generate logs. The longer and more actively you use a GitLab instance, the more disk space you need to store logs. To free up space, you can delete unnecessary pipelines, thus also deleting jobs and their logs.

With the GitLab web interface, you can only delete one pipeline at a time. To bulk delete unnecessary pipelines and their logs, use a script:

#!/bin/sh
set -e

TOKEN=<your_token>
PER_PAGE=<number_of_pipelines_to_delete>
UPDATED_BEFORE=<date_of_last_update>
GITLAB_URL=<GitLab_instance_FQDN>

for PROJECT in $(curl -sL --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects?per_page=$PER_PAGE" | jq '.[].id') ; do
  echo "Deleting in project $PROJECT"
  for PIPELINE in $(curl -sL --header "PRIVATE-TOKEN: $TOKEN" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines?per_page=$PER_PAGE&sort=asc&updated_before=${UPDATED_BEFORE}T00:00:00Z" | jq '.[].id') ; do
      echo "Deleting pipeline $PIPELINE"
      curl -L --header "PRIVATE-TOKEN: $TOKEN" --request "DELETE" "$GITLAB_URL/api/v4/projects/$PROJECT/pipelines/$PIPELINE"
  done
done

Where:

• TOKEN: GitLab token of the user with the Owner role in the project.
• PER_PAGE: Number of pipelines to delete per request. You can delete a maximum of 100 records at a time. If you have more, you will need to run the script several times.
• UPDATED_BEFORE: Upper limit of the update date in YYYY-MM-DD format. The script will delete all pipelines updated prior to this date.
• GITLAB_URL: GitLab instance FQDN.

Configure disk space cleanup policiesConfigure disk space cleanup policies

You can use any of the following options:

• Set the expiration time for job artifacts at the instance level. Its default value is 30 days.
• Create and enable a tag cleanup policy for separate projects utilizing GitLab Container Registry.
• If you are using a Yandex Container Registry integration, create and configure a Docker image cleanup policy for Yandex Container Registry.