VictoriaMetrics backups quickly

Quick backstory

Victoriametrics is a (mostly) PromQL-compatible TSDB that can consume data in many different formats, is very lightweight, has a great ecosystem of tools, a supportive community, and it’s fast. But what happens when you send bad or corrupted data to it? Or if you’re running a single instance and your storage disk(s) fail?

VictoriaMetrics provides a set of tools for backing up (and restoring) hardlinked copies of the entire dataset: vmbackup and vmrestore.

While configuring these tools is not difficult, having a quick example of a working backup script writing to GCP is nice. The biggest issue (for me) was getting GCP permissions into the vmbackup container, which wound up being as simple as passing a service account’s credentials into the container.

So anyway:


set -eo pipefail

# ensure we keep the backup container up to date
docker pull "${BACKUP_CONTAINER}"
CURRENT_VM_CONTAINER=$(docker ps --filter ancestor=victoriametrics/victoria-metrics:latest -q)

echo "Finding oldest snapshot..."
SNAPLIST=$(docker exec "${CURRENT_VM_CONTAINER}" wget -O- -q localhost:8428/snapshot/list | jq '.snapshots|length')
OLDEST=$(docker exec "${CURRENT_VM_CONTAINER}" wget -O- -q localhost:8428/snapshot/list | jq -r '.snapshots[0]')
echo "Generating a new snapshot..."
SNAPSHOT=$(docker exec "${CURRENT_VM_CONTAINER}" wget -O- -q localhost:8428/snapshot/create | jq -r .snapshot)

# incremental
echo "Uploading incremental snapshot..."
docker run --rm -v /opt/victoriametrics/vm-backup-auth.json:/backup-auth.json -e GOOGLE_APPLICATION_CREDENTIALS=/backup-auth.json -v /opt/victoriametrics/data/vm_data/:/data "${BACKUP_CONTAINER}" -storageDataPath=/data -snapshotName "${SNAPSHOT}" -dst gs://bucket/victoriametrics/latest
# daily
# this step handles moving data into "older" buckets over time, while ensuring we have a "latest" snapshot available for recovery.
# Managing object removal is handled by GCS
echo "Organizing daily snapshots..."
docker run --rm -v /opt/victoriametrics/vm-backup-auth.json:/backup-auth.json -e GOOGLE_APPLICATION_CREDENTIALS=/backup-auth.json -v /opt/victoriametrics/data/vm_data/:/data "${BACKUP_CONTAINER}" -storageDataPath=/data -snapshotName "${SNAPSHOT}" -dst "gs://bucket/victoriametrics/$(date +%Y%m%d)" -origin gs://bucket/victoriametrics/latest

if [[ "${SNAPLIST}" -gt 3 ]]; then
    # delete oldest snapshot
    echo "Removing our oldest snapshot: ${OLDEST}"
    docker exec "${CURRENT_VM_CONTAINER}" wget -O- -q "localhost:8428/snapshot/delete?snapshot=${OLDEST}"