Runbook · RB-01

Sync Queue Stuck

Your bucket sync queue depth is elevated and not draining. Changes are not replicating to your BYO storage bucket. This runbook explains the causes, what Vulos checks automatically, and how to restore normal sync.

How sync works

Vulos uses cr-sqlite (conflict-free replicated SQLite) to sync changes between your device and your BYO MinIO/S3 bucket. Each write produces a signed change record that is queued for upload. The sync daemon processes the queue in batches; under normal conditions the queue depth stays close to zero.

A stuck queue means the daemon cannot reach the bucket, is encountering permission errors, or the bucket is full.

Signals on your Status page

Signal	What it means	Threshold
Sync queue depth	Number of change records queued but not yet uploaded	>100 → warn, >1 000 → critical
Bucket sync lag	Age of the oldest queued record in seconds	>300 s → warn, >1 800 s → critical
Disk used (%)	Percentage of disk used on the device	>85 % → warn (can cause queue growth)

What Vulos does automatically

Auto-remediation (SUPPORT-02) cross-reference: when sync queue depth exceeds the critical threshold, Vulos pauses non-essential background writes to reduce queue growth, retries the bucket connection with exponential backoff, and rotates to a fallback relay endpoint if the primary bucket is unreachable. A critical alert fires and the account owner is notified.

Common causes

Cause	How to identify
Expired bucket credentials	Alert detail will include 403 Forbidden or SignatureDoesNotMatch
Bucket deleted or renamed	Alert detail will include NoSuchBucket
Network partition / firewall rule	Sync lag grows but no bucket error; check connectivity from device
Bucket quota exceeded	Alert detail will include QuotaExceeded or 507
Disk full on device	Disk used % signal is >95 %; see RB-02 — Disk Full

What you can do

1. Read the alert detail

Open your Status page, expand the Sync group, and read the alert detail for the sync queue signal. The error code in the detail identifies the root cause.

2. Rotate bucket credentials (403 / credential error)

Navigate to your bucket provider (MinIO, Tigris, AWS S3, or Cloudflare R2) and generate a new access key. Update the key in your Vulos device configuration:

# On the device (SSH or console):
sudo vulos-config set bucket.access-key-id     AKIAXXX...
sudo vulos-config set bucket.secret-access-key SECRETXXX...
sudo systemctl restart vulos-sync

The sync daemon will reconnect within 60 seconds and begin draining the queue.

3. Recreate or rename the bucket (NoSuchBucket)

If the bucket was deleted or renamed, recreate it with the original name or update the bucket endpoint in Vulos configuration:

sudo vulos-config set bucket.name    your-new-bucket-name
sudo vulos-config set bucket.endpoint https://s3.amazonaws.com
sudo systemctl restart vulos-sync

Data note: if the old bucket was deleted with unsynced data, those change records are still in the on-device queue and will upload to the new bucket. No data is lost as long as the device itself is intact.

4. Check connectivity

From the device, confirm you can reach the bucket endpoint:

curl -I https://your-bucket.s3.amazonaws.com/

A firewall rule blocking outbound HTTPS to the bucket host is a common culprit in self-hosted or enterprise network environments.

5. If queue does not drain after fixes

Open a support ticket from your Account page. Include the alert detail text, the sync queue depth, and the bucket provider. Our team can inspect sync daemon logs to diagnose corruption or protocol-level issues.

Prevention

Rotate bucket credentials on a 90-day schedule and update Vulos config before the old key expires.
Set bucket lifecycle rules to archive old change records after 90 days to avoid quota exhaustion.
Monitor disk usage — a full disk blocks the sync daemon from writing the queue journal.
Use the Status page to watch queue depth; a consistently non-zero queue warrants investigation.

RB-02 — Disk Full

← PreviousRun a relay — redundancy & scaling Next →RB-02: Disk Full