Upgrade and roll back
Migrations are built into the binary and run when it starts, so an upgrade is: back up, swap the image, check. A rollback is the same in reverse, with one difference explained below.
Upgrade
Back up
Take a backup you can restore. See Backup and restore.
docker exec sentinel /app/sentinel-admin db backup -out /backups/before-upgrade.dump
Read the release notes
Look for new environment variables and anything marked as a change to behaviour. Verify the image signature before you run it: see Installation.
Replace the container
Start the new image with the same environment variables and the same database.
docker pull "$SENTINEL_IMAGE:$NEW_VERSION"
docker stop sentinel && docker rm sentinel
docker run -d --name sentinel ... "$SENTINEL_IMAGE:$NEW_VERSION"
It applies each pending migration and logs which ones it ran. If a migration fails, the container stops instead of serving a schema the code doesn't match.
Check
GET /healthz returns the new version. Sign in, open the alert queue, and send one test event. Rule activations and screening lists take up to 30 seconds to reach every instance.
With several instances
Instances starting together take turns applying migrations, and the others wait. Replace them one at a time behind your load balancer. Zero-downtime upgrades aren't built yet, so do them in a quiet period and have the sending services retry. Events carry an idempotency_key, so a retry never counts twice.
Roll back
Going back to the previous image alone works only when the new version made no schema changes. The application only moves the schema forward and doesn't undo migrations. So:
| The upgrade | To go back |
|---|---|
| Ran no migrations | Start the previous image |
| Ran migrations | Stop Sentinel, restore the backup you took before the upgrade into the database with -replace, then start the previous image |
docker stop sentinel
docker run --rm --entrypoint /app/sentinel-admin -v /backups:/backups -e SENTINEL_DATABASE_URL ... \
"$SENTINEL_IMAGE:$OLD_VERSION" db restore -in /backups/before-upgrade.dump -replace
docker run -d --name sentinel ... "$SENTINEL_IMAGE:$OLD_VERSION"
Restoring brings back the database as it was at the backup, so anything that happened since is gone. That includes events, alerts and filings. Resend events from your services. Alerts and filings made in that time have to be redone by hand, so decide quickly, and don't roll back for something a fix forward would solve.
The log shows which migrations an upgrade ran. If it ran none, you don't need the restore.