Recover failed deployments with a compatible previous image #41

New issue

Open

opened 2026-11-01 20:11:58 +00:00 by viacheslavshuka · 0 comments

viacheslavshuka commented

2026-11-01 20:11:58 +00:00

Owner

A failed rollout should restore the previous healthy image when recovery is eligible. Image recovery must not silently reverse or erase database data.

Scope

Connect deployment verification failures to automatic application-image rollback. Check database compatibility and current vulnerability results before selecting the recovery image.

Acceptance criteria

A failed deployment can select the previous healthy image only when its database compatibility remains valid and its current vulnerability results permit deployment. Every vulnerability blocks unless an exact reviewed exception remains valid. Temporary vulnerability acceptance expires within thirty days.
Compatibility records honor the seven-day window for each replaced production version, including overlapping windows from frequent releases.
Recovery deploys the retained eligible image and verifies readiness plus its exact release identity. A Dokploy completion flag alone is insufficient.
The attempted release remains failed even after successful recovery. Reports distinguish the attempted image, recovery image, and outcome.
No eligible image produces a concrete failure requiring an operator recovery decision. The workflow cannot silently select an unverified older image.
Rollback never restores the database, deletes committed records, or assumes a previous schema merely because the application image changed.
Recovery participates in environment serialization. Concurrent rollout and rollback cannot leave an older image winning accidentally.
No disaster-recovery target, automated restore drill, private operations chat, or application backup system is added. Dokploy/R2 backups remain operator-managed.

Validation

Exercise failed readiness with an eligible previous image, incompatible schema, blocked vulnerability status, unavailable previous image, and failed recovery readiness. Check persisted data survives every recovery attempt.

Dependencies

#8: Apply PostgreSQL migrations with release compatibility checks. Rollback needs the recorded schema compatibility contract.
#36: Promote published releases without rebuilding their images. Recovery must respond to actual production deployment verification.

A failed rollout should restore the previous healthy image when recovery is eligible. Image recovery must not silently reverse or erase database data. ## Scope Connect deployment verification failures to automatic application-image rollback. Check database compatibility and current vulnerability results before selecting the recovery image. ## Acceptance criteria - A failed deployment can select the previous healthy image only when its database compatibility remains valid and its current vulnerability results permit deployment. Every vulnerability blocks unless an exact reviewed exception remains valid. Temporary vulnerability acceptance expires within thirty days. - Compatibility records honor the seven-day window for each replaced production version, including overlapping windows from frequent releases. - Recovery deploys the retained eligible image and verifies readiness plus its exact release identity. A Dokploy completion flag alone is insufficient. - The attempted release remains failed even after successful recovery. Reports distinguish the attempted image, recovery image, and outcome. - No eligible image produces a concrete failure requiring an operator recovery decision. The workflow cannot silently select an unverified older image. - Rollback never restores the database, deletes committed records, or assumes a previous schema merely because the application image changed. - Recovery participates in environment serialization. Concurrent rollout and rollback cannot leave an older image winning accidentally. - No disaster-recovery target, automated restore drill, private operations chat, or application backup system is added. Dokploy/R2 backups remain operator-managed. ## Validation Exercise failed readiness with an eligible previous image, incompatible schema, blocked vulnerability status, unavailable previous image, and failed recovery readiness. Check persisted data survives every recovery attempt. ## Dependencies - [#8: Apply PostgreSQL migrations with release compatibility checks](https://code.viacheslavshuka.com/viacheslavshuka/quorum/issues/8). Rollback needs the recorded schema compatibility contract. - [#36: Promote published releases without rebuilding their images](https://code.viacheslavshuka.com/viacheslavshuka/quorum/issues/36). Recovery must respond to actual production deployment verification. ## Related work - [#30: Block vulnerable dependencies and unsafe container configuration](https://code.viacheslavshuka.com/viacheslavshuka/quorum/issues/30) - [#42: Retain release images and redact CI evidence](https://code.viacheslavshuka.com/viacheslavshuka/quorum/issues/42)

viacheslavshuka added the

labels

2026-11-01 20:13:56 +00:00

viacheslavshuka referenced this issue

2026-11-01 20:15:11 +00:00

Apply PostgreSQL migrations with release compatibility checks #8

viacheslavshuka referenced this issue

2026-11-01 20:16:17 +00:00