Every blue-green deployment pitch makes the same promise: if the release goes wrong, flip the router back and you’re instantly safe. Senior teams have built entire release strategies on that promise, and for the application tier it holds up. The moment a schema migration or a new data format sits underneath the deployment, though, the guarantee quietly stops applying, and most teams don’t notice until the switch back fails.
The database is what breaks the symmetry. Blue and green usually share one, and that shared database is the single point of coupling the marketing material never mentions.
Rollback as a trivial reversal of the router is a fair description of stateless compute, and it’s roughly how Google’s own SRE workbook frames blue-green as a form of canary release. It is not a fair description of anything stateful. Add a column the old code doesn’t expect, drop one it still reads, or write a new JSON structure where the old version expects a scalar, and the idle environment is no longer a safe rollback target. Switching back doesn’t undo the change, it just points broken code at broken data, or working code at data it can no longer parse. The failure is not that blue-green is a bad pattern. It’s that its central promise was scoped to the application layer from the start, and most teams never noticed the scope was that narrow.
The clearest evidence comes from the vendors selling the pattern. AWS’s own RDS and Aurora Blue/Green Deployments feature, the product literally named after this technique, ships with no built-in rollback: once switchover completes, writes to the old environment simply stop replicating anywhere, and AWS publishes a separate guide on manually rebuilding a reverse replication path because the platform doesn’t give you one. Azure App Service deployment slots swap application content and configuration, but never touch the database, which only works cleanly for as long as nobody changes the schema. Google Cloud Deploy states outright, in its own documentation, that rolling back application code does not reverse a migration that has already run. Every platform automates the half of the problem that’s easy and leaves the hard half exactly where it always was.
GitLab’s January 2017 outage is the sharpest public lesson here. A destructive command against the production database, run during an unrelated fix, lost roughly six hours of data, including thousands of projects, comments and new accounts, and the postmortem admitted candidly that database recovery had never actually been tested end to end. GitHub’s May 2021 incident ran the same problem in reverse: a schema field exhausted its integer range, and the only way out was a migration with no reverse gear at all, because the data had already outgrown what the old schema could represent. When schema rollback truly isn’t available, restore from snapshot is the real fallback, not a debugging session under pressure, a point covered in our comparison of snapshot APIs and traditional backup agents. Neither company was careless by industry standards. Both discovered, under pressure, that the rollback plan they trusted covered code and not data.
None of this means abandoning blue-green, it means stopping the database from being the single shared dependency that breaks it. Expand and contract migrations, sometimes called parallel change, add new structures without removing old ones, run both in parallel for a release or two, and only drop the old path once nothing depends on it any more, which is exactly the zero-downtime discipline behind the slot-warming approach explored in our guide to Azure App Service slot automation. Stripe’s well documented approach to large-scale data migration went further still, dual writing to old and new stores and shadow reading between them until the mismatch rate hit zero before cutover. Both patterns share the same underlying idea: keep the previous version of the application able to run against current data for as long as rollback needs to remain real.
The uncomfortable finding sits in the survey data. Redgate’s most recent State of the Database Landscape report found 71% of organisations still rely on manual methods for test data, and most teams have no formal process for sharing database change practices at all. DORA’s own research treats database change management, tested against production-like data, as one of the practices that separates elite delivery performance from everyone else. Put those together with the disaster recovery patterns explored in our piece on multi-region cloud architecture, and a pattern emerges: teams rehearse infrastructure failover far more often than they rehearse a database rollback, even though the database is where the actual risk sits.
None of this is an argument against blue-green deployment. For fully stateless releases, the instant rollback promise is real and worth keeping. The argument is against treating that promise as universal when the release underneath it touches a shared database, because that’s exactly where every postmortem in this piece went wrong. The practical test is simple: before your next schema-touching release ships, ask whether anyone has actually flipped back to the old environment and confirmed it still works against current data, not whether the code deploy succeeded. If the honest answer is no, the rollback plan is a belief, not a capability. Treat every migration that isn’t purely additive as a decision that needs its own rollback design, not an assumption inherited from the deployment pattern sitting on top of it.
Useful Links
- Bytebase, Database Blue-Green Deployment: A Practical Guide
- Martin Fowler, Evolutionary Database Design
- Stripe Engineering, Online migrations at scale
- Shopify Engineering, Safely Adding NOT NULL Columns to Your Database Tables
- GitHub, gh-ost, GitHub’s online schema migration tool
- GitLab, Postmortem of database outage of January 31
- AWS Database Blog, Implement a rollback strategy for an Amazon Aurora blue/green deployment switchover
- Google Cloud, Migrate to Google Cloud: Best practices for validating a migration plan
- Google SRE, Canarying Releases
- DORA, Capabilities: Database change management
