Somewhere in your estate there is a warm standby environment that has never taken production traffic. It replicates data continuously, consumes budget every month, appears in the disaster recovery section of every architecture review, and has never once been asked to do its job. According to Cockroach Labs’ State of Resilience 2025 research, 71% of organisations perform no failover testing at all, and only 20% describe themselves as fully prepared for an outage. Quest Software’s 2026 survey of 650 IT and security leaders found that just 24% test disaster recovery at the recommended six-month cadence, while another 24% never test their plans. The industry has quietly settled into a position where disaster recovery testing is acknowledged as essential and practised by almost nobody. That gap is not a process failure. It is an architectural liability with a monthly invoice attached.
An untested disaster recovery environment is not a capability. It is a hypothesis, and an expensive one. The warm standby pattern’s entire value proposition, as AWS defines it, is that a scaled-down but fully functional copy of production can accept traffic immediately, needing only to scale up. Every word of that claim depends on the standby still matching production. The moment your teams ship a schema migration, rotate a certificate, adjust an IAM policy, or introduce a new managed service dependency without mirroring it to the standby, the hypothesis weakens. Nobody notices, because nothing is exercising the environment. The replication dashboards stay green, which measures data movement and nothing else. Whether the application tier can actually serve requests, whether DNS cutover works, whether the standby region has quota headroom for the scale-up, and whether your runbooks reflect the system as it exists today are all questions that only a failover can answer.
Configuration drift is the quiet killer here, and it compounds with time. A standby built eighteen months ago encodes eighteen-month-old assumptions: instance families that have since been superseded, security groups that predate your current network segmentation, secrets that have rotated in production but not in the recovery region. Drift is not a hygiene problem to be tidied up quarterly. It is the mechanism by which a warm standby degrades from recovery capability to recovery theatre, and the degradation is invisible precisely because the environment is idle. The multi-region patterns explored in our guide to multi-region cloud architecture at enterprise scale only deliver their promised RTO if the passive side is continuously reconciled against the active one, and reconciliation without verification is just optimism with tooling.
The public post-mortem record makes the point better than any survey. GitLab’s 2017 database incident revealed that of five backup and replication mechanisms, none was working reliably when needed: the secondary had been wiped during replication recovery, and the logical backup job had been failing silently for weeks because its failure notifications were being rejected by the mail server. Cloudflare’s November 2023 control plane outage showed a more subtle version of the same lesson: the disaster recovery facility worked for the core control plane, but more services were offline than anticipated because newer products had not been built to the same recovery standard. Coinbase’s May 2026 exchange outage exposed the absence of automated cross-zone failover for its matching engine, and the remediation plan is telling: the company committed to regular production failover exercises, not merely better documentation. In each case the organisation believed it had a recovery posture. In each case the belief had never been tested against reality, and reality disagreed. These failures echo the verification gap discussed in our review of AWS backup strategies beyond lifecycle rules: the existence of a mechanism proves nothing about its function.
Then there is the economics, which should embarrass more CFOs than it does. A warm standby is a continuous cost: compute, storage, replication traffic, and the engineering time spent maintaining it. ITIC’s hourly cost of downtime research finds that over 90% of mid-size and large enterprises put a single hour of downtime above $300,000, and Cockroach Labs found a third of organisations losing between $100,000 and $1 million or more per outage. Those numbers justify the standby’s premium, but only if the policy pays out. Paying for warm standby you never test is the worst of both worlds: you carry the full cost of the insurance and the full risk of the disaster. An untested standby that fails during a real event belongs in the same category as the waste catalogued in our piece on zombie resources draining cloud budgets, with the added insult that it fails at the precise moment it was bought for.
The honest position is binary. Either failover is a routine, scheduled, production-grade operation, exercised at least twice a year with the standby actually taking traffic, or your organisation does not have warm standby and should stop paying for it. Downgrading deliberately to backup and restore, with tested restores and an accepted longer RTO, is a defensible architecture. Warm standby that exists only in diagrams is not: it is the most expensive form of hope in your estate. Audit when your DR environment last served a production request. If the answer is never, you already know what your recovery posture actually is.
Useful Links
- AWS: Disaster Recovery Options in the Cloud
- AWS Well-Architected: Use Defined Recovery Strategies (REL13-BP02)
- Cockroach Labs: The State of Resilience 2025
- GitLab: Postmortem of Database Outage of January 31
- Cloudflare: Post Mortem on the Control Plane and Analytics Outage
- Coinbase: A Postmortem of Our May 7, 2026 Outage
- Quest Software: State of ITDR Survey 2026
- Gremlin: Modernizing Disaster Recovery Testing
