Backup Is Not Disaster Recovery
I've had this conversation at least twice a year for the last decade. Someone says "we have backups, we're covered." And I say "okay, when's the last time you tested a restore?" The silence that follows is always the same.
Backup is not DR
Backup answers the question: "can I get my data back?" Disaster recovery answers: "can I get my business back online?" These are completely different problems with different RPOs, different RTOs, and different architectures.
A backup that takes 48 hours to restore from tape is not a disaster recovery solution. It's a compliance checkbox. If your RTO is 4 hours and your restore takes 2 days, you don't have DR. You have a backup and a hope.
I've seen organisations spend six figures on backup infrastructure and zero on DR planning. Then a storage array fails and suddenly everyone's looking at each other wondering why the "backup" they paid for isn't bringing systems back online.
RPO and RTO are not abstract
The recovery point objective is how much data you're willing to lose, measured in time. The recovery time objective is how long you're willing to be down. These numbers should drive every technical decision in your backup and DR architecture, but in practice they're usually numbers someone put in a spreadsheet three years ago and never revisited.
Here's what I've learned from actually recovering systems at 3am:
RPO is a lie if you don't test it. Your backup logs say the last successful backup was at 2am. Great. Did you test that particular backup? Is the backup file corrupt? Does the restore actually work? I've had backup software report "successful" for weeks while writing corrupted data. The only thing a "successful" backup log tells you is that the software didn't crash. It doesn't tell you the backup is restorable.
RTO is always longer than you think. Every DR plan I've seen estimates RTO based on data transfer time plus a bit of overhead. Reality: you need to spin up infrastructure, configure networking, validate that services are functioning, test that users can access them, and coordinate communication with the business. Data transfer time is maybe 30% of actual RTO. The rest is process, validation, and the inevitable troubleshooting when something doesn't come back clean.
The backup window is shrinking. With 24/7 global operations, there's no "overnight" anymore. Backup windows that worked five years ago now overlap with production hours. I've had to redesign backup schedules around business hours in three different time zones. It's tedious but necessary.
What I actually do
For every environment I manage, I maintain a simple document: for each critical system, what is the RPO, what is the RTO, what is the backup mechanism, when was the last restore test, and who gets called if it fails. This document lives in the same repo as the infrastructure code. It gets updated when the architecture changes. It gets reviewed quarterly.
I test restores monthly. Not "spot-check one file." Full system restores to isolated environments. Yes, it's time-consuming. Yes, it catches problems. The cost of finding a restore failure during a test is a few hours of my time. The cost of finding it during an actual incident is measured in business impact.
I treat backup infrastructure as production infrastructure. If the backup server goes down, it's a P1 incident. Because without verified backups, you don't have a safety net. You have a false sense of security, which is worse than no backup at all.
The conversation I keep having
"Should we back up to the same cloud region or a different one?"
Different one. Always. If a regional outage takes down your production and your backups simultaneously, you're not recovering anything. Cross-region backup storage costs marginally more. The alternative costs your job.
"But cross-region egress charges..."
Yes, there are egress charges. They're called insurance premiums. Pay them.
"What about off-site backups?"
Yes. If your backup data is in the same physical location as your production, a fire or flood takes out both. Off-site storage or a second region is not optional if you're serious about DR. It costs more. That's insurance.
This isn't rocket science. Backup and DR are well-understood problems. The hard part isn't the technology. It's the discipline: testing restores, maintaining documentation, treating backup infrastructure as production, and having the uncomfortable conversation about what "recovery" actually means before you need it.