The outage that taught me the most comes down to one character. About four years ago, on my e-commerce server, I typed rm -rf /var instead of rm -rf var. The first command empties a local folder. The second wipes the system directory that held the whole site. I had no daily backup, and I had to rebuild three days of orders by hand.
01What happened
I wanted to delete a var folder inside the project directory, a routine task. I added a leading slash without noticing. With that slash, the path no longer starts from the current folder but from the root of the server, and /var is where the site's files lived.
The command ran without asking anything. That is how rm behaves with the -f option: no questions. GNU rm's built-in protection, the --preserve-root option that is on by default, only refuses to delete the root itself, as the rm manual page explains. It does not protect /var.
The store ran on one EC2 instance, an RDS database and CloudFront in front. The site vanished at once.
02What I had to redo
Getting the site back online was not the slow part. The orders were. I had no daily backup: everything since the last usable backup was gone on the site side, and I rebuilt three days of orders by hand, one at a time.
That is where an outage really costs you. A server can be rebuilt. Customer orders, with their addresses, payments and shipping status, have to be re-entered by hand, and every typo becomes a problem for a real customer.
A backup that is several days old means several days of manual work on the day you need it.
03What I took from it
The lesson fits in two lines. A destructive command is typed with full focus, never on autopilot. And one slash is the whole difference between cleaning a folder and wiping a server.
Since then, one rule never moves: a daily backup on every project, and up to two a day on some of them.
Backup frequency should match what you are willing to redo by hand. If losing half a day of orders is already too much, one backup a day is not enough. I go deeper in my article on backups.
04The guardrails I recommend
Nothing clever. Just habits that make the mistake harder to make.
GNU rm's -I option asks once before removing more than three files or removing recursively, according to the same manual. It is not absolute protection, but it forces a pause at the right moment.
The rest is about the backups themselves: test them with a real restore, and keep them off the server they protect. A backup stored in /var would have disappeared with everything else.
If you run an online store, my guide to e-commerce infrastructure covers the rest of the stack.
05FAQ
Can rm -rf / still wipe a whole server?
On a recent GNU system, rm refuses to delete the root itself thanks to --preserve-root, which is on by default. But rm -rf /var, /home or /etc goes through without a question. The protection only covers the bare slash.
How often should an online store be backed up?
At least once a day, and more often if you take many orders. The real question is how many hours of orders you are willing to re-enter by hand. On some projects, my answer became two backups a day.
How do you avoid this kind of mistake?
Use explicit relative paths like ./var, check the current directory before deleting, turn on rm -I and, above all, keep a recent backup stored somewhere else. The mistake will happen one day: the backup decides what it costs.
06Sources
One slash was enough to wipe a production site. Since then, no project without a daily backup, and two a day where data changes fast.
Talk about your project ↗