← All articlesInfra and cloud

rm -rf /var: the mistake that taught me the most

One extra character, a whole site wiped, three days of orders to rebuild. And daily backups everywhere since.

Julien Mauclair · ~3 min · intermediate
StackLinuxAWSBackups

The outage that taught me the most comes down to one character. About four years ago, on my e-commerce server, I typed rm -rf /var instead of rm -rf var. The first command empties a local folder. The second wipes the system directory that held the whole site. I had no daily backup, and I had to rebuild three days of orders by hand.

01What happened

I wanted to delete a var folder inside the project directory, a routine task. I added a leading slash without noticing. With that slash, the path no longer starts from the current folder but from the root of the server, and /var is where the site's files lived.

The command ran without asking anything. That is how rm behaves with the -f option: no questions. GNU rm's built-in protection, the --preserve-root option that is on by default, only refuses to delete the root itself, as the rm manual page explains. It does not protect /var.

The store ran on one EC2 instance, an RDS database and CloudFront in front. The site vanished at once.

02What I had to redo

Getting the site back online was not the slow part. The orders were. I had no daily backup: everything since the last usable backup was gone on the site side, and I rebuilt three days of orders by hand, one at a time.

That is where an outage really costs you. A server can be rebuilt. Customer orders, with their addresses, payments and shipping status, have to be re-entered by hand, and every typo becomes a problem for a real customer.

Careful

A backup that is several days old means several days of manual work on the day you need it.

03What I took from it

The lesson fits in two lines. A destructive command is typed with full focus, never on autopilot. And one slash is the whole difference between cleaning a folder and wiping a server.

Since then, one rule never moves: a daily backup on every project, and up to two a day on some of them.

Backup frequency should match what you are willing to redo by hand. If losing half a day of orders is already too much, one backup a day is not enough. I go deeper in my article on backups.

04The guardrails I recommend

Nothing clever. Just habits that make the mistake harder to make.

guardrails.shBASHCopy
# Always an explicit relative path, from a checked directory
cd /path/to/project || exit 1
rm -rf ./var/cache
# Ask for confirmation before a recursive or large delete
alias rm='rm -I'
# Two backups a day, at 3 am and 3 pm (crontab)
0 3,15 * * * /usr/local/bin/backup.sh

GNU rm's -I option asks once before removing more than three files or removing recursively, according to the same manual. It is not absolute protection, but it forces a pause at the right moment.

The rest is about the backups themselves: test them with a real restore, and keep them off the server they protect. A backup stored in /var would have disappeared with everything else.

If you run an online store, my guide to e-commerce infrastructure covers the rest of the stack.

05FAQ

Can rm -rf / still wipe a whole server?

On a recent GNU system, rm refuses to delete the root itself thanks to --preserve-root, which is on by default. But rm -rf /var, /home or /etc goes through without a question. The protection only covers the bare slash.

How often should an online store be backed up?

At least once a day, and more often if you take many orders. The real question is how many hours of orders you are willing to re-enter by hand. On some projects, my answer became two backups a day.

How do you avoid this kind of mistake?

Use explicit relative paths like ./var, check the current directory before deleting, turn on rm -I and, above all, keep a recent backup stored somewhere else. The mistake will happen one day: the backup decides what it costs.

In short

One slash was enough to wipe a production site. Since then, no project without a daily backup, and two a day where data changes fast.

Talk about your project ↗
Julien MauclairJulien Mauclair is the co-founder and CTO of Klickbee, where he builds the agency’s websites, tracking and AI automations, and develops Klickbee Tracking. He also runs Stralya, which designs and runs AWS infrastructure for scale-ups. Before that, he built the entire AWS infrastructure of an e-commerce business, since sold.

Read next