In a script: sudo chmod -R apache:apache . /
Note that space? I didn't.
Devops Horror Stories
71–80 of 102 posts
Re: Devops Horror Stories
#72One thing I've learned: the real value of replication is how easy it makes it to handle strange events without getting stressed out.
It's 3AM. You're being paged with a high latency alert in one datacenter. You run one command to drain traffic out of that datacenter. The latency graph starts looking normal again. You go back to bed at 3:05. You look at the logs and figure out what went wrong tomorrow morning.
Re: Devops Horror Stories
#73You can hardly be surprised when OVH or Hetzner go down, just consider the price. Putting every server in one location is just stupid... as always the best way to fight downtimes is to spread servers across multiple providers & DC's.
Reminds me of a downtime report with my previous shell provider. They lost all Internet connection because someone had broken into a junction under a nearby freeway and cut all the fiberoptics and cables -- while preparing a robbery (presumably trying to cut alarm and/or off-site connections to cctv).
Turned out the two "redundant" providers of fiber both had fiber going through that junction...
Re: Devops Horror Stories
#74Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.
/mnt and /media exist for reasons. And root_squash and ... Why no, I've NEVER accidentally deleted whole file systems, I have completely earned superiority here. Delete /proc and /dev on a running server. Thankfully not really disastrous but damn if people don't notice right away. Thanks for the tmpwatch info btw.
Long ago I was trying to solve some thorny problem that I now forget and I thought it might be a good idea to uninstall and reinstall glibc. I knew it was a bad idea but I was at the point where I was past caring and figured why not see what happens. Turns out, in order to uninstall glibc there's a confirmation prompt, and instead of just 'y' or 'n' you have to type a whole sentence that's something like "yes, I understand this is a really bad idea". Well, I did that, and it was a really bad idea, the system was, unsurprisingly, basically unusable after that.
Re: Devops Horror Stories
#75Let my cofounder near the backups. Whoops.
Had a friend who recently took down his nic over ssh; he claimed he managed to get back in using some sort of serial over lan magic but I suspect he really just got someone on the other end to help.
Re: Devops Horror Stories
#76A few devops horror stories:
- Someone on the hardware team deleted several VMs that were being used as build machines, there were no backups. That wasted around 2 days to get things back to normal.
- During a show I volunteer for: a scissor lift drove over a just run (several hundred foot) ethernet line and severed it, they had to run a new line.
- PCs running windows being set up, as point-of-sale systems, to run with static IPs on the internet, without a firewall running. Disavowed all responsibility and left them on their own for that. They would have run them unpatched too without intervention.
- Someone checked a private key into the repository. Plan of action: obliterate from all branches everywhere, delete from all build drops (which contain source listings too), track down all build drop backups on tape and restore-delete-then-recreate them. Luckily I handed that job off to someone else.
Re: Devops Horror Stories
#77Coworker says "I'm going to do some clean-up on the server." Two minutes later, "Oh crap." He had wiped out /var/lib. And tell you what, the server kept working. We didn't dare rebooting it though.
Another fun one was coming in one morning, and cleaning up after somebody used some foul PHP provisioning scripts on a customer system and had the unfortunate idea to use a function called "archive". Turned out the function didn't so much "archive" as "delete". Henceforth deletion, especially unintended deletion, was known as "shotgun archival".
Re: Devops Horror Stories
#78service network stop
ifdown eth0
Re: Devops Horror Stories
#79Tape Archive System: write a tape, restore it again, and do MD5sum against the original data. Then we know it can be restored correctly, and the original data is deleted. Should be bullet proof? Alas, the 'write to tape' scripts I'd inherited didn't warn if they couldn't load a tape into the drive. There was a tape jammed in the drive, so the tape robot was refusing to load any new tapes, but kept on writing and rest…
I know one large company where contract operators managed to destroy every copy of a very large companys payroll by loading tape after tape onto a malfunctioning tape deck.
Yes, this was why I was paranoid about the whole write/restore/compare process.
I, being mainly a software guy, didn't consider the hardware robot as something that might fail w/o error.
Now, the process looks like:
Check drive is empty. Load Tape. Write Tape Unload. Load tape "42" from anther slot. Write 'slartibartfast' to that tape. Unload. Load original tape. Restore & Compare. Unload. Load tape "42". Restore, and make sure all it has is 'slartibartfast'.
This seems to me to have removed most of the possible silent-failure situations. If anyone can think of part of this algorithm that might fail, let me know!
Re: Devops Horror Stories
#80You can hardly be surprised when OVH or Hetzner go down, just consider the price. Putting every server in one location is just stupid... as always the best way to fight downtimes is to spread servers across multiple providers & DC's.
Yeah, I was kind off worries when they talked about OVH and Hetzner... they are notoriously bad... If you have to out-source your servers, at least pick a company that does it's job well (like LeaseWeb for instance).