Live data from Hacker News

Devops Horror Stories

statuspage.io

31–40 of 102 posts

Re: Devops Horror Stories

#31
My worst horror story was a full server room shutdown. We killed servers, then the chillers, and then started work. About an hour after we started, we pulled our first floor tile to move some power cables. There was water under the floor! We spent the next few hours cleaning up all the water.

Apparently water kept flowing into the humidifier tray of the chiller, and the mechanical auto-shutoff never triggered. The pump didn't remove water from the tray because the power was off.

Facilities "fixed" the humidifier, but it still happened again when that circuit was cut off for work elsewhere in the building. No one caught the water overflow, and it flowed out down the conduits to the first floor. So we had flooding on 2 different floors from a single chiller.

Re: Devops Horror Stories

#32
post #20

Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.

So tmpwatch traverses filesystems. Is that a bug? (thinks: "-xdev").

Edit: Reading tmpwatch.c: "Try hard not to go onto a different device". Perhaps old bug?

Re: Devops Horror Stories

#33
TODO: Monday Morning: T1 install will be complete. Tuesday: Test/bootup period. Wednesday: Sales start Thursday: Sales continue, TV ad goes live Friday: Champagne!

Reality: Monday Morning: T1 did not get installed. Tuesday: Emergency ISDN solution (stolen from Chiropractors next door) Wednesday: Modem rack catches fire Thursday: TV ad goes Live Friday: T1 goes live. Champagne.

Re: Devops Horror Stories

#34

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

It's not always the person who pushed it on a friday that ends up fixing it, though. They can be unreachable, without a computer, etc etc. It's just easier to change less during hours you have less people on hand, is all.

Re: Devops Horror Stories

#36

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

> - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

this is why I refuse to do "View Source" on the HealthCare.gov website. I'm afraid of what I might see.

Re: Devops Horror Stories

#37
post #17

"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?

This was likely an inode issue on the operating system/kernel and how it represents S3 on the filesystem, especially if you're using something like S3FS.

Re: Devops Horror Stories

#38

TODO: Monday Morning: T1 install will be complete. Tuesday: Test/bootup period. Wednesday: Sales start Thursday: Sales continue, TV ad goes live Friday: Champagne! Reality: Monday Morning: T1 did not get installed. Tuesday: Emergency ISDN solution (stolen from Chiropractors next door) Wednesday: Modem rack catches fire Thursday: TV ad goes Live Friday: T1 goes live. Champagne.

this is why given a choice between theory/plans/estimates/schedules or, say... reality and iterating and observing what-actually-happens ... I always prefer the latter. in software engineering, in human relationships, and in the physical world around me in general.

Re: Devops Horror Stories

#39
post #17

"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?

This was likely an inode issue on the operating system/kernel and how it represents S3 on the filesystem, especially if you're using something like S3FS.

Ah, my brain was somehow not seeing that in the article, but I see it now that I'm looking for it. Thanks!

Re: Devops Horror Stories

#40
post #20

Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.

/mnt and /media exist for reasons. And root_squash and ...

Why no, I've NEVER accidentally deleted whole file systems, I have completely earned superiority here.

Delete /proc and /dev on a running server. Thankfully not really disastrous but damn if people don't notice right away.

Thanks for the tmpwatch info btw.

Post reply on HN