Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devops Horror Stories

statuspage.io

Post reply on HN
Print view View original

41–50 of 102 posts

Re: Devops Horror Stories

#41

Post by wizzard »

mkramlich wrote: post #38
fit2rule wrote: post #33

TODO: Monday Morning: T1 install will be complete. Tuesday: Test/bootup period. Wednesday: Sales start Thursday: Sales continue, TV ad goes live Friday: Champagne! Reality: Monday Morning: T1 did not get installed. Tuesday: Emergency ISDN solution (stolen from Chiropractors next door) Wednesday: Modem rack catches fire Thursday: TV ad goes Live Friday: T1 goes live. Champagne.

this is why given a choice between theory/plans/estimates/schedules or, say... reality and iterating and observing what-actually-happens ... I always prefer the latter. in software engineering, in human relationships, and in the physical world around me in general.

Well, sure, as logical people we know that you can't predict failures and that it's always better to play it by ear. Unknown unknowns and all that.

I have worked at several places where salespeople have sold a feature without even asking if it was POSSIBLE, much less created/deployed/tested. "We just sold [Feature X], we told them it'd be ready by [date pulled out of thin air]."

Re: Devops Horror Stories

#42

Post by Fizzadar »

You can hardly be surprised when OVH or Hetzner go down, just consider the price. Putting every server in one location is just stupid... as always the best way to fight downtimes is to spread servers across multiple providers & DC's.

Re: Devops Horror Stories

#43

Post by jmccree »

ecopoesis wrote: post #24
0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

I'd agree that in a perfect scenario, you should be able to push code at any time confidently. But for many companies and projects this is not really available. As well, in many organizations the person who has to fix broken stuff is not the same as who develops and pushes code. I'm not saying that's a good thing, but it is reality for many people.

Even if it only happens once in your career, once you've had a dev push out code at 5pm friday night, jet out the door and hit the bar, meanwhile you (the sysadmin/ops on call) get woken up at 1am by site down alert, and have to debug/rollback the changes while the dev who pushed them is unreachable, you learn to really avoid friday evening pushes. Fool me once...

Re: Devops Horror Stories

#44

Post by Ives »

nailer wrote: post #20

Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.

I did something similar. I moved some files around between a university computer and an SFTP mount (mounted through the ubuntu UI). When I leave those computers, I always execute "rm -rf ~" because they are restored to some image every time they boot anyway and I'd rather not leave anything personal behind. It was only when I started seeing "access denied" in my terminal that I realised that ubuntu had mounted the entire server on the other end in some hidden directory in my home folder, and that I was deleting every file on it for which I had write permissions. Luckily a quick CTRL+C saved my own files but I'm not sure the same could be said about a few students who were unlucky and had "loose" permissions in their home directories.

Re: Devops Horror Stories

#45

Post by [deleted] »

ecopoesis wrote: post #24
0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

[deleted]

Re: Devops Horror Stories

#47

Post by wahnfrieden »

rdw wrote: post #17

"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?

It's still an extremely bad idea to have overly common key prefixes on S3 since it prevents balancing key distribution across their clusters.

Re: Devops Horror Stories

#48

Post by jeffbarr »

rdw wrote: post #17

"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?

There has never been an S3 limit. Some of our customers have millions of objects in a single bucket.

When you do something like this you need to make sure that you have a good distribution of keys across the name space, and you need to think twice before you decide to write code to list the entire bucket. In most use cases at this scale, metadata and indexing are handled by something other than S3.

Re: Devops Horror Stories

#49

Post by aaronem »

mkramlich wrote: post #36
0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

> - Troubleshooting a bug in a site, view source.... and see SQL in the JS. this is why I refuse to do "View Source" on the HealthCare.gov website. I'm afraid of what I might see.

I'm just gonna leave this right here:

        if ('en' === 'en') {
            $('#desktop-nav .middle').append('My Profile');
            $('.mobile-nav-right').append('My Profile');
        } else {
            $('#desktop-nav .middle').append('My Profile');
            $('.mobile-nav-right').append('My Profile');            
        }

Re: Devops Horror Stories

#50

Post by splitrocket »

We launched our brand new service into production pointing the backend at our dev instance, at the office. The entire internet showed up at our wee little DSL connection, effectively DDOS-ing our office. We had to leave, go to a cafe with public wifi to fix it.
Post reply on HN