Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devops Horror Stories

statuspage.io

Post reply on HN
Print view View original

81–90 of 102 posts

Re: Devops Horror Stories

#81

Post by lil_cain »

pygy_ wrote: post #54
0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

- It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. How did you locate it? Measuring ping latency from other machines?

I'd imagine tracing routing, and then MAC address tables would be rather a lot faster, and more accurate.

Re: Devops Horror Stories

#83

Post by pygy_ »

lil_cain wrote: post #81
pygy_ wrote: post #54

Earlier quoted context omitted.

- It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. How did you locate it? Measuring ping latency from other machines?

I'd imagine tracing routing, and then MAC address tables would be rather a lot faster, and more accurate.

Indeed :-)

Re: Devops Horror Stories

#84

Post by djrobstep »

wahnfrieden wrote: post #47
rdw wrote: post #17

"you can't have more than 64,000 objects in a folder in S3 - even though S3 doesn't have folders." Is this for real, or are these stories made up? All documentation I've read about S3 suggests that it does not have any file count limitations. The timeline of Togetherville suggests that this story took place between 2008 and 2010. Did S3 have a limit back then that they lifted?

It's still an extremely bad idea to have overly common key prefixes on S3 since it prevents balancing key distribution across their clusters.

Do you have a citation for that? I'd hope that the key distribution is a bit smarter than prefix-based.

Re: Devops Horror Stories

#85

Post by vacri »

ch4ch4 wrote: post #5
jamescun wrote: post #4

Earlier quoted context omitted.

Not so much an example of why not to use budget providers, more an example of why you should build highly available infrastructure. I don't believe there is any provider, "budget" or not, that guarantees a servers reliability 100% of the time.

I was alluding more to the customer support aspect of it. If a technician spends one hour troubleshooting your network problems, then they've already lost their profit for the month.

This is one thing where I find AWS shines. I'm on the lowest level of paid support, and I've had nothing but excellent service from good technicians who will try to actually replicate your problem, then contact other teams if they fail or there's follow up. Out of a dozen or so tickets, I've only had one where the response wasn't genuinely useful, and that was for an issue that may have been due to internet weather anyway.

Support is one of things that you can get along without... until you need it. Then you really, really wish you had it.

Re: Devops Horror Stories

#86

Post by redbad »

The first two stories are notable in how they reflect the terrible practices of the teller.

"Our distributed application produces the same type of error after the same period of time in totally different data centers. We have no idea why, but moving data centers seems to help, so we just keep doing it. #YOLO"

"We've built a product on a data store and library we don't understand even the highest-level constraints of. That ignorance bit us in the ass at peak load. We patched over the problem and continue gleefully into the future. #YOLO"

These stories should be embarrassing, but they're seemingly being celebrated, or at least laughed about. Am I off base?

Re: Devops Horror Stories

#87

Post by vincentkriek »

SEJeff wrote: post #70
mhurron wrote: post #40

Earlier quoted context omitted.

/mnt and /media exist for reasons. And root_squash and ... Why no, I've NEVER accidentally deleted whole file systems, I have completely earned superiority here. Delete /proc and /dev on a running server. Thankfully not really disastrous but damn if people don't notice right away. Thanks for the tmpwatch info btw.

You can't delete /proc, it is a pseudo filesystem the kernel creates. They aren't really files. Deleting /dev is a real pita however

With udev deleting /dev shouldn't be a problem I thought?

Re: Devops Horror Stories

#88

Post by scootklein »

redbad wrote: post #86

The first two stories are notable in how they reflect the terrible practices of the teller. "Our distributed application produces the same type of error after the same period of time in totally different data centers. We have no idea why, but moving data centers seems to help, so we just keep doing it. #YOLO" "We've built a product on a data store and library we don't understand even the highest-level constraints of.…

your first characterization seems incorrect (did you read the story? it wasn't application errors), and your second characterization is hyperbolic at best. calling it a high-level constraint doesn't mean it's common, nor obvious.

calling them "terrible practices" is redundant, all devops horror stories can be characterized as exposing terrible practices if you're simply looking at the post-hoc view. it's a feature, not a bug, to make light of them. they're laughed about, but with the intent that they're not made again.

Re: Devops Horror Stories

#89

Post by mikko-apo »

rdoherty wrote: post #28
danudey wrote: post #10

Earlier quoted context omitted.

I built a system where our developers can do instant deployments of any of our software packages (and instant point-in-time rollbacks), and then do zero-downtime restarts of services. Now we deploy dozens of times a day and I never get called on a Friday night because someone did something stupid. Edit: I do get called when I did something stupid and it broke the deployment system. But that's gotten much rarer lately…

Would love to hear more about this system, sounds really cool!

I've built a similar system, around 5 year ago. Users were able to deploy any version to any cluster from a nice UI. Basically you could select any version, click "Install" button and follow the logs in real time.

Behind the scenes it was a decentralized continuous delivery system. Very cool stuff, highly automated. Reduced a lot of work and sped up development cycles from months to minutes. Served quite a large software development organization (1000+). I think we had 1000 servers in 5 datacenters around the globe.

Nowdays I'm working on an open source version of that system, it's still missing few critical features but hopefully I'll get the first release out next spring.

btw, I'm looking for projects/contacts that would be interested in trying out how the system would fit their needs.

Re: Devops Horror Stories

#90

Post by existencebox »

mikko-apo wrote: post #89
rdoherty wrote: post #28

Earlier quoted context omitted.

Would love to hear more about this system, sounds really cool!

I've built a similar system, around 5 year ago. Users were able to deploy any version to any cluster from a nice UI. Basically you could select any version, click "Install" button and follow the logs in real time. Behind the scenes it was a decentralized continuous delivery system. Very cool stuff, highly automated. Reduced a lot of work and sped up development cycles from months to minutes. Served quite a large soft…

Hey Mikko; I'm a sysadmin at a research university, and I'd be very curious in at least "picking your brain" about your tool. I can't make any promises about actual usage, but I always love to see a novel approaches to relevant problems.

Do you have any sort of github/project page?

Post reply on HN