Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devops Horror Stories

statuspage.io

Post reply on HN
Print view View original

91–100 of 102 posts

Re: Devops Horror Stories

#91

Post by mikko-apo »

existencebox wrote: post #90
mikko-apo wrote: post #89

Earlier quoted context omitted.

I've built a similar system, around 5 year ago. Users were able to deploy any version to any cluster from a nice UI. Basically you could select any version, click "Install" button and follow the logs in real time. Behind the scenes it was a decentralized continuous delivery system. Very cool stuff, highly automated. Reduced a lot of work and sped up development cycles from months to minutes. Served quite a large soft…

Hey Mikko; I'm a sysadmin at a research university, and I'd be very curious in at least "picking your brain" about your tool. I can't make any promises about actual usage, but I always love to see a novel approaches to relevant problems. Do you have any sort of github/project page?

Hi

Cool, sysadmin at a research university sounds like a nice position to be at.

Yes, it's already on github. Unfortunately, since it's missing those critical features it's not easy to see how the whole system is going to work. If you to talk just drop me an email at gmail. mikko.apo is the account.

Re: Devops Horror Stories

#93

Post by hga »

In an early phase of MIT's EECS transition from Multics (going away, Honeywell sucks) to UNIX(TM) on MicroVAX IIs, i.e. some users, but not as many as latter.

# kill % 1

Instead of %1. So I zapped the initializer, parent of everything else, logging everyone out without warning.

I had more than enough capital to avoid anything more than the deserved ribbing, but it was my Crowning Moment of Awesome devop lossage; harsh but minor screwups in the decade previous had trained me to be very careful.

I've avoided being handed the horrors of many other posters by primarily being a programmer. You full timers earn my respect.

ADDED: Ah, one big consequential goof, related to my not being a full time sysadmin but knowing more than anyone else in my startup. Buying a Cheswick and Bellovin style Gauntlet Firewall from TIS ... not realizing they'd just been bought by Network Associates, who promptly fired anyone who knew anything about supporting that product.... (At that time I didn't even know about iptable's predecessor, although given it was a Microsoft shop....)

I was fired from that job in part because I was the least worst sysadmin in the company, totally consumed with a big programming and database migration effort (Microsoft Jet -> DB2 -> DB2 on a real server), and gave opinions that others sometimes accepted and implemented without due diligence. E.g. I said "this is a competent ISP", not "you should also use their brand new email system" (which I didn't even know existed) ... visibility all the way up to the CEO is of course not always good....

Re: Devops Horror Stories

#94

Post by hga »

nailer wrote: post #60
mhurron wrote: post #40

Earlier quoted context omitted.

/mnt and /media exist for reasons. And root_squash and ... Why no, I've NEVER accidentally deleted whole file systems, I have completely earned superiority here. Delete /proc and /dev on a running server. Thankfully not really disastrous but damn if people don't notice right away. Thanks for the tmpwatch info btw.

Would have used /media but was thinking, if, say I forgot to unmount it or someone looked at a disk free or whatever, that it would be obvious that it was there temporarily. Obviously that was incorrect, but the reasoning was, I think, sound.

I like /mnt/scratch/ for that. If I was using systems with lots of others, I'd make it clearer with /mnt/tmp/.

Re: Devops Horror Stories

#95

Post by hga »

patmcguire wrote: post #46

At least Amazon doesn't lose your servers. http://www.informationweek.com/server-54-where-are-you/65055...

Eh, this tale of ultimate unattended service reminds me of my favorite Daniel Boone the frontiersman quote: "I can't say as ever I was lost, but I was bewildered once for three days."

Re: Devops Horror Stories

#96

Post by liquidcool »

ecopoesis wrote: post #24
0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

Maybe it's me, but I have no problem staying late on a Friday to fix my screw-up. However, I'm terrified of having to fix something Monday morning while everyone else is watching.

But the real reason we deploy weekday mornings is so everyone is on deck and we can get outside help if required. When I was doing system integration, the problem was never in my code, it was the vendor's. Testing can only get you so close to the real world.

Re: Devops Horror Stories

#97

Post by liquidcool »

Mine was simple: I did a middle-mouse-button paste of "init 6" into a root window of our main Solaris server that hosted about 100 users, mid-day. Boss shrugged it off, stuff happens.

But that's because it was properly configured so a reboot was smooth and didn't have any snags or affect other systems once back online. At another data center across the hall, if their main server needed to be rebooted (not accidentally!), it was 3 days of troubleshooting to get it back up. I learned that after the boss hired one of their admins - not surprisingly, a big mistake.

Re: Devops Horror Stories

#98

Post by linc01n »

0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

We won't deploy any code even after Wed afternoon. Don't get ourselves in any troubles.

Re: Devops Horror Stories

#100

Post by DCoder »

aaronem wrote: post #49
mkramlich wrote: post #36

Earlier quoted context omitted.

> - Troubleshooting a bug in a site, view source.... and see SQL in the JS. this is why I refuse to do "View Source" on the HealthCare.gov website. I'm afraid of what I might see.

I'm just gonna leave this right here: if ('en' === 'en') { $('#desktop-nav .middle').append(' My Profile '); $('.mobile-nav-right').append(' My Profile '); } else { $('#desktop-nav .middle').append(' My Profile '); $('.mobile-nav-right').append(' My Profile '); }

Seen this pattern many times before. One of those 'en' strings is the current user's language being written into the source, the other is hardcoded. If your server-side templating engine is impotent and only supports variable interpolation without conditionals, this approach is easier than pulling the right JS snippet from somewhere else.
Post reply on HN