Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devops Horror Stories

statuspage.io

Post reply on HN
Print view View original

61–70 of 102 posts

Re: Devops Horror Stories

#61

Post by peterwwillis »

ecopoesis wrote: post #24
0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

> It's only bad when you have shit developers or shit processes that don't catch broken code.

Or when the bug is only triggered in specific user profiles.

Or when all the devs went on a retreat in the mountains with no cell service.

Or when a dev makes a mistake (which we know never happens to even the best devs)

Or when the only developer that knows which one of the 1000 changes that were pushed could be the one breaking, turned his phone off.

Or when a flaw is discovered in the process for the first time (which we know never happens because everyone's process is perfect, until it isn't)

Or how change management's requirement that the fix be tested and verified by all affected teams might have people staying a few hours after 5pm on a Friday when they just want to get their weekend started.

Or how 10 different people from 10 different teams might need to be called and kept to work until 2am because the change can't be pulled because the database was already modified and the old client data is already expired from cache and a refresh would destroy the frontend servers.

Or another reason.

Re: Devops Horror Stories

#62

Post by mst »

cpt1138 wrote: post #57

(worst) update table set column = 'blah' WITHOUT a where clause (thank god for backups) (2nd worst) delete from table where created Lesson learned, always backup and write the WHERE clause first

It is possible to tell psql to always issue an implicit BEGIN so you also have to COMMIT before your change becomes permanent.

This has saved me from paying the price for that particular class of mistake on a number of occasions.

Re: Devops Horror Stories

#63

Post by brazzy »

ecopoesis wrote: post #24
0xbadcafebee wrote: post #8

My Devops horror stories, one sentence each: - Somebody deployed new features on a Friday at 5pm. - Fifteen hundred machines running mod_perl. - Supporting Oracle - TWICE. - It turns out your entire infrastructure is dependent on a single 8U Sun Solaris machine from 15 years ago, and nobody knows where it is. - Troubleshooting a bug in a site, view source.... and see SQL in the JS.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

Yeah, because all problems are foreseeable and only ever caused by crap code... right.

No matter how great your processes and your code are, no test can catch everything that can go wrong in a live environment, and doubly not if your system interfaces with anything third party.

Re: Devops Horror Stories

#64

Post by cstrat »

ch4ch4 wrote: post #3

The customer.io story seems like a great example of why NOT to use budget providers like OVH and Hetzner for mission-critical applications. You get what you pay for.

Yeah they had a terrible experience!

The idea of using DC hosting providers is because the uptime, environment and usually strong network connectivity. Given the poor run they had I think the developer would have had a better uptime if he had hosted this at home on a server.

Re: Devops Horror Stories

#65

Post by nailer »

zwp wrote: post #32
nailer wrote: post #20

Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.

So tmpwatch traverses filesystems. Is that a bug? (thinks: "-xdev"). Edit: Reading tmpwatch.c: "Try hard not to go onto a different device". Perhaps old bug?

Actually yes, I think it was patched to not traverse file systems afterward. It was around 2005 when this happened.

Re: Devops Horror Stories

#66

Post by illicium »

aaronem wrote: post #49
mkramlich wrote: post #36

Earlier quoted context omitted.

> - Troubleshooting a bug in a site, view source.... and see SQL in the JS. this is why I refuse to do "View Source" on the HealthCare.gov website. I'm afraid of what I might see.

I'm just gonna leave this right here: if ('en' === 'en') { $('#desktop-nav .middle').append(' My Profile '); $('.mobile-nav-right').append(' My Profile '); } else { $('#desktop-nav .middle').append(' My Profile '); $('.mobile-nav-right').append(' My Profile '); }

Ah, now I see why it has 500 million lines of source code.

Re: Devops Horror Stories

#68

Post by ryan-allen »

peterwwillis wrote: post #61
ecopoesis wrote: post #24

Earlier quoted context omitted.

I really hate the idea that deploying on a Friday afternoon is a bad idea. It's only bad when you have shit developers or shit processes that don't catch broken code. Personally, I think it's better to release at 5pm on a Friday. Once people stay late a few times to fix their broken shit they'll be smarter about not checking in crap.

> It's only bad when you have shit developers or shit processes that don't catch broken code. Or when the bug is only triggered in specific user profiles. Or when all the devs went on a retreat in the mountains with no cell service. Or when a dev makes a mistake (which we know never happens to even the best devs) Or when the only developer that knows which one of the 1000 changes that were pushed could be the one bre…

Yes, this! "Good code" and a CI box and deployment automation and some chef recipies don't spell ultimate success.

It drives me nuts when people tell me off for saying 'yeah yeah, no, automating our entire infrastructure of 5 servers isn't really worth it right now', like I'm some unprofessional bozo.

I pretty much have experience with all but one or two of your suggested scenarios, and by now I have no patience for annoying software developers who think that using chef or puppet somehow sufficiently embiggens them to run ops on their own (of course dev ops is almost a political assault on existsing ops guys, not merely a nice new solution to existing problems).

Sigh. This is why I don't work on teams these days (if I can help it).

EDIT: Though I also agree with the sub-parent, that deploying on 5pm is fine in certain teams and certain projects, the most important thing is are the guys pushing to do the deploy going to own the deployment? Are they going to hang around for another 60 minutes to check everything is OK? Are they going to be available at 10pm or on Saturday if something goes wrong and are they going to own it? If the answer is no, then nope, don't do it.

Re: Devops Horror Stories

#69

Post by mkramlich »

illicium wrote: post #66
aaronem wrote: post #49

Earlier quoted context omitted.

I'm just gonna leave this right here: if ('en' === 'en') { $('#desktop-nav .middle').append(' My Profile '); $('.mobile-nav-right').append(' My Profile '); } else { $('#desktop-nav .middle').append(' My Profile '); $('.mobile-nav-right').append(' My Profile '); }

Ah, now I see why it has 500 million lines of source code.

and costs 200M$+

because, you know, static HTML and some CRUD/lookup logic behind-the-scenes is just that hard

Re: Devops Horror Stories

#70

Post by SEJeff »

mhurron wrote: post #40
nailer wrote: post #20

Temporarily mounted an NFS volume to a folder under /tmp. Forgot about tmpwatch, a default entry in the RHEL cron table to clear out old temp files. 4AM the next morning, recursive deletion on anything wiuth a change time older than n days.

/mnt and /media exist for reasons. And root_squash and ... Why no, I've NEVER accidentally deleted whole file systems, I have completely earned superiority here. Delete /proc and /dev on a running server. Thankfully not really disastrous but damn if people don't notice right away. Thanks for the tmpwatch info btw.

You can't delete /proc, it is a pseudo filesystem the kernel creates. They aren't really files. Deleting /dev is a real pita however
Post reply on HN