Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

221–230 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#221

Post by groby_b »

vlunkr wrote: post #98
everdrive wrote: post #54

Earlier quoted context omitted.

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

The world would be a better place when airlines and ISPs worked that way too. (Roads work is usually communicated well in advance)

Downtime for these is critically impacting people, and you're damn right you should explain what went wrong and how you'll avoid this in the future. Because the only reasonable alternative is nationalizing them. "I dun fucked up, but meh" is not an acceptable attitude.

Re: The anatomy of a 2AM mental breakdown

#222

Post by 91bananas »

mcguire wrote: post #191
vlunkr wrote: post #98

Earlier quoted context omitted.

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

Are you assuming airline employees aren't scrambling when a flight is delayed? Or that road crews don't get called in the middle of the night? A minor power outage gets a crew called no matter what time it is; a major outage will bring in crews from different states, much less power companies.

But none of them are working in panic-mode as them doing this work is just an expected thing. As OP has pointed out, downtime in other industries is expected and agreed upon as OK. For some reason because it's on a computer that does not apply to our industry, we are to go in to panic-mode to fix, and write about how it will never happen again, stock prices fall, people lose their jobs.

Re: The anatomy of a 2AM mental breakdown

#223

Post by vlunkr »

groby_b wrote: post #221
vlunkr wrote: post #98

Earlier quoted context omitted.

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

The world would be a better place when airlines and ISPs worked that way too. (Roads work is usually communicated well in advance) Downtime for these is critically impacting people, and you're damn right you should explain what went wrong and how you'll avoid this in the future. Because the only reasonable alternative is nationalizing them. "I dun fucked up, but meh" is not an acceptable attitude.

> "I dun fucked up, but meh" is not an acceptable attitude.

I'm cool with it. I don't care that some hard drive went bad and the monitoring system didn't pick up on it because some queue was full. Who cares? If this isn't happening constantly, I assume they're going to learn their lesson.

Re: The anatomy of a 2AM mental breakdown

#224

Post by vundercind »

camdenreslink wrote: post #90
mtnviewer wrote: post #46

Earlier quoted context omitted.

How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.

With a $2m+ net worth you could move to 90% of the country, retire and live off of the interest.

To support a sibling post, depends very much on age and circumstances. 50, healthy, no wife or kids and no plan to have them? Might be fine retiring at $2m. 30 with a family? Ehhhh…

Likely-time-remaining is a huge factor in figuring out how much it’ll take to retire at a given level of risk (of running out of money too early).

Re: The anatomy of a 2AM mental breakdown

#225

Post by wonderwonder »

Nothing good comes from working at 2am. Company I worked at pushed a major waterfall upgrade one day. Completely tanked the database while attempting to update the schema.

I spent hours on a call with the clients sr. engineer and we eventually came up with a script to fix it. It was after midnight, my director said, good job, you are tired, I'll run the script, call it a night.

An hour later director ran the wrong script... and then called me.

Clients sr. engineer was legitimately flabbergasted, only time I have ever seen that word apply in real life.

Was a not good, very bad day.

Re: The anatomy of a 2AM mental breakdown

#226

Post by sleazebreeze »

flumpcakes wrote: post #120

I've been on my end of plenty of operational outages. I don't want to be harsh but this could have been written by one of my colleagues, the type of colleagues that I really wish I didn't work with. Console logging for hours ? Randomly disabling things? Sometimes when you feel "imposter syndrome" you shouldn't ignore it and maybe up your game a bit. In fact, I have dealt with an extremely similar situation where a bu…

Agree with everything you said.

And then I’d add: Start with reading the error message. In his panic state, he seems to have thought it was a red herring. Error messages are gold. It gives you a concrete thing to work backwards from.

Re: The anatomy of a 2AM mental breakdown

#227

Post by ryandrake »

Alupis wrote: post #160
vlunkr wrote: post #148

Earlier quoted context omitted.

That really depends on what type of service we're talking about. Yeah, if you're doing really high volume sales on products that can be purchased elsewhere for close to the same price and shipping options, then uptime is critical. That's not the situation most of us are in.

> That's not the situation most of us are in. From an ecommerce website's perspective - it is exactly the situation most of us are in. Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. In an age where more and more people don't even consider shopping anywhere but Amazon,…

Lots of things can be unacceptable, but not unacceptable enough to warrant being treated as a life-or-death emergency, where klaxons ring out and people get paged at 2AM. So your e-commerce website is down for a few hours. Fine. What's the worst thing that happens? You miss out on a small number of purchases, maybe a few customers are gone for good, and the shareholders reap a little less profit this quarter. Nobody died.

I think this is what OOP meant about "zero-downtime culture" being out of control. Unless you're a hospital keeping people on life support, I don't want to hear about zero downtime.

Re: The anatomy of a 2AM mental breakdown

#229

Post by ryandrake »

photonthug wrote: post #124
vlunkr wrote: post #98

Earlier quoted context omitted.

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

Great point. I honestly can't remember the last time that I've had a smooth experience with any kind of business or commercial thing, even when I'm the one paying for services. Bank declines transactions randomly when balance is sufficient, on vendors that have been used before. Orders of equipment over $10k are days late, no explanation provided, no number to speak to a human, just ineffective support robots. Planni…

Every company seems to just be phoning it in lately. The number of times I've ordered something from an online retailer, and either 1. the wrong thing arrived, 2. the thing arrived broken/doa, or 3. the thing never arrived, has at least tripled since say 5 years ago. And to a lot of companies' credit, their indifferent but friendly customer support tends to just say " just keep whatever we sent you and we'll refund you. You can try to order the thing again, and we'll maybe send you the right, working thing next time. Or maybe not. Who knows?"

Re: The anatomy of a 2AM mental breakdown

#230

Post by ryandrake »

burntwater wrote: post #217
burningChrome wrote: post #177

Earlier quoted context omitted.

>> Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems THIS. Worked for a huge corporation when I first started contracting as a developer. This was a daily occurrence. Someone would push some stuff to prod and it would break their very large e-commerce site. Sometimes it was minor stuff, other times it would cripple the…

>> At my previous job if something went down on our site, it was a four alarm fire, war room, and all hands on deck to get it resolved or heads would roll. It was so dysfunctional to work somewhere when something broke or stopped working, nobody was in any hurry to fix it. Several times I just thought, "Is this what they mean when they say "the inmates are running the asylum""? That's funny, because reading your firs…

Yea, nobody seems to be able to handle problems that require solid, but moderate, non-urgent effort. The problem is either a 0, where nobody even knows, let alone cares that it's happening or a 10, where it's treated as though everyone's chair is on fire and the company is losing $10M per millisecond.
Post reply on HN