Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

201–210 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#201

Post by ChrisMarshallNY »

sb8244 wrote: post #135
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

Some of the worst mistakes that I saw were from over-reaction in an active incident. One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done. I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on…

> One of my programming mantras is "no black magic."

This. * 1000

I am very grateful that my earliest training was as an RF bench technician. It taught me how to find and fix really weird problems, without freaking out.

However, it does appear that this particular event may have been caused by reliance on a dependency that may not have merited that reliance.

I'm really, really careful about dependencies. That's an attitude that wins me few friends, in this crowd, but it's been a long time, since I've had 2AM freakouts.

Re: The anatomy of a 2AM mental breakdown

#202

Post by nozzlegear »

supportengineer wrote: post #158
Der_Einzige wrote: post #122

Earlier quoted context omitted.

One thing I did not expect about working in enterprise tech companies was the amount of coworkers who popped Xanax regularly. I've been an anxiety basket case my whole life and the idea that an addictive pill could take it all away terrifies me - I'd be hooked for life.

It's hard to compete against the people who are "enhanced" with drugs.

I used to occasionally take Xanax for [health] anxiety myself. It doesn't have that kind of effect, it's like a sedative.

Re: The anatomy of a 2AM mental breakdown

#203

Post by c0balt »

pknomad wrote: post #182
DowagerDave wrote: post #176

Earlier quoted context omitted.

>> One of my VP's (one of the coolest dude I know) would often say "slow is smooth and smooth is fast". Apparently you worked for a Navy Seal, so yeah, likely a very cool dude :)

How did you know?

Well known quote, that is often promoted as being used by the Navy Seals. Enter it in your favorite search machine to learn more about the background of the phrase.

Re: The anatomy of a 2AM mental breakdown

#204

Post by coryrc »

Alupis wrote: post #194
lovethevoid wrote: post #190

Earlier quoted context omitted.

>Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. That's not true. You don't have to manufacture your own goods to not compete with others at a commodity level. It's quite literally why branding is so important. But I do find it funny you wound up doing exactly what the…

> It's quite literally why branding is so important. Most of the goods you purchase are from brands not owned by the retail website. Pick a hobby and consider the prominent brands, then consider the prominent websites people in that hobby often patronize - they are very often not the same. > and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while Because it kind…

I'm about to order some electric bike motors. Probably from ebikes.ca. If their website goes down I'll just wait until it comes back up.

If it was groceries, I'd go to the store.

Unless in-person-purchasing is an option, I'm just going to wait for my preferred vendor. My account's already set up and clearly they have what I want.

Re: The anatomy of a 2AM mental breakdown

#205

Post by JohnMakin »

chasd00 wrote: post #195
JohnMakin wrote: post #62

Earlier quoted context omitted.

One of the funnier situations (funny now, wasn't so much the first time I saw this) that I run into at new gigs or contracts is when a business has absolutely zero monitoring or alerting. Go into their backend, it's predictably a dumpster fire. Start plugging in monitoring and the business realizes everything is on fire and PANICS. It's very difficult to explain to someone who definitely doesn't want to hear that it'…

> It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time. if no one noticed then can you really say it was broken?

Depends - IME yes, lol. Stuff like email campaigns being broken is super hard for the business to detect, as is a lot of random customer facing stuff that doesn't directly drive revenue. Stuff that isn't a total outage and a degradation will often be tolerated or unnoticed for quite a long time til it actually blows up.

My favorite anti-pattern ever that happens here is when you notice a bug that's existed for a long time, you fix it, then some other part of the system that had adapted and expected the behavior over the years breaks because of how they were handling the bug no longer works. Then of course the business comes to you like "why did you break this?"

Re: The anatomy of a 2AM mental breakdown

#206

Post by Octabrain »

I can relate to this. I used to be on call for many years and honestly, it destroyed my mental health. In the last company I did it, it felt like falling in a meat grinder for a week. I remember once spending a whole weekend giving support on an bug that was introduced by a recent release. 72 hours of working non stop. Because of that among others, I got a severe burnt out that took me to the deepest dark place I've ever been.

To this day, I simply refuse to do on call. There's no enough money you can pay me that would make me to suffer that again.

PS: Fuck you, Rackspace.

Re: The anatomy of a 2AM mental breakdown

#207

Post by riiii »

SketchySeaBeast wrote: post #7
riiii wrote: post #4

I heard a sleep expert say that during the night your logic and reasoning abilities are greatly reduced. I think it was in relation to dreaming, you don't want to apply much logic to that stuff. That's why trying to solve problems in the middle of the night just ends up in stress.

It also makes sense why I'll wake up in the middle of the night terrified of things that don't bother me during the day. My fears all are much more immediate in the witching hour, and I can't talk them away.

I'm the same. The only thing that has a chance of working is to remind myself that it's the middle of the night and however terrifying this is I'll deal with it in the morning.

Re: The anatomy of a 2AM mental breakdown

#208

Post by silverquiet »

zappchance wrote: post #72
mtnviewer wrote: post #46

Earlier quoted context omitted.

How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.

If you're scared about your financial situation with over a $2M net worth either you are prone to paranoia or you're living well above your means. I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors". My family and friends do okay with a net worth way below that, in the Bay Area. I've worked minimum wage jobs here in the bay, and had co-workers actually struggling to make…

Try having a chronic health condition. My ability to continue walking will at some point depend on paying doctors to replace part of my skeleton; that doesn't come cheap in America.

Re: The anatomy of a 2AM mental breakdown

#209

Post by adamredwoods »

sltkr wrote: post #133
adamredwoods wrote: post #91

So Zarar needs to keep the localdev as close to prod as possible, or have a separate pre-prod environment that can run integration tests to catch vital function disruptions.

Your advice is useful for detecting bugs in the code that you release, before your push it to production, but it would not have helped here, because the bug was in Posthog code that was pushed to production asynchronously. In fact, it would have made debugging this particular issue harder, because the difference in Posthog configuration between dev and prod is what clued the author in on Posthog causing the problem.…

Ah, thanks, I didn't know it was a live dependency.

>> Instead, pin all dependencies to fixed versions and host them yourself.

Indeed.

Re: The anatomy of a 2AM mental breakdown

#210

Post by tristor »

During the mid-early part of my Ops/SRE career I had a senior who was a mentor to me. I noticed as we dealt with outage after outage he was always calm, cool, and collected, so one day I asked him ", how do you always stay so calm when everything is down?" That's when I found that before he'd been in tech, he'd be a Force Recon Marine and had been deployed. His answer was "Nothing we do here is life or death, and nobody is shooting at me. It'll be alright, whatever it is."

While I have never experienced anything similar myself, it really helped me to put things in perspective. Since then, I've worked on some critical systems that were actually life or death, but I no longer do. For the /vast/ majority of technology systems, nobody will die if you let the outage last just a few hours longer. The worst case scenario is a financial cost to the company who employs you, which might be your own company. Smart companies de-risk this by getting various forms of business insurance, and that should include you if it's your own company.

So, do everything you can to fix the outage, but approach it with some perspective. It's not life or death, nobody is shooting at you.

Post reply on HN