Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

81–90 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#81

Post by caseyy »

agumonkey wrote: post #45
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

After a few times I realized that everytime I step into blind-ish panic over a mistake, I should get as far as possible from the machines. Otherwise I try one bad idea after the other and create an actual catastrophic failure.

What is this response called? I see it in many people. It’s like trying to catch a boiling mug of coffee when it falls — panic and not thinking leads to severe consequences; inaction would be better.

It seems to relate to fight or flight, as opposed to freezing. All three are not great and can significantly escalate a bad situation.

Re: The anatomy of a 2AM mental breakdown

#82

Post by amsterdorn »

foodevl wrote: post #34

> This is no good. Let me just try reverting to a version from a month ago. Nothing. Three months ago? Nothing. Still failing. A year ago? Zilch. Reverting your own code, but still using a broken PostHog update from that same day? For me, the lesson is to make sure that I can revert everything, including dependencies.

This is the key lesson. Your own deploys mean nothing if you link to another CDN for parts of your application.

You handled it well OP, the silver lining of incidents like this is the grab bag of valuable takeaways!

Re: The anatomy of a 2AM mental breakdown

#83

Post by simpaticoder »

M4v3R wrote: post #71
simpaticoder wrote: post #47

This person's stress was caused by a single line of code in PostHog. This is the reversion: https://github.com/PostHog/posthog-js/pull/1371/commits/7598... Highlights two lessons. 1. If you ship it, you own it. Therefore the less you ship, the better. Keep dependencies to a minimum. 2. Keep non-critical things out of the critical path. A failing AC compressor should not prevent your engine from running. Very difficul…

These are valuable lessons for sure, but then someone from the marketing comes and demands you add PostHog or any other tracking script to the site and won't take no for an answer.

Then communicate clearly the trade-off the project leader is making.

Re: The anatomy of a 2AM mental breakdown

#85

Post by robinhouston »

As others have mentioned, the bug that led to this late-night stress was a one-line change to the PostHog library[0].

I take this as a reminder of the importance of giving precise names to variables. The code

    res = await originalFetch(url, init)
looks harmless enough. But in fact the `url` parameter is not necessarily a URL, as the TypeScript declaration makes clear:

    url: URL | RequestInfo
The problem arises in the case where it is not a URL, but a RequestInfo object, which has been “used up” by the construction of the Request object earlier in the function implementation and cannot be used again here.

It would have been more difficult to overlook the problem with this change if the parameter were named something more precise such as `urlOrRequestInfo`.

(A much more speculative idea is the thought that it is possible to formalise the idea of a value being “used up” using linear types, derived from linear logic, so conceivably a suitable type system could prevent this class of bug.)

[0] https://github.com/PostHog/posthog-js/pull/1351/commits/2497...

Re: The anatomy of a 2AM mental breakdown

#86

Post by everdrive »

caseyy wrote: post #77
everdrive wrote: post #54

Earlier quoted context omitted.

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…

> It feels a bit like terrorism [..] It’s important not to sweat it Hmm… interesting thought. It’s interesting how labeling something a terror attack makes the public lose critical thinking and act incoherently for a bit. It’s like a stunning shock response. The very same this author describes. Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode? People i…

>Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode?

I'm sure some do. Much more often I observe that the managers themselves are just reacting to the incentive system in their environment -- and that those incentive systems are complex, not-widely-understood, and often built "accidentally." ie, companies are often accidentally incentivizing the wrong behaviors and are often powerless to prevent this.

Re: The anatomy of a 2AM mental breakdown

#87

Post by silverquiet »

caseyy wrote: post #81
agumonkey wrote: post #45

Earlier quoted context omitted.

After a few times I realized that everytime I step into blind-ish panic over a mistake, I should get as far as possible from the machines. Otherwise I try one bad idea after the other and create an actual catastrophic failure.

What is this response called? I see it in many people. It’s like trying to catch a boiling mug of coffee when it falls — panic and not thinking leads to severe consequences; inaction would be better. It seems to relate to fight or flight, as opposed to freezing. All three are not great and can significantly escalate a bad situation.

Action bias and it seems rather fundamental to human behavior. You don't want to be seen as doing nothing when there are problems, but of course with a complex system, it can be very difficult to diagnose an issue, so you tend to screw things up if you just rush in. I think my favorite saying in that instance is, "don't just do something, stand there".

Re: The anatomy of a 2AM mental breakdown

#88

Post by mynameisvlad »

sillysaurusx wrote: post #61

For what it's worth, I'm not sure this is a mental breakdown, and it might give the wrong impression to people who do legitimately suffer breakdowns related to stress about tech. For me it's only happened once. It was an anxiety attack, and I'm very lucky my wife was there to talk me through it and help me understand what was happening. She's had them many times, but it was my first (and thankfully only). It turns ou…

https://www.webmd.com/mental-health/signs-nervous-breakdown

Not all breakdowns come in the form of panic and anxiety attacks. Those are certainly a way that breakdowns can manifest, but it’s not the only way. Stress manifests in wildly different ways for different people and even stressors.

You weren’t in his head, experiencing what he was experiencing, so it’s pretty much impossible to “diagnose” from the outside. It certainly sounds like he was functionally paralyzed for hours, even if he didn’t have a full blown panic attack.

Re: The anatomy of a 2AM mental breakdown

#89

Post by justinclift »

sillysaurusx wrote: post #61

For what it's worth, I'm not sure this is a mental breakdown, and it might give the wrong impression to people who do legitimately suffer breakdowns related to stress about tech. For me it's only happened once. It was an anxiety attack, and I'm very lucky my wife was there to talk me through it and help me understand what was happening. She's had them many times, but it was my first (and thankfully only). It turns ou…

> Xanax is worth having on hand

Doing a search online seems to indicate that Xanax can be addictive?

https://www.drugs.com/xanax.html

Doesn't seem to be the kind of thing to take lightly?

Re: The anatomy of a 2AM mental breakdown

#90

Post by camdenreslink »

mtnviewer wrote: post #46
minkles wrote: post #30

Earlier quoted context omitted.

This is the entire risk: being fired and financially in trouble. I spent 5 years eliminating that risk. Now I don't give a single fuck. There is no fear. They get better, more rational work and I have security.

How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.

With a $2m+ net worth you could move to 90% of the country, retire and live off of the interest.
Post reply on HN