Having a(n accurate) service graph of all your (internal and external) dependencies is a game changer in troubleshooting issues like this.
The anatomy of a 2AM mental breakdown
21–30 of 286 posts
Re: The anatomy of a 2AM mental breakdown
#22I heard a sleep expert say that during the night your logic and reasoning abilities are greatly reduced. I think it was in relation to dreaming, you don't want to apply much logic to that stuff. That's why trying to solve problems in the middle of the night just ends up in stress.
Funny enough, I almost exclusively do my best problem solving at night. I'll wake up with solutions, or I'll stay awake and get things done. I'd say 90% of my Master's work was done between midnight and 3am. But indeed, regardless of time of day, if I just wake up, or am woken up, I'm basically a big, dangerous toddler when it comes to problem solving. Context and nuance is important, of course. We're all so differen…
That often happened for me in grad school as well. Generally the questions I had trouble with on a take home exam would yield to late night inspiration. And if they didn't yield by a semi-reasonable time, I would go to bed and many times, as I was drifting off, an insight would come to me. One memorable time though, that didn't happen and I woke up several times in the middle of the night from stress dreams where I was trying to solve the problem. And when I thought about the dream, nothing I had been doing in it made any logical sense to actually help me with a solution. Fortunately I woke up early and was able to figure it out in the morning. It was not a very restful night of sleep though.
Re: The anatomy of a 2AM mental breakdown
#23Great post, but kind of buries the lede: PostHog is having a CrowdStrike moment.
Re: The anatomy of a 2AM mental breakdown
#24Ha, that was a stressful yet funny read. The self flagellation bit hits too close to home though. I run a somewhat successful iOS/MacOS app and pushed a release that completely broke about 350k+ installations. Not entirely my fault but doesn't matter as it's my product. The cold sweats and shame I felt, man... Plus it's on the App Store so there's the review process to deal with which extends the timeline for a fix.…
My first employer as a developer, due to their incompetence, not intelligence, ‘let’ me break our customers’ shit from a young age. As I’ve progressed through my career, and through my transition to leadership, I’ve realised that it was a very valuable experience. I may have stressed about it in the past, but those memories are too distant for me to even reach now. I certainly don’t stress about it now.
I’ll, maybe controversially, sometimes allow my (early-career) team members to break prod, if I can see it happening ahead of time, but am confident that we’ll be able to recover quickly. It’s common knowledge that being given room to fail is important. But many leaders draw the line at failures that actually hit customers. If one finds oneself in the very common and very fortunate position to be building software that isn’t in charge of landing planes or anything similarly important, they should definitely let their team experience prod failures, even if it’s at the expense of so-and-so from Spokane Washington not being able to use their product for a few minutes.
Re: The anatomy of a 2AM mental breakdown
#25All monitoring comes at a cost and adds complexity. I wish people realized that, I struggle with this in my own team, we keep adding layers upon layer of monitoring, metrics etc.
I’ve definitely come across this genre of issue on many sites in the past when checking out the console when a site is broken. Page is just a blank white screen? Oh, looks like the render function was placed after the init for some 3rd party user monitoring, which crashed because the script didn’t load properly. “Complete Checkout” button just does nothing at all? Oh, looks like the code to take my money runs in a callback to some analytics script that my ad blocker blocked. Oops.
Re: The anatomy of a 2AM mental breakdown
#26Re: The anatomy of a 2AM mental breakdown
#27"Maybe PostHog, I have the api_key blanked out locally to reduce costs"
Come on, if POST requests work locally and not on PROD, isn't this an obvious place to start?
Re: The anatomy of a 2AM mental breakdown
#28[flagged]
Thank you for this. The article started interesting, but then devolved in boring all caps recap of panic thoughts which killed its momentum. I was still interested in the origin of the bug though.
It's not your style ; I still found it an entertaining read that conveyed well the prolonged stress the author was feeling.
Re: The anatomy of a 2AM mental breakdown
#29Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…
And ultimately you carry none of the risk. It's not your company, and the company can (and will) cut you off at a random time. Unless it is your company :)
Re: The anatomy of a 2AM mental breakdown
#30Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…
And ultimately you carry none of the risk. It's not your company, and the company can (and will) cut you off at a random time. Unless it is your company :)
This is the entire risk: being fired and financially in trouble. I spent 5 years eliminating that risk. Now I don't give a single fuck. There is no fear. They get better, more rational work and I have security.