This person's stress was caused by a single line of code in PostHog. This is the reversion: https://github.com/PostHog/posthog-js/pull/1371/commits/7598... Highlights two lessons. 1. If you ship it, you own it. Therefore the less you ship, the better. Keep dependencies to a minimum. 2. Keep non-critical things out of the critical path. A failing AC compressor should not prevent your engine from running. Very difficul…
The anatomy of a 2AM mental breakdown
71–80 of 286 posts
Re: The anatomy of a 2AM mental breakdown
#72Earlier quoted context omitted.
This is the entire risk: being fired and financially in trouble. I spent 5 years eliminating that risk. Now I don't give a single fuck. There is no fear. They get better, more rational work and I have security.
How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.
I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors". My family and friends do okay with a net worth way below that, in the Bay Area. I've worked minimum wage jobs here in the bay, and had co-workers actually struggling to make it to the next paycheck. It really gets on my nerves when someone in software chimes in with "we're all struggling"—No, we are not all struggling on the same level. Guess how much the person making your fast food or working at your local grocery store is making... And think about how they still live in the same city as you.
"$2M isn't that much". Man, I can't imagine what it takes to come to that conclusion.
Re: The anatomy of a 2AM mental breakdown
#73I heard a sleep expert say that during the night your logic and reasoning abilities are greatly reduced. I think it was in relation to dreaming, you don't want to apply much logic to that stuff. That's why trying to solve problems in the middle of the night just ends up in stress.
Re: The anatomy of a 2AM mental breakdown
#74Earlier quoted context omitted.
This is the entire risk: being fired and financially in trouble. I spent 5 years eliminating that risk. Now I don't give a single fuck. There is no fear. They get better, more rational work and I have security.
How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.
My net worth is less than yours. I could live on your interest fine and just retire.
Re: The anatomy of a 2AM mental breakdown
#75For what it's worth, I'm not sure this is a mental breakdown, and it might give the wrong impression to people who do legitimately suffer breakdowns related to stress about tech. For me it's only happened once. It was an anxiety attack, and I'm very lucky my wife was there to talk me through it and help me understand what was happening. She's had them many times, but it was my first (and thankfully only). It turns ou…
Re: The anatomy of a 2AM mental breakdown
#76I heard a sleep expert say that during the night your logic and reasoning abilities are greatly reduced. I think it was in relation to dreaming, you don't want to apply much logic to that stuff. That's why trying to solve problems in the middle of the night just ends up in stress.
https://www.faa.gov/documentLibrary/media/Advisory_Circular/...
Re: The anatomy of a 2AM mental breakdown
#77Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…
Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…
Hmm… interesting thought. It’s interesting how labeling something a terror attack makes the public lose critical thinking and act incoherently for a bit. It’s like a stunning shock response. The very same this author describes.
Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode?
People in customer service know clients definitely do.
It is interesting how these concepts connect.
Re: The anatomy of a 2AM mental breakdown
#78Earlier quoted context omitted.
PostHog cofounder here. This affected users that did not have a specific version of the JS library pinned and deployed a new version, or were using the snippet, and had network capture enabled, (a feature we introduced very recently and is only enabled on 3% of projects), and had recordings enabled on that particular session (for most customers, only a small percentage of sessions are recorded due to sampling or bill…
You're trying to phrase this as if those conditions make it any less bad, but they don't. This affected users that were using the latest version and used... features? Give me a break. Every product has bugs, but trying to downplay the issue after you've just read a distressed user of yours struggle with it is definitely not what you should be doing.
On the other hand the author of the article did the exact same thing. They either pushed a release without testing, or they automatically just pull in the latest version of an external library, without any testing or verification. Now I lean towards this being the latter, as if they pushed a release and then the site broke, they would have considered a rollback. Kinda hard to blame others for failing to do testing that you also didn't do.
Edit: So others have pointed out that PostHog will just pull down the latest version on it's own, unless you actively disable that feature. That seems like a brave move.
Re: The anatomy of a 2AM mental breakdown
#79From the PR: > fetch() broken on August 19: TypeError: ... Not broken at this version, broken on August 19. This is why I'm terrified of putting anything on the web. It is a dark scary place where runtime dependency on servers that you don't control is considered normal. One day I'll start my own p2p thing with just a bunch of TUI's and I'll only manage to convince six people to use it each for less than a month and…
Yeah, that is terrifying. In a nearby comment [1], a PostHog co-founder wrote this affected sites which "did not have a specific version of the JS library pinned and deployed a new version, or were using the snippet". I gather from that is it possible to pin the version, and this incident highlights the value of doing so.
Re: The anatomy of a 2AM mental breakdown
#80Ha, that was a stressful yet funny read. The self flagellation bit hits too close to home though. I run a somewhat successful iOS/MacOS app and pushed a release that completely broke about 350k+ installations. Not entirely my fault but doesn't matter as it's my product. The cold sweats and shame I felt, man... Plus it's on the App Store so there's the review process to deal with which extends the timeline for a fix.…
My first employer as a developer, due to their incompetence, not intelligence, ‘let’ me break our customers’ shit from a young age. As I’ve progressed through my career, and through my transition to leadership, I’ve realised that it was a very valuable experience. I may have stressed about it in the past, but those memories are too distant for me to even reach now. I certainly don’t stress about it now. I’ll, maybe c…
That was honestly one of the best things that could’ve happened to me and I still use the story for new hires to this day. It’s a very humbling reminder that nobody is perfect and we all make mistakes. And I’m still here, so it’s never the end of the world.