So Zarar needs to keep the localdev as close to prod as possible, or have a separate pre-prod environment that can run integration tests to catch vital function disruptions.
The anatomy of a 2AM mental breakdown
91–100 of 286 posts
Re: The anatomy of a 2AM mental breakdown
#92Earlier quoted context omitted.
PostHog cofounder here. This affected users that did not have a specific version of the JS library pinned and deployed a new version, or were using the snippet, and had network capture enabled, (a feature we introduced very recently and is only enabled on 3% of projects), and had recordings enabled on that particular session (for most customers, only a small percentage of sessions are recorded due to sampling or bill…
You're trying to phrase this as if those conditions make it any less bad, but they don't. This affected users that were using the latest version and used... features? Give me a break. Every product has bugs, but trying to downplay the issue after you've just read a distressed user of yours struggle with it is definitely not what you should be doing.
Yeah, honestly not a good look to come in and “well… actually”. It’s certainly far from a “crowdstrike moment” but tact is still needed when you’ve clearly affected multiple people and their customers with your bug.
Re: The anatomy of a 2AM mental breakdown
#93Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…
"Splashing my face and hands with extremely cold water is my trick."
Yes, literally cooling down. Combined with conscious breathing. And then go on doing, what you can do.
Re: The anatomy of a 2AM mental breakdown
#94Having a(n accurate) service graph of all your (internal and external) dependencies is a game changer in troubleshooting issues like this.
Do you need another dependency for that? A dependency to manage dependency hell.
Re: The anatomy of a 2AM mental breakdown
#95[deleted]
Re: The anatomy of a 2AM mental breakdown
#96Earlier quoted context omitted.
How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.
If you're scared about your financial situation with over a $2M net worth either you are prone to paranoia or you're living well above your means. I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors". My family and friends do okay with a net worth way below that, in the Bay Area. I've worked minimum wage jobs here in the bay, and had co-workers actually struggling to make…
I'm not one of these rich people that drives a gold-plated Lamborghini, like my obnoxious boss with his huge collection of supercars.
I just drive a regular, normal Lamborghini - like my father, and his father before him.
Re: The anatomy of a 2AM mental breakdown
#97Side loading 3rd party scripts in a critical path is asking for problems. Try https://partytown.builder.io/ runs 3rd party scripts like this in a web worker. I'm not sure it would help in this case. Maybe? Probably couldn't hurt to try.
Re: The anatomy of a 2AM mental breakdown
#98Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…
Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…
The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.
Re: The anatomy of a 2AM mental breakdown
#99Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…
Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…
There are even times to prefer to site being down to being up. If you've been hacked and the site is serving CSAM/malware/leaking PII maliciously, as the SRE on deck, your job is to keep it down!
Re: The anatomy of a 2AM mental breakdown
#100For what it's worth, I'm not sure this is a mental breakdown, and it might give the wrong impression to people who do legitimately suffer breakdowns related to stress about tech. For me it's only happened once. It was an anxiety attack, and I'm very lucky my wife was there to talk me through it and help me understand what was happening. She's had them many times, but it was my first (and thankfully only). It turns ou…
https://www.webmd.com/mental-health/signs-nervous-breakdown Not all breakdowns come in the form of panic and anxiety attacks. Those are certainly a way that breakdowns can manifest, but it’s not the only way. Stress manifests in wildly different ways for different people and even stressors. You weren’t in his head, experiencing what he was experiencing, so it’s pretty much impossible to “diagnose” from the outside. I…
That's a good point. I didn't mean to gatekeep, which is what I ended up doing. Thank you.
If he was functionally paralyzed for hours, then that absolutely qualifies. I was reading through it and thinking "If you can debug stuff, you're probably not having a mental breakdown" and wanted to highlight that some people do break down (which is why it's called a breakdown) and that it's ok.