Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

161–170 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#161

Post by photonthug »

languagehacker wrote: post #116

I know that it's not a real post-mortem, but this is the opposite of what a good post-mortem looks like. It includes: * Blaming the tools (and the author) * Not focusing on facts in the timeline * Not considering improvements But that doesn't make for engaging content, right? > * At $TIME we observed HTTP POST calls failing > * At $TIME customers reported inability to make changes to ticket prices and promo codes > *…

The trouble with post-mortems is always the things that you can't say.

Are we going to talk about how someone wanted analytics everywhere even though it's expensive and they don't use it, that they chose the vendor and didn't provide time to evaluate it, that they wanted it added now-now-now to production even though it hadn't gone through dev, even though it was unplanned work in a busy sprint with other risky work, and that they wanted it on the critical path despite specific objections from some in engineering?

Not saying any of this was the case here, but this kind of thing happens frequently. While engineering is saying "let's use blameless post-mortems to highlight problems in the system" a lot of other people are thinking "now we get to see who could or couldn't fix a problem we caused, focus on that, and downplay our own role in causing trouble". Devs talking about CDNs and HTTP-POST and engineering processes are super helpful for steering the conversation away from broken business processes.

Easily 80% of PMs I've been involved in probably come down to "we had rules for moving tested code through dev/qa/prod, but we were forced to ignore the process we had agreed to because $PERSON / $DEPARTMENT said so". No one can actually talk about the elephant in the room though, because it's career suicide after you're branded as not a team-player. For all the kids out there.. it's really important to learn to be able to recognize the difference between a PM that's going to be an earnest effort to understand things, vs a PM that is going to be an unhelpful but necessary ritual of humiliation.

Re: The anatomy of a 2AM mental breakdown

#162

Post by maeil »

zappchance wrote: post #72
mtnviewer wrote: post #46

Earlier quoted context omitted.

How did you get to this point? I have $2M+ net worth, and while I live somewhere rent controlled, $2M isn't that much in the Bay Area. I'm still scared to lose my job.

If you're scared about your financial situation with over a $2M net worth either you are prone to paranoia or you're living well above your means. I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors". My family and friends do okay with a net worth way below that, in the Bay Area. I've worked minimum wage jobs here in the bay, and had co-workers actually struggling to make…

> I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors".

The amount of people I've seen on here saying "It's just not an option to quit Google/Meta no matter how awful they are, I have a mortgage to pay!" even when the tech job market was still red-hot is quite incredible.

Re: The anatomy of a 2AM mental breakdown

#163

Post by erik_seaberg »

hn83746 wrote: post #155
zappchance wrote: post #72

Earlier quoted context omitted.

If you're scared about your financial situation with over a $2M net worth either you are prone to paranoia or you're living well above your means. I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors". My family and friends do okay with a net worth way below that, in the Bay Area. I've worked minimum wage jobs here in the bay, and had co-workers actually struggling to make…

It makes especially... interesting reading to those of us from the global South. To make $2 mil at my current salary, I will have to work for about 100-150 years, saving every penny and spending nothing, and I make decent money by our standards. There's even no relatively safe way to make some smart investments (which I imagine is how most of this money was acquired) because of incessant economical crises and non-exi…

(Who are Eathlicans? I find no search results.)

Re: The anatomy of a 2AM mental breakdown

#164

Post by maeil »

RHSeeger wrote: post #110
zappchance wrote: post #72

Earlier quoted context omitted.

If you're scared about your financial situation with over a $2M net worth either you are prone to paranoia or you're living well above your means. I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors". My family and friends do okay with a net worth way below that, in the Bay Area. I've worked minimum wage jobs here in the bay, and had co-workers actually struggling to make…

> If you're scared about your financial situation with over a $2M net worth either you are prone to paranoia or you're living well above your means. As someone who was layed off and went without a job for 13 months (tech bubble), I am _always_ scared about my financial situation. Sure, if I lose my job, I have the resources to hold out while I find another, but - It _will_ impact my lifestyle, because I'll need to be…

You intentionally moved to that area, and it even sounds like you own real estate there.

Yes, it will impact your lifestyle, and you may have to be more careful about what money you spend, depending on how you're currently spending it. So what? 99% of people on earth live like that, and most do just fine.

The fact that having to live a non >$2M net worth lifestyle would make your family unhappy sounds concerning.

Re: The anatomy of a 2AM mental breakdown

#166

Post by mynameisvlad »

anonfordays wrote: post #142
timgl wrote: post #35

Earlier quoted context omitted.

PostHog cofounder here. This affected users that did not have a specific version of the JS library pinned and deployed a new version, or were using the snippet, and had network capture enabled, (a feature we introduced very recently and is only enabled on 3% of projects), and had recordings enabled on that particular session (for most customers, only a small percentage of sessions are recorded due to sampling or bill…

>This affected users that did not have a specific version of the JS library pinned and deployed a new version Par for the course honestly. The amount of garbage that gets called "production" these days is mindboggling. No blue/green or canary deployments, shipping code that has nothing pinned, no clear rollback, etc. This is what happens when anyone can become an EngineerTM after a two week JavaScript boot camp.

No, actually, it's because Posthog explicitly recommends that as the way to do it, makes their standard npm package unpinnable (as it will always lazy load the most recent version of its modules) and calls version pinning via npm as an "advanced" installation[1].

The ecosystem has plenty of versioning and best practices, but they do jack squat when you recommend to your customers to bypass them and trust that you'll never break your latest build.

[1] https://posthog.com/docs/libraries/js

Re: The anatomy of a 2AM mental breakdown

#167

Post by AlotOfReading »

robinhouston wrote: post #85

As others have mentioned, the bug that led to this late-night stress was a one-line change to the PostHog library[0]. I take this as a reminder of the importance of giving precise names to variables. The code res = await originalFetch(url, init) looks harmless enough. But in fact the `url` parameter is not necessarily a URL, as the TypeScript declaration makes clear: url: URL | RequestInfo The problem arises in the c…

The problem with linear/affine type systems is that they have an incredibly high barrier to entry. Just look at ownership semantics in something like Rust. They're not impenetrable (especially with experience), but they're severe enough to be the number one complaint for learners.

Re: The anatomy of a 2AM mental breakdown

#168

Post by cgh »

everdrive wrote: post #54
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…

> It's important not to sweat it, because a company does not care about your sleep or mental health. It will push you as far as it can.

One time a few years ago, a coworker called me to tell me the stress we were under was causing her hair to fall out. I mentioned I'd lost significant weight and wasn't sleeping well, especially since another coworker had quit and now I was shouldering his responsibilities (programming and managing a junior). She said she was going to quit. I wished her the best. Around six months later, I had to quit too. It took around a year for me to get back to normal sleep patterns and for my heart to stop racing in the night.

Oh yeah, another guy from that team had a "cardiac event" due to stress and had to take a month off.

Re: The anatomy of a 2AM mental breakdown

#169

Post by [deleted] »

justinclift wrote: post #89
sillysaurusx wrote: post #61

For what it's worth, I'm not sure this is a mental breakdown, and it might give the wrong impression to people who do legitimately suffer breakdowns related to stress about tech. For me it's only happened once. It was an anxiety attack, and I'm very lucky my wife was there to talk me through it and help me understand what was happening. She's had them many times, but it was my first (and thankfully only). It turns ou…

> Xanax is worth having on hand Doing a search online seems to indicate that Xanax can be addictive? https://www.drugs.com/xanax.html Doesn't seem to be the kind of thing to take lightly?

[deleted]

Re: The anatomy of a 2AM mental breakdown

#170

Post by sethammons »

ricardobeat wrote: post #138
bluepnume wrote: post #36

Looks like the bug was in a monkey-patched `window.fetch` https://github.com/PostHog/posthog-js/blob/759829c67fcb8720f... The biggest lesson here is, if you're writing a popular library that monkey-patches global functions, it needs to be really well tested. There's a difference between "I'll throw posthog calls in a try/catch just in case" and "With posthog I literally can't make fetch() calls with POST"

I was poking around to understand how this was not caught in a test - any ordinary fetch call could have triggered the error, and besides how poor coverage it has for all the ways `fetch` can be used, it seems excessive mocking may have played a part: https://github.com/PostHog/posthog-js/blob/main/src/__tests_... The whole fetch and XHR functions are mocked and become no-ops, so obviously this won't catch any issues…

I have seen so many mocked tests where you end up asserting the logic in the mock works; effectively testing 1=1.

The number of issues that can be prevented with an acceptance level test that has a user log in and do one simple interaction is amazing. Where I can convince the powers that be, PRs to main are gated by a build that runs, among others, that simple kind of AC test. If it was merged to main, you _know_ it will not totally break production.

We had regular outages with our internal emailing system at a small e-commerce shop. I stepped in and added one test that actually sent an email to a known sink that we could verify and had that test run pre-deploy. We went to zero email outages. Tests had the occasional flake that auto-retried. Also, if your acceptance tests are flaky, how do you know your software isn't? Bad excuse to avoid acceptance level testing

Post reply on HN