Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

31–40 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#31

Post by charles_f »

Such an entertaining read, conveyed very well the sense of stress and abandonment felt by the author. It adds to it that this was written fresh and right in the moment, and feels as an expiation.

I'm struggling to find the lesson to take out of that. Limit your dependencies? Have a safe mode that deactivates everything optional?

Re: The anatomy of a 2AM mental breakdown

#33

Post by sdenton4 »

JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

OTOH, a lot of people who really needed to be able to change their event pricing at 2am (+/- two timezones) on jumpcomedy.com were really let down. Probably some of them died.

Imagine how much more the damage would have been if someone was telling this solo developer to stop trying to make 'fetch' happen.

Re: The anatomy of a 2AM mental breakdown

#34

Post by foodevl »

> This is no good. Let me just try reverting to a version from a month ago. Nothing. Three months ago? Nothing. Still failing. A year ago? Zilch.

Reverting your own code, but still using a broken PostHog update from that same day? For me, the lesson is to make sure that I can revert everything, including dependencies.

Re: The anatomy of a 2AM mental breakdown

#35

Post by timgl »

jonnycat wrote: post #23

Great post, but kind of buries the lede: PostHog is having a CrowdStrike moment.

PostHog cofounder here. This affected users that did not have a specific version of the JS library pinned and deployed a new version, or were using the snippet, and had network capture enabled, (a feature we introduced very recently and is only enabled on 3% of projects), and had recordings enabled on that particular session (for most customers, only a small percentage of sessions are recorded due to sampling or billing limits)

This outage was definitely disruptive and we shouldn't have let this happen. We will be doing a full post mortem write up, but this affected a small percentage of our users, so the comparison with Crowdstrike isn't fair.

Re: The anatomy of a 2AM mental breakdown

#36

Post by bluepnume »

Looks like the bug was in a monkey-patched `window.fetch`

https://github.com/PostHog/posthog-js/blob/759829c67fcb8720f...

The biggest lesson here is, if you're writing a popular library that monkey-patches global functions, it needs to be really well tested.

There's a difference between "I'll throw posthog calls in a try/catch just in case" and "With posthog I literally can't make fetch() calls with POST"

Re: The anatomy of a 2AM mental breakdown

#38

Post by f1shy »

I would love to read from the author what are the lessonS learned.

Use better tools? Know better your tools? Know better how to debug? Add yet another tool to detect the error?

In all big companies where I worked, at the end of such an event, it boiled down to answer the 3 questions: - what happened?

- why did it happen?

- what do we do so it does not ever happen again?

Re: The anatomy of a 2AM mental breakdown

#39

Post by roywiggins »

foodevl wrote: post #34

> This is no good. Let me just try reverting to a version from a month ago. Nothing. Three months ago? Nothing. Still failing. A year ago? Zilch. Reverting your own code, but still using a broken PostHog update from that same day? For me, the lesson is to make sure that I can revert everything, including dependencies.

It seems that PostHog just always loads the latest version of this piece of itself:

https://github.com/PostHog/posthog/issues/24471#issuecomment...

Though you can opt to bundle it yourself:

https://github.com/PostHog/posthog/issues/24471#issuecomment...

Re: The anatomy of a 2AM mental breakdown

#40

Post by mrsilencedogood »

misja111 wrote: post #27

"Maybe PostHog, I have the api_key blanked out locally to reduce costs" Come on, if POST requests work locally and not on PROD, isn't this an obvious place to start?

The author mentioned significantly more notable prod/dev differences than the posthog API key, which I suspect is where they looked first and second. So no, not an obvious place to start.
Post reply on HN