Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

131–140 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#131

Post by photonthug »

caseyy wrote: post #81
agumonkey wrote: post #45

Earlier quoted context omitted.

After a few times I realized that everytime I step into blind-ish panic over a mistake, I should get as far as possible from the machines. Otherwise I try one bad idea after the other and create an actual catastrophic failure.

What is this response called? I see it in many people. It’s like trying to catch a boiling mug of coffee when it falls — panic and not thinking leads to severe consequences; inaction would be better. It seems to relate to fight or flight, as opposed to freezing. All three are not great and can significantly escalate a bad situation.

This is a kind of error-cascade, common for lots of systems.

Maybe more specific though, situations where mistakes cause more dire mistakes are sometimes called "incident pits". Not unknown to engineers, but I think we ripped it off the scuba-divers, mountaineers, and other folks that are interested in studying safety-critical situations. https://www.outdoorswimmingsociety.com/the-incident-pit/

Re: The anatomy of a 2AM mental breakdown

#132

Post by ricardobeat »

roywiggins wrote: post #39
foodevl wrote: post #34

> This is no good. Let me just try reverting to a version from a month ago. Nothing. Three months ago? Nothing. Still failing. A year ago? Zilch. Reverting your own code, but still using a broken PostHog update from that same day? For me, the lesson is to make sure that I can revert everything, including dependencies.

It seems that PostHog just always loads the latest version of this piece of itself: https://github.com/PostHog/posthog/issues/24471#issuecomment... Though you can opt to bundle it yourself: https://github.com/PostHog/posthog/issues/24471#issuecomment...

Ouch. That just adds insult to injury.

Re: The anatomy of a 2AM mental breakdown

#133

Post by sltkr »

adamredwoods wrote: post #91

So Zarar needs to keep the localdev as close to prod as possible, or have a separate pre-prod environment that can run integration tests to catch vital function disruptions.

Your advice is useful for detecting bugs in the code that you release, before your push it to production, but it would not have helped here, because the bug was in Posthog code that was pushed to production asynchronously.

In fact, it would have made debugging this particular issue harder, because the difference in Posthog configuration between dev and prod is what clued the author in on Posthog causing the problem.

To avoid this kind of problem, the solution is to avoid “live” dependencies which can change in production without your testing. Instead, pin all dependencies to fixed versions and host them yourself.

Re: The anatomy of a 2AM mental breakdown

#134

Post by uaas »

thruway516 wrote: post #94
uaas wrote: post #21

Having a(n accurate) service graph of all your (internal and external) dependencies is a game changer in troubleshooting issues like this.

Do you need another dependency for that? A dependency to manage dependency hell.

Optimally, that’s already part of your observability stack, and might well be a built-in feature. In a certain scale/landscape it easily pays off.

Re: The anatomy of a 2AM mental breakdown

#135

Post by sb8244 »

JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

Some of the worst mistakes that I saw were from over-reaction in an active incident.

One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done.

I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on something, but as I think back I'm not sure that was ever the case in the end.

It was wild to see the top brass—normally very cool and composed—start suggesting arbitrary potential fixes during an incident.

Re: The anatomy of a 2AM mental breakdown

#136

Post by ataru »

bluepnume wrote: post #36

Looks like the bug was in a monkey-patched `window.fetch` https://github.com/PostHog/posthog-js/blob/759829c67fcb8720f... The biggest lesson here is, if you're writing a popular library that monkey-patches global functions, it needs to be really well tested. There's a difference between "I'll throw posthog calls in a try/catch just in case" and "With posthog I literally can't make fetch() calls with POST"

I think it worked as defined, it hogged the POST requests?

Re: The anatomy of a 2AM mental breakdown

#138

Post by ricardobeat »

bluepnume wrote: post #36

Looks like the bug was in a monkey-patched `window.fetch` https://github.com/PostHog/posthog-js/blob/759829c67fcb8720f... The biggest lesson here is, if you're writing a popular library that monkey-patches global functions, it needs to be really well tested. There's a difference between "I'll throw posthog calls in a try/catch just in case" and "With posthog I literally can't make fetch() calls with POST"

I was poking around to understand how this was not caught in a test - any ordinary fetch call could have triggered the error, and besides how poor coverage it has for all the ways `fetch` can be used, it seems excessive mocking may have played a part: https://github.com/PostHog/posthog-js/blob/main/src/__tests_...

The whole fetch and XHR functions are mocked and become no-ops, so obviously this won't catch any issues when interacting with the underlying (native or otherwise) libraries. They have Cypress set up so I don't see why you'd want to mock the browser APIs.

Re: The anatomy of a 2AM mental breakdown

#139

Post by vasco »

Alupis wrote: post #113
vlunkr wrote: post #98

Earlier quoted context omitted.

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

The obvious difference is no matter how long the county or city takes to reopen that road - you'll go right back to using it because you don't have an alternate choice. For a website - particularly ecommerce websites - you have many choices. Rarely is a product only sold on a particular website. Being down when someone is trying to place an order can and does result in losing the sale, and potentially the customer fo…

You are of course right, but in practice I've seen many more incidents created by doing changes to make things more robust than from simpler incremental product changes that usually are feature flagged and so on. At deeper levels usually there's less ability to do such containment (or is too expensive or takes too long or people are lazy) and so many times I wonder if it's better to do the trade-off or just keep things simple and eat only the "simple" sources of downtime to fix.

For example the classic thing is to always have minimum of 3 or 5 nodes for every stateful system. But in some companies, 1 hour of planned downtime on Monday mornings at 7AM to 8AM for operations and upgrades (which you only use when you need) + eating the times when the machine actually dies, would be less downtime than all the times you'd go down because of problems related to the very thing that should make you more robust. An incident here because replication lag was too high, an incident there because the quorum keeping system ran out of space etc and you're probably already behind. And then we have kubernetes. At some point it does make sense, when you have enough people and complexity to deal with this properly, but usually we do it too early.

Re: The anatomy of a 2AM mental breakdown

#140

Post by Vegenoid »

sb8244 wrote: post #135
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

Some of the worst mistakes that I saw were from over-reaction in an active incident. One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done. I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on…

It's a good reminder that if things get bad, people will just start burning things to try and appease the gods.
Post reply on HN