Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

141–150 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#141

Post by JohnMakin »

sb8244 wrote: post #135
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

Some of the worst mistakes that I saw were from over-reaction in an active incident. One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done. I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on…

I have a similar mantra - "if you don't know why a fix worked, you may not have fixed it."

I'm willing to throw shit at the wall early in the triaging process, but only when they are low-impact and "simple" things. stuff like -

have we tried clearing cache?

have we checked DNS resolver for errors?

have we restarted the server?

etc. I try to find the "dumb" problems before jumping to some wild fix. In one of the worst outages of my career, a team I was working for tried to do a full database restore, which had never been done in production, based on a guess. At 3am on a saturday. I push back really hard at stuff like that.

Re: The anatomy of a 2AM mental breakdown

#142

Post by anonfordays »

timgl wrote: post #35
jonnycat wrote: post #23

Great post, but kind of buries the lede: PostHog is having a CrowdStrike moment.

PostHog cofounder here. This affected users that did not have a specific version of the JS library pinned and deployed a new version, or were using the snippet, and had network capture enabled, (a feature we introduced very recently and is only enabled on 3% of projects), and had recordings enabled on that particular session (for most customers, only a small percentage of sessions are recorded due to sampling or bill…

>This affected users that did not have a specific version of the JS library pinned and deployed a new version

Par for the course honestly. The amount of garbage that gets called "production" these days is mindboggling. No blue/green or canary deployments, shipping code that has nothing pinned, no clear rollback, etc. This is what happens when anyone can become an EngineerTM after a two week JavaScript boot camp.

Re: The anatomy of a 2AM mental breakdown

#143

Post by Vegenoid »

caseyy wrote: post #81
agumonkey wrote: post #45

Earlier quoted context omitted.

After a few times I realized that everytime I step into blind-ish panic over a mistake, I should get as far as possible from the machines. Otherwise I try one bad idea after the other and create an actual catastrophic failure.

What is this response called? I see it in many people. It’s like trying to catch a boiling mug of coffee when it falls — panic and not thinking leads to severe consequences; inaction would be better. It seems to relate to fight or flight, as opposed to freezing. All three are not great and can significantly escalate a bad situation.

I think calling it a panic response is appropriate. To me, that evokes a person doing irrational things to try and fix a sudden and distressing situation.

Re: The anatomy of a 2AM mental breakdown

#144

Post by hibikir »

JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

My proudest SRE moment was when, in a team that dealt with a key system where seconds meant millions, I was able to find a precursor to the problem, so instead of having to intervene on a page in 3 minutes before disaster, the alarm now had 20 minutes of headroom.

We all slept so much better after that

Re: The anatomy of a 2AM mental breakdown

#145

Post by janalsncm »

photonthug wrote: post #124
vlunkr wrote: post #98

Earlier quoted context omitted.

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

Great point. I honestly can't remember the last time that I've had a smooth experience with any kind of business or commercial thing, even when I'm the one paying for services. Bank declines transactions randomly when balance is sufficient, on vendors that have been used before. Orders of equipment over $10k are days late, no explanation provided, no number to speak to a human, just ineffective support robots. Planni…

> Orders of equipment over $10k are days late, no explanation provided, no number to speak to a human, just ineffective support robots.

I would be furious. If I spend $10k I want to be speaking with a human if something goes wrong. I feel like in the US we have given up on decent customer experiences.

Re: The anatomy of a 2AM mental breakdown

#147

Post by gcommer »

I know solo projects always have an infinite list of "nice to haves". But personally I never skimp on vendoring dependencies.

In my experience, not vendoring has _always_ led to breakages that are hard to debug and fix.

Meanwhile, vendoring is quite easy nowadays. Every reasonable package manager, and even npm, can do this near-trivially.

Re: The anatomy of a 2AM mental breakdown

#148

Post by vlunkr »

Alupis wrote: post #113
vlunkr wrote: post #98

Earlier quoted context omitted.

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

The obvious difference is no matter how long the county or city takes to reopen that road - you'll go right back to using it because you don't have an alternate choice. For a website - particularly ecommerce websites - you have many choices. Rarely is a product only sold on a particular website. Being down when someone is trying to place an order can and does result in losing the sale, and potentially the customer fo…

That really depends on what type of service we're talking about. Yeah, if you're doing really high volume sales on products that can be purchased elsewhere for close to the same price and shipping options, then uptime is critical. That's not the situation most of us are in.

Re: The anatomy of a 2AM mental breakdown

#149

Post by toomuchtodo »

fragmede wrote: post #121
prewett wrote: post #102

Earlier quoted context omitted.

What is your $2m generating? For instance, if you invested in high quality dividend stocks, you can get 3% and it should generally grow about what GDP does (which should also include inflation effects). So that's $60k/year, or after 15% dividend tax + let's say 5% California state tax, for a net of $48k. If you're in somewhere rent-controlled, seems like even with a family you're not going to on the streets starving.…

if you're only getting 3% with dividends, can I interest you with an HYSA paying 5%?

10 year US treasuries my good friend, no state or local income taxes and HYSA rates will decline as the Fed lowers the federal funds rate.

3% is on par for SCHD, which is a popular ETF dividend fund (that provides income, but also growth). Different risk profile than treasuries though.

https://finance.yahoo.com/quote/SCHD/

Re: The anatomy of a 2AM mental breakdown

#150

Post by kridsdale3 »

fragmede wrote: post #121
prewett wrote: post #102

Earlier quoted context omitted.

What is your $2m generating? For instance, if you invested in high quality dividend stocks, you can get 3% and it should generally grow about what GDP does (which should also include inflation effects). So that's $60k/year, or after 15% dividend tax + let's say 5% California state tax, for a net of $48k. If you're in somewhere rent-controlled, seems like even with a family you're not going to on the streets starving.…

if you're only getting 3% with dividends, can I interest you with an HYSA paying 5%?

3% comes from traditional FIRE texts which find it to be the withdrawal rate that would survive all historic recessions.
Post reply on HN