Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The anatomy of a 2AM mental breakdown

zarar.dev

Post reply on HN
Print view View original

191–200 of 286 posts

Re: The anatomy of a 2AM mental breakdown

#191

Post by mcguire »

vlunkr wrote: post #98
everdrive wrote: post #54

Earlier quoted context omitted.

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…

The zero-downtime culture is pretty out of control. Your flight can be delayed for hours, your roads can be closed for weeks, your internet can be out for hours because someone cut a line, that's all ok, but someone can't get into your website for 3 minutes to check the status of their order and we all must lose our minds and publicly apologize and explain in detail what happened and why it will never happen again.

Are you assuming airline employees aren't scrambling when a flight is delayed? Or that road crews don't get called in the middle of the night? A minor power outage gets a crew called no matter what time it is; a major outage will bring in crews from different states, much less power companies.

Re: The anatomy of a 2AM mental breakdown

#192

Post by cma »

lukan wrote: post #93
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

"Splashing my face and hands with extremely cold water is my trick." Yes, literally cooling down. Combined with conscious breathing. And then go on doing, what you can do.

It's the mammalian diving reflex: https://en.wikipedia.org/wiki/Diving_reflex

Re: The anatomy of a 2AM mental breakdown

#194

Post by Alupis »

lovethevoid wrote: post #190
Alupis wrote: post #160

Earlier quoted context omitted.

> That's not the situation most of us are in. From an ecommerce website's perspective - it is exactly the situation most of us are in. Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. In an age where more and more people don't even consider shopping anywhere but Amazon,…

>Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price. That's not true. You don't have to manufacture your own goods to not compete with others at a commodity level. It's quite literally why branding is so important. But I do find it funny you wound up doing exactly what the…

> It's quite literally why branding is so important.

Most of the goods you purchase are from brands not owned by the retail website. Pick a hobby and consider the prominent brands, then consider the prominent websites people in that hobby often patronize - they are very often not the same.

> and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while

Because it kind of is. The smaller you are, the more significant of an impact an outage has. Nobody is going to think "this website I've never heard of totally isn't working right now, so I'll just wait around until they fix it". No, instead, they will go on to their next google search result - or Amazon. This hurts even worse if they clicked one of your ads while your website is down... costing you actual money and not just the sale opportunity. A customer that might have had an expected lifetime of 1-3+ orders now becomes 0.

Websites don't have "business hours", and routine maintenance is not acceptable for customers that shop when it's convenient for themselves. Remember, ecommerce exists as a better way to do over-the-phone and catalog/mail ordering. One of those benefits is you can order whenever you want - so it is in fact important for the website to be online at close to 100% as possible.

Re: The anatomy of a 2AM mental breakdown

#195

Post by chasd00 »

JohnMakin wrote: post #62
everdrive wrote: post #54

Earlier quoted context omitted.

Bear in mind that when something "breaks" a business can freak out, but they're not freaking out about lots of other arguably more important problems: - The software they purchased doesn't do anything because it was never staffed or configured correctly. - Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements. - Some capability doesn't actually do anything, but no one cares en…

One of the funnier situations (funny now, wasn't so much the first time I saw this) that I run into at new gigs or contracts is when a business has absolutely zero monitoring or alerting. Go into their backend, it's predictably a dumpster fire. Start plugging in monitoring and the business realizes everything is on fire and PANICS. It's very difficult to explain to someone who definitely doesn't want to hear that it'…

> It's very difficult to explain to someone who definitely doesn't want to hear that it's actually been broken for a long time, they just didn't care enough to notice or to invest in doing it right the first time.

if no one noticed then can you really say it was broken?

Re: The anatomy of a 2AM mental breakdown

#196

Post by jjnoakes »

x0x0 wrote: post #151
bluepnume wrote: post #36

Looks like the bug was in a monkey-patched `window.fetch` https://github.com/PostHog/posthog-js/blob/759829c67fcb8720f... The biggest lesson here is, if you're writing a popular library that monkey-patches global functions, it needs to be really well tested. There's a difference between "I'll throw posthog calls in a try/catch just in case" and "With posthog I literally can't make fetch() calls with POST"

it's common with all these analytics toolkits. When you're monkeying with such a core api, I have no idea how you can actually test everything. eg heap analytics still (as of this month) bad touches something inside hotwire and randomly entirely breaks hotwire causing every click to do a full page load. ime, it affects 30-60% of page loads. You can fix it (but thanks for 50+ hours of debugging) but making heap load a…

> I have no idea how you can actually test everything.

While a worthy goal (if unattainable sometimes), in this case, folks would settle for "test anything", which would have caught the regression.

Re: The anatomy of a 2AM mental breakdown

#197

Post by dmvdoug »

JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.

Re: The anatomy of a 2AM mental breakdown

#198

Post by throwaheyy »

fragmede wrote: post #121
prewett wrote: post #102

Earlier quoted context omitted.

What is your $2m generating? For instance, if you invested in high quality dividend stocks, you can get 3% and it should generally grow about what GDP does (which should also include inflation effects). So that's $60k/year, or after 15% dividend tax + let's say 5% California state tax, for a net of $48k. If you're in somewhere rent-controlled, seems like even with a family you're not going to on the streets starving.…

if you're only getting 3% with dividends, can I interest you with an HYSA paying 5%?

HYSA varies with interest rates

Re: The anatomy of a 2AM mental breakdown

#199

Post by [deleted] »

dmvdoug wrote: post #197
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.

[deleted]

Re: The anatomy of a 2AM mental breakdown

#200

Post by aPoCoMiLogin »

sb8244 wrote: post #135
JohnMakin wrote: post #13

Working as a SRE for a year in a large global company broke me out of this "panic" mode described in this post. To a business, every problem seems like a world-ending event. It's very easy to give in to panic in those situations. However, in reality, it's rarely that bad, and even if it is, you'll probably survive without harm. The key in these situations, and what I try to do (totally relate to breaking out in a swe…

Some of the worst mistakes that I saw were from over-reaction in an active incident. One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done. I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on…

> One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done.

so many ppl nowadays don't care about knowing how thing works, they just push shit and cross the fingers.

Post reply on HN