Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

111–120 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#111

Post by PeterStuer »

phyzome wrote: post #84
muhammadusman wrote: post #23

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

And that is before 'going red' has ties to performance metrics with SLA impacts ...

Re: Post-mortem for last week's incident at Kagi

#112

Post by lrem »

TeeWEE wrote: post #110
phyzome wrote: post #84

Earlier quoted context omitted.

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

Connect your status page to actual metrics and decide a treshold for downtime. Boom you’re done.

Does anyone serious do this?

That’s an honest question, from a pretty experienced SRE.

Re: Post-mortem for last week's incident at Kagi

#113

Post by dotnet00 »

This was fun to see playing out, it made me realize how I didn't even think about how impressive it was that you guys (being a small team) were pushing out updates with very few bugs or visible downtime.

I didn't mind the downtime, just gave me an excuse to take a break and focus on more casual coding.

Re: Post-mortem for last week's incident at Kagi

#114

Post by darkwater »

lrem wrote: post #112
TeeWEE wrote: post #110

Earlier quoted context omitted.

Connect your status page to actual metrics and decide a treshold for downtime. Boom you’re done.

Does anyone serious do this? That’s an honest question, from a pretty experienced SRE.

In a world of unicorns and rainbows, absolutely. In the real world, it's as you probably already know: it's not that easy in a complex enough system.

Quick counter-example for GP: what if the 500 spike is due to a spike in malformed requests from a single (maybe malicious) user?

Re: Post-mortem for last week's incident at Kagi

#115

Post by zilti »

HaZeust wrote: post #18
zilti wrote: post #14

[flagged]

I'll bite; how do you figure?

GCP prices are ludicrously high. You're far better off to just get a VPS and self-host Postgres and your webserver. Which is what we did at my last startup; only thing we used GCP for was build servers, and only because we got a ton of free credits. We could've gotten a paid intern for the money we saved.

Re: Post-mortem for last week's incident at Kagi

#116

Post by DANmode »

PeterStuer wrote: post #111
phyzome wrote: post #84

Earlier quoted context omitted.

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

And that is before 'going red' has ties to performance metrics with SLA impacts ...

Which then means not going yellow or red technically constitutes fraud.

Re: Post-mortem for last week's incident at Kagi

#117

Post by laeri »

darkwater wrote: post #114
lrem wrote: post #112

Earlier quoted context omitted.

Does anyone serious do this? That’s an honest question, from a pretty experienced SRE.

In a world of unicorns and rainbows, absolutely. In the real world, it's as you probably already know: it's not that easy in a complex enough system. Quick counter-example for GP: what if the 500 spike is due to a spike in malformed requests from a single (maybe malicious) user?

A malformed request should not lead to a 500, they should be handled and validated.

Re: Post-mortem for last week's incident at Kagi

#118

Post by hammyhavoc »

JohnMakin wrote: post #49
blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”

I read "varying degrees of severity" as "varying degrees of sanity" and was all "omg, he just like me fr" until I reread it.

༼ ༎ຶ ᆺ ༎ຶ༽

Re: Post-mortem for last week's incident at Kagi

#119

Post by jon_adler »

laeri wrote: post #117
darkwater wrote: post #114

Earlier quoted context omitted.

In a world of unicorns and rainbows, absolutely. In the real world, it's as you probably already know: it's not that easy in a complex enough system. Quick counter-example for GP: what if the 500 spike is due to a spike in malformed requests from a single (maybe malicious) user?

A malformed request should not lead to a 500, they should be handled and validated.

True, however it also doesn’t impact other users and doesn’t justify reporting an incident on the status page.

Re: Post-mortem for last week's incident at Kagi

#120

Post by PeterStuer »

DANmode wrote: post #116
PeterStuer wrote: post #111

Earlier quoted context omitted.

And that is before 'going red' has ties to performance metrics with SLA impacts ...

Which then means not going yellow or red technically constitutes fraud.

Not necessarily. The situation can be genuinely unclear to the point where it is a judgement call, and then it becomes a matter of how to weigh the consequences.
Post reply on HN