Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

121–130 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#121

Post by tetha »

JohnMakin wrote: post #49
blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”

> I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”

There is coincidental function, and planned function.

If you just throw numbers at a configuration parameter until it works and then call it "fixed",it may be fixed, but that's just coincidental. If the system load changes, you can't really tell how to modify the value afterwards.

On the other hand, if you can explain why a certain setting will fix it, that's planned function, and if something changes, you most likely know how to proceed from there.

Re: Post-mortem for last week's incident at Kagi

#122

Post by darkwater »

laeri wrote: post #117
darkwater wrote: post #114

Earlier quoted context omitted.

In a world of unicorns and rainbows, absolutely. In the real world, it's as you probably already know: it's not that easy in a complex enough system. Quick counter-example for GP: what if the 500 spike is due to a spike in malformed requests from a single (maybe malicious) user?

A malformed request should not lead to a 500, they should be handled and validated.

Well, in the real world it might. It should trigger a bug creation and a fix to the code, but not an incident. Now all of a sudden to decide this you need more complex and/or specific queries in your monitoring system (or a good ML-based alert system), so complexity is already going up.

Re: Post-mortem for last week's incident at Kagi

#123

Post by Gareth321 »

phyzome wrote: post #84
muhammadusman wrote: post #23

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

This is exactly why those status pages are almost always a lie. Either they need to be fully automated without some middle manager hemming and hawing, or they shouldn’t be there at all. From a customer’s perspective, I’ve been burned so many times on those status pages that I ignore them completely. I just assume they’re a lie. So I’ll contact support straight away - the very thing these status pages were intended to mitigate.

Re: Post-mortem for last week's incident at Kagi

#124

Post by tsegers »

sedatk wrote: post #99

I’m a paid user of Kagi, and having experienced downtime made me realize how much I took Google’s reliability for granted. Google has never gone down on me, maybe once in the last two decades. Losing access to your search engine is quite crippling. I LOVE Kagi, that’s why I pay for it, but experiencing downtime in my second month was quite off-putting. I love post-mortems, but I hope to never read them. :) That said,…

As another paying user of Kagi I wonder what prevented you from using another search engine for the six hours that Kagi was unavailable. Search engines are not like your email provider or ISP in that you're locked in.

Re: Post-mortem for last week's incident at Kagi

#125

Post by jnsaff2 »

blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Yeah, the "coincidence" leads you to jump to conclusions about your change being the cause.

This is very human, we do it all the time.

However having been through these enough times I have picked up a habit of questioning more assumptions and not flagging something as verified data before it is.

I'm still nowhere near perfect at removing these biases and early conclusions but it has helped.

Having an open mind is hard work.

Re: Post-mortem for last week's incident at Kagi

#127

Post by tnolet »

blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Oh yes, the amount of times I rolled back a change during an outage that had nothing to do with the outage...

It's a critical skill for engineers: being able to critically reason, debug and "test in isolation" any changes that address outages. Much harder than it seems and typically a "senior" skill to have.

Re: Post-mortem for last week's incident at Kagi

#129

Post by taneq »

virtue3 wrote: post #86
phyzome wrote: post #84

Earlier quoted context omitted.

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

I think your bit at the end is the most important. ANY communication is better than no communication "everything is fine, it must be you" is the worst feeling in these cases. Especially if your business is reliant on said service and you can't figure out why you are borked (eg the github ones).

Once an ISP support person insisted that I drive down to the shop and buy a phone handset so I could confirm presence of a dial tone on a line that my vdsl modem had line sync on before they’d tell me their upstream provider had an outage. I was… unimpressed.

Re: Post-mortem for last week's incident at Kagi

#130

Post by DANmode »

PeterStuer wrote: post #120
DANmode wrote: post #116

Earlier quoted context omitted.

Which then means not going yellow or red technically constitutes fraud.

Not necessarily. The situation can be genuinely unclear to the point where it is a judgement call, and then it becomes a matter of how to weigh the consequences.

If you're asking how many users are affected, and your service is listed as green...
Post reply on HN