Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

131–140 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#131

Post by lesam »

tnolet wrote: post #127
blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Oh yes, the amount of times I rolled back a change during an outage that had nothing to do with the outage... It's a critical skill for engineers: being able to critically reason, debug and "test in isolation" any changes that address outages. Much harder than it seems and typically a "senior" skill to have.

Assuming you have a tested rollback strategy, I’m all for rolling back first and asking questions later.

It at least can often eliminate the rolled back change as the cause, even if it doesn’t fix the problem.

Re: Post-mortem for last week's incident at Kagi

#132

Post by tazjin »

lrem wrote: post #112
TeeWEE wrote: post #110

Earlier quoted context omitted.

Connect your status page to actual metrics and decide a treshold for downtime. Boom you’re done.

Does anyone serious do this? That’s an honest question, from a pretty experienced SRE.

https://www.buildkitestatus.com/

Re: Post-mortem for last week's incident at Kagi

#134

Post by AtNightWeCode »

sabujp wrote: post #64

We were later in contact with an account that we blocked who claimed they were using their account to perform automated scraping of our results, which is not something our terms allow for." Set QPS limits for every possible incoming RPC / API / HTTP request , especially public ones!

And a public endpoint is any Internet facing endpoint including the ones where the user needs to be logged in. People seems to forget that.

Re: Post-mortem for last week's incident at Kagi

#135

Post by jabradoodle »

darkwater wrote: post #122
laeri wrote: post #117

Earlier quoted context omitted.

A malformed request should not lead to a 500, they should be handled and validated.

Well, in the real world it might. It should trigger a bug creation and a fix to the code, but not an incident. Now all of a sudden to decide this you need more complex and/or specific queries in your monitoring system (or a good ML-based alert system), so complexity is already going up.

You need to validate your inputs and return 4xx

Re: Post-mortem for last week's incident at Kagi

#136

Post by throwaway167 »

virtue3 wrote: post #86
phyzome wrote: post #84

Earlier quoted context omitted.

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

I think your bit at the end is the most important. ANY communication is better than no communication "everything is fine, it must be you" is the worst feeling in these cases. Especially if your business is reliant on said service and you can't figure out why you are borked (eg the github ones).

Your point highlights thinking about what's being designed.

everything is fine is different from nothing has been reported. A green is misleading, there should be no green as green is unknown, there should be nothing with a note that there's nothing, and that's not the same as a green light.

Re: Post-mortem for last week's incident at Kagi

#137

Post by PeterStuer »

DANmode wrote: post #130
PeterStuer wrote: post #120

Earlier quoted context omitted.

Not necessarily. The situation can be genuinely unclear to the point where it is a judgement call, and then it becomes a matter of how to weigh the consequences.

If you're asking how many users are affected, and your service is listed as green...

What if the answer is 0.00001%?

Re: Post-mortem for last week's incident at Kagi

#138

Post by darkwater »

jabradoodle wrote: post #135
darkwater wrote: post #122

Earlier quoted context omitted.

Well, in the real world it might. It should trigger a bug creation and a fix to the code, but not an incident. Now all of a sudden to decide this you need more complex and/or specific queries in your monitoring system (or a good ML-based alert system), so complexity is already going up.

You need to validate your inputs and return 4xx

Yeah and you also shall not write bugs in your code. Real world has bugs, even trivial ones.

Re: Post-mortem for last week's incident at Kagi

#139

Post by pembrook »

kelnos wrote: post #29
z64 wrote: post #9

Earlier quoted context omitted.

Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is…

Hi Zac, thank you for chiming in here. Been using Kagi since the private beta, and have been overwhelmingly impressed by the service since I first used it. Don't worry too much about all the people being harsh in the comments here. There's always a tendency for HN users to pile on with criticism whenever anyone has an outage. I've always found this bizarre, because I've worked at places with worse issues, and more ho…

Agreed. Even though this site is on YC’s domain, I think only a few of the folks in the comments are actually early-adopting startup types. Probably just due to power law statistics, I’d guess most commenters are big company worker bees who’ve never worked on/at a seed stage startup.

If everything at Kagi was FAANG-level bulletproof, with extensive processes around outages/redundancy, then the team absolutely would not be making the best use of their time/resources.

If you’re risk averse and aren’t comfortable encountering bugs/issues like this, don’t try any new software product of moderate complexity for about 7-10 years.

Re: Post-mortem for last week's incident at Kagi

#140

Post by petesergeant »

virtue3 wrote: post #86
phyzome wrote: post #84

Earlier quoted context omitted.

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

I think your bit at the end is the most important. ANY communication is better than no communication "everything is fine, it must be you" is the worst feeling in these cases. Especially if your business is reliant on said service and you can't figure out why you are borked (eg the github ones).

> ANY communication is better than no communication

Better for the consumer, although not necessarily better for the provider if they have an SLA.

Post reply on HN