Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

71–80 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#71

Post by WhitneyLand »

rwiggins wrote: post #69

Aaaahhh, it's crazy how much this incident resonates with me! I've personally handled this exact same kind of outage more times than I'd care to admit. And just like the fine folks at Kagi, I've fallen into the same rabbit hole (database connection pool health) and tried all the same mitigations - futilely throwing new instances at the problem, the belief that if I could just "reset" traffic it'd all be fixed, etc...…

I was wondering the same thing.

They would have some writes indirectly due to searches, say if someone chooses to block a search result. They’re also going to have some history and analytics surely.

But yeah it’s not obvious what should cause per search write lock contention…

Re: Post-mortem for last week's incident at Kagi

#72

Post by Tempest1981 »

kelnos wrote: post #29
z64 wrote: post #9

Earlier quoted context omitted.

Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is…

Hi Zac, thank you for chiming in here. Been using Kagi since the private beta, and have been overwhelmingly impressed by the service since I first used it. Don't worry too much about all the people being harsh in the comments here. There's always a tendency for HN users to pile on with criticism whenever anyone has an outage. I've always found this bizarre, because I've worked at places with worse issues, and more ho…

> people being harsh in the comments here

I've read most of the comments here, and don't recall anything negative, just supportive.

Re: Post-mortem for last week's incident at Kagi

#73

Post by wiml »

Neikius wrote: post #37
muhammadusman wrote: post #23

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…

Microsoft is notorious for their lax status page updates...

Is there anyone who isn't?

Re: Post-mortem for last week's incident at Kagi

#74

Post by mightyham »

As much as people gush over Kagi on HN, I still have yet to actually try it because I cannot for the life of me get authentication to work. Even after immediately resetting my password, I get an "incorrect email or password" or "try again later" error on the login page. I've tried at least 3 times over the last few months with the same results each time.

If such a fundamental part of a web service company's website is broken, it makes me weary of their competence.

Re: Post-mortem for last week's incident at Kagi

#75

Post by leesalminen »

sabujp wrote: post #64

We were later in contact with an account that we blocked who claimed they were using their account to perform automated scraping of our results, which is not something our terms allow for." Set QPS limits for every possible incoming RPC / API / HTTP request , especially public ones!

So much this. I learned this the hard way.

We had a search function with typeahead abilities. I had intentionally removed the rate limit from that endpoint to support fast typers.

One day around 6AM, someone in Tennessee came into work and put their purse down on their keyboard. The purse depressed a single key and started hitting the API with each keystroke.

Of course after 15 minutes of this the db became very unhappy. Then a web server crashed because the db was lagging too much. Cascading failures until that whole prod cluster crashed.

Needless to say the rate limit was readded that day ;).

Re: Post-mortem for last week's incident at Kagi

#76

Post by SadCordDrone »

fancy_pantser wrote: post #48
rconti wrote: post #46

Earlier quoted context omitted.

What does "pathological" mean in this context?

being such to a degree that is extreme, excessive, or markedly abnormal (with a connotation of it happening on purpose)

What does " being such to a degree that is extreme, excessive, or markedly abnormal (with a connotation of it happening on purpose) " mean in this context?

Re: Post-mortem for last week's incident at Kagi

#77

Post by fancy_pantser »

SadCordDrone wrote: post #76
fancy_pantser wrote: post #48

Earlier quoted context omitted.

being such to a degree that is extreme, excessive, or markedly abnormal (with a connotation of it happening on purpose)

What does " being such to a degree that is extreme, excessive, or markedly abnormal (with a connotation of it happening on purpose) " mean in this context?

pathological

Re: Post-mortem for last week's incident at Kagi

#78

Post by ok_dad »

ashton314 wrote: post #68
dylan604 wrote: post #65

Earlier quoted context omitted.

nah, unless you were the intern, you should be safe it's like Star Trek where you're fine unless you're the one wearing the red shirt

If I ever run a company, all the interns will get red shirts

Think about the optics of your funny joke. You’re saying you’ll advertise that the interns are replaceable fodder. Even if that’s true, it’s quite a dick move.

Re: Post-mortem for last week's incident at Kagi

#79

Post by ethbr1 »

ok_dad wrote: post #78
ashton314 wrote: post #68

Earlier quoted context omitted.

If I ever run a company, all the interns will get red shirts

Think about the optics of your funny joke. You’re saying you’ll advertise that the interns are replaceable fodder. Even if that’s true, it’s quite a dick move.

Or acknowledging that interns perform a valuable role and celebrating their selfless sacrifice of their very lives for the good of the company.

That we should all die such a noble corporate death, at the hands of a colleague possessed by strange galactic energies.

Re: Post-mortem for last week's incident at Kagi

#80

Post by PNewling »

ok_dad wrote: post #78
ashton314 wrote: post #68

Earlier quoted context omitted.

If I ever run a company, all the interns will get red shirts

Think about the optics of your funny joke. You’re saying you’ll advertise that the interns are replaceable fodder. Even if that’s true, it’s quite a dick move.

It also could have just been a joke, not one they were actually going to put into practice...
Post reply on HN