Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

51–60 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#51

Post by JohnMakin »

z64 wrote: post #9
fragmede wrote: post #7

That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.

Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is…

> Kagi is a small team.

I figured this was the case when you said “our devops engineer” singular and not “one of our devops engineers.”

I’m glad you’re willing at this stage to invest in SRE. It’s a decision a lot of companies only make when they absolutely have to or have their backs against a wall.

Re: Post-mortem for last week's incident at Kagi

#52

Post by 3np »

Wouldn't some level of per-account rate-limiting make sense? Say, 1000 searches per hour? It's commendable and impressive that Kagi has apparently been able to get this far and perform this consistently without any account-level automated rate-limiting but the only alternative is an inevitable cat-and-mouse and whack-a-mole of cutting off paying customers who knowingly or not violate the ToS. Returning 429 with actionable messages makes it clear to users what they should expect.

You obviously want the block-interval to be long enough to not cause too much additional churn on the database.

Applying restrictions on IP-level when you can avoid it is just a world of frustration for everyone involved.

Re: Post-mortem for last week's incident at Kagi

#53

Post by BeetleB »

spiderice wrote: post #19
jacob019 wrote: post #13

If you're listening, Kagi, please add an à la carte plan for search. Maybe hide it behind the API options as not to disrupt your normal plans. I love the search and I'm happy to pay, but I'm cost sensitive now and it's the only way that I'm going to feel comfortable using it long-term.

They have a $5 for 300 searches option. Is that not what you're referring to?

They used to have a $5/mo option with N searches, and then charge some cents per search after that. For a lot of people, the net amount would still be under $10/mo.

Re: Post-mortem for last week's incident at Kagi

#54

Post by precision1k »

Reminds me of a time I was running a proof-of-concept for a new networking tool at a customer site, and about two minutes after we got it running their entire network went down. We were in a sandboxed area so there was no way our product could of caused a network wide outage, but in my head I'm thinking: "there's no way, right. . . .RIGHT?!?!".

Re: Post-mortem for last week's incident at Kagi

#55

Post by sa46 »

blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Oh man, last week we had a small outage. Database queries took much longer than normal. I just so happened to be doing ad-hoc querying on the same table at the same time.

“Luckily”, the problem was unrelated to my querying but two coincidences are proper scary.

Re: Post-mortem for last week's incident at Kagi

#56

Post by herpdyderp »

muhammadusman wrote: post #23

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…

I envy your experiences with other services. I've never seen any service's status page show downtime when or even soon after I start experiencing it. Often they simply never show it at all.

Re: Post-mortem for last week's incident at Kagi

#57

Post by NetOpWibby »

muhammadusman wrote: post #23

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…

It's worth noting that the status page software they use doesn't auto-update automatically.

> Please note that with all that cState can do, it cannot do automatic monitoring out of the box.

https://github.com/cstate/cstate

Re: Post-mortem for last week's incident at Kagi

#58

Post by ParetoOptimal »

NetOpWibby wrote: post #57
muhammadusman wrote: post #23

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…

It's worth noting that the status page software they use doesn't auto-update automatically. > Please note that with all that cState can do, it cannot do automatic monitoring out of the box. https://github.com/cstate/cstate

I guess a status page that doesn't auto-update is good for PR, but it's not very useful to show... you know... the status.

Re: Post-mortem for last week's incident at Kagi

#59

Post by eep_social »

fragmede wrote: post #35
hacker_newz wrote: post #22

Earlier quoted context omitted.

What are "the right datadog dashboards and splunk queries"?

This is easy for me to say in hindsight, but a graph of queries per user, for the top 100 accounts would have lit up the issue like a Christmas tree. But it's the kind of chart you stuff on the full page of charts that are usually not interesting. So many problems have been avoided by "huh, that looks weird" on a dashboard. In a more targeted search, asking Splunk to show out where the traffic is coming from on a map…

If daily volume is ~400k a sharp 60k spike should show even on a straight rps dashboard. I suspect that didn’t exist because most cloudy SaaS tools don’t seem to put any emphasis on traffic over a particular unit time, like in the Cloudflare site dashboard one must hover over the graph and subtract timestamps to find out if the number of request is per hour, minute, etc. Splunk is similarly bad — the longer the period query covers, the less granularity you get and the minimum seems to be minutely.

Drill down by user, locale, etc would just make it even easier to figure out what’s going on once you spot the spike and start to dig.

My unsolicited advice for Zac in case they’re reading is — start thinking about SLIs (I for indicators) sooner rather than later to help you think through cases like this.

Re: Post-mortem for last week's incident at Kagi

#60

Post by eep_social »

3np wrote: post #52

Wouldn't some level of per-account rate-limiting make sense? Say, 1000 searches per hour? It's commendable and impressive that Kagi has apparently been able to get this far and perform this consistently without any account-level automated rate-limiting but the only alternative is an inevitable cat-and-mouse and whack-a-mole of cutting off paying customers who knowingly or not violate the ToS. Returning 429 with actio…

> we need to set some automated limits to help us enforce this. From analyzing our user’s usage, we have picked some limits that no good-faith user of Kagi should reasonably hit.

> These new limits should already be in place by the time of this post, and we will monitor their impact and continue to tune them as needed.

Post reply on HN