Post-mortem for last week's incident at Kagi • Hacker News Board

Re: Post-mortem for last week's incident at Kagi

#21

Post by timwis » Tue, Jan 16, 2024, 10:43 PM UTC

z64 wrote: post #9

fragmede wrote: post #7

That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.

Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is…

Thank you for sharing! I’m surprised to hear that, given how impressive your product is, but I’m an even bigger fan now.

Re: Post-mortem for last week's incident at Kagi

#22

Post by hacker_newz » Tue, Jan 16, 2024, 10:45 PM UTC

fragmede wrote: post #7

That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.

What are "the right datadog dashboards and splunk queries"?

Re: Post-mortem for last week's incident at Kagi

#23

Post by muhammadusman » Tue, Jan 16, 2024, 10:45 PM UTC

I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated.

In the past, services I heavily rely on (e.g. Github), have updated their status pages immediately and this allows me to rest assured that people are aware of the issue and it's not an issue with my devices. When this happened with Kagi, I was looking up the nearest grocery stores open since we were getting snow later that day so it was almost like I got let down b/c I had to go to Google for this.

I will continue using Kagi b/c 99.9% of the other time I've used it, it has been better than Google but I hope the authors of the post-mortem do mean it when they say they'll be moving their status page code to a different service/platform.

And thanks again Zac for being transparent and writing this up. This is part of good engineering!

Re: Post-mortem for last week's incident at Kagi

#24

Post by blantonl » Tue, Jan 16, 2024, 10:50 PM UTC

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources

I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariably push a hotfix that breaks something else and then you are in a world of hurt. \

Muphy's law is a cruel thing to sysadmins and developers.

Re: Post-mortem for last week's incident at Kagi

#25

Post by blantonl » Tue, Jan 16, 2024, 10:51 PM UTC

hacker_newz wrote: post #22

fragmede wrote: post #7

That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.

What are "the right datadog dashboards and splunk queries"?

Lots and lots of money to catch what you don't know, which means "oh crap, now we need to log this also"

Re: Post-mortem for last week's incident at Kagi

#26

Post by scblzn » Tue, Jan 16, 2024, 10:56 PM UTC

hacker_newz wrote: post #22

fragmede wrote: post #7

That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.

What are "the right datadog dashboards and splunk queries"?

surely these ones for datadog: https://nitter.net/TurnerNovak/status/1654577231937544192

/s

Re: Post-mortem for last week's incident at Kagi

#27

Post by JumpCrisscross » Tue, Jan 16, 2024, 10:57 PM UTC

spiderice wrote: post #19

jacob019 wrote: post #13

If you're listening, Kagi, please add an à la carte plan for search. Maybe hide it behind the API options as not to disrupt your normal plans. I love the search and I'm happy to pay, but I'm cost sensitive now and it's the only way that I'm going to feel comfortable using it long-term.

They have a $5 for 300 searches option. Is that not what you're referring to?

> Is that not what you're referring to?

It sounds like they'd prefer a 2¢ per search option via API.

Re: Post-mortem for last week's incident at Kagi

#28

Post by JumpCrisscross » Tue, Jan 16, 2024, 10:59 PM UTC

renewiltord wrote: post #3

Interesting. The classic problem. You offer to not meter something and then someone will use it to max capacity. Then you’re forced to place a limit so that one user won’t hose everyone else.

> you’re forced to place a limit so that one user won’t hose everyone else

Soft limits at the tail of usage comports with the term unlimited as it's commonly used. For Kagi, a rate limit derived from how quickly a human can type makes sense.

Re: Post-mortem for last week's incident at Kagi

#29

Post by kelnos » Tue, Jan 16, 2024, 10:59 PM UTC

z64 wrote: post #9

fragmede wrote: post #7

That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.

Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is…

Hi Zac, thank you for chiming in here. Been using Kagi since the private beta, and have been overwhelmingly impressed by the service since I first used it.

Don't worry too much about all the people being harsh in the comments here. There's always a tendency for HN users to pile on with criticism whenever anyone has an outage.

I've always found this bizarre, because I've worked at places with worse issues, and more holes in monitoring or whatever than a lot of the companies that get skewered here. Perhaps many of us are just insecure about our own infra and project our feeling onto other companies when they have outages.

Y'all are doing fine, and I think it's to your credit that you're able to run Kagi's users table off a single, fairly cheap primary database instance. I've worked at places that haven't much thought to optimization, and "solve" scaling problems by throwing more and bigger hardware at it, and then wonder later on why they're bleeding cash on infrastructure. Of course, by that point, those inefficiencies are much more difficult to fix.

As for monitoring, unfortunately sometimes you don't know everything you need to monitor until something bad happens because your monitoring was missing something critical that you didn't realize was critical. That's fine; seems like y'all are aware and are plugging those holes. I'm sure there will be more of those holes in the future, that's just life.

At any rate, keep doing what you're doing, and I know the next time you get hit with something bad, things will be a bit better.

Re: Post-mortem for last week's incident at Kagi

#30

Post by z64 » Tue, Jan 16, 2024, 11:03 PM UTC

layoric wrote: post #8

"This didn’t exactly come as a surprise to us, as for the entirety of Kagi’s life so far we have actually used the cheapest, single-core database available to us on GCP!" Outages suck, but I love the fact that they are building such a lean product. Been paying for Kagi as a part of de-Google-ifying my use of online services and the experience so far (I wasn't impacted by this outage) has been great. A few years ago I…

Very cool! The deluge of cloud solutions are absolutely one of those things that can be a distraction from figuring out "What do I actually need the computer to do?".

Internally I try to promote solutions with the fewest moving cloud-parts, and channel the quiet wisdom of those running services way larger than ours with something like an sqlite3 file that they rsync... I know they're out there. Not to downplay the feats of engineering of huge distributed solutions, but sometimes things can be that simple!