101–110 of 186 posts
My search went bad and I assumed I’d broken something, tried a few things, took a break, tried a few more and it was working. I moved on.
It wasn’t ever at my end.
> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.
Unlimited searches as a real user is not the same thing as running a bot against the service.
> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.
Unlimited searches as a real user is not the same thing as running a bot against the service.
Earlier quoted context omitted.
Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is…
Mh, I work quite a bit in the OPs-side and monitoring and observability are part of my job, for a bit of time now too. I'll say: Effective observability, monitoring and alerting of complex systems is a really hard problem. Like, you look at a graph of a metric, and there are spikes. But... are the spikes even abnormal? Are the spikes caused by the layer below, because our storage array is failing? Are the spikes caus…
Aaaahhh, it's crazy how much this incident resonates with me! I've personally handled this exact same kind of outage more times than I'd care to admit. And just like the fine folks at Kagi, I've fallen into the same rabbit hole (database connection pool health) and tried all the same mitigations - futilely throwing new instances at the problem, the belief that if I could just "reset" traffic it'd all be fixed, etc...…
I was wondering the same thing. They would have some writes indirectly due to searches, say if someone chooses to block a search result. They’re also going to have some history and analytics surely. But yeah it’s not obvious what should cause per search write lock contention…
Well, at least until you get a user who does 60k "in a short time period"... :-)
I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…
As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…
Earlier quoted context omitted.
nah, unless you were the intern, you should be safe it's like Star Trek where you're fine unless you're the one wearing the red shirt
If I ever run a company, all the interns will get red shirts
> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.
I was one of the users that went and reported this issue on Discord. I love Kagi but I was a bit disappointed to see that their status page showed everything was up and running. I think that made me a bit uneasy and it shows their status pages are not given priority during incidents that are affecting real users. I hope in the future the status page is accurately updated. In the past, services I heavily rely on (e.g.…
As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…