Muphy's law
Post-mortem for last week's incident at Kagi
91–100 of 186 posts
Re: Post-mortem for last week's incident at Kagi
#92Earlier quoted context omitted.
It also could have just been a joke, not one they were actually going to put into practice...
I’ve seen enough brainless execs and startup founders around here that you have to take stupid ideas like that seriously. I hope it was a joke, but it’s not very funny for an intern.
You definitely do not have to take stupid ideas like that seriously. Simply your life.
Re: Post-mortem for last week's incident at Kagi
#93Earlier quoted context omitted.
> Is that not what you're referring to? It sounds like they'd prefer a 2¢ per search option via API.
Yes, exactly. Or 5 cents for the first 100 searches, 2 cents after, something like that.
They do have the FastGPT api which is approx 1.5cents per api call
Re: Post-mortem for last week's incident at Kagi
#94Earlier quoted context omitted.
I’ve seen enough brainless execs and startup founders around here that you have to take stupid ideas like that seriously. I hope it was a joke, but it’s not very funny for an intern.
You definitely do not have to take stupid ideas like that seriously. Simply your life.
A little more credit to people for making a funny and a little less assumptions that people are that boneheaded would make the world a better place to be sure. I know for one, I rather enjoy being able to see the humor and having a laugh rather than getting all enraged and venting on the internet to show some sort of self perceived superiority.
Re: Post-mortem for last week's incident at Kagi
#95> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts.
Don't advertise something as unlimited if it's not actually unlimited.
Re: Post-mortem for last week's incident at Kagi
#96That speaks volumes about the observability they have of their internal systems. It's easy for me to say they should have seen it sooner, but the right datadog dashboards and splunk queries should have made that clear as day much faster. Hopefully they take it as a learning experience and invest in better monitoring.
Hi there, I'm Zac, Kagi's tech lead / author of the post-mortem etc. This has 100% been a learning experience for us, but I can provide some finer context re: observability. Kagi is a small team. The number of staff we have capable of responding to an event like this is essentially 3 people, seated across 3 timezones. For myself and my right-hand dev, this is actually our very first step in our web careers - this is…
I work in a giant investment bank with hundreds of people who can answer across all time zones. We still f up, we still don't always know where problems lie and we still sometimes can spend hours on a simple DoS.
You'll only get better at guessing what the issue could be: an exploit by a user is something you'll remember forever and will overly protect against from now on, until you hit some other completely different problem which your metric will be unprepared for, and you'll fumble around and do another post mortem promising to look at that new class of issues etc.
You'll marvel at the diversity of potential issues, especially in human-facing services like yours. But you'll probably have another long loss of service again one day, and you're right to insist on the transparency / speed of signaling to your users: they can forgive everything as long as you give them an early signal, a discount and an apology, in my experience.
Re: Post-mortem for last week's incident at Kagi
#97> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.
confused how both it is advertised, and against ToS
Re: Post-mortem for last week's incident at Kagi
#98> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.
Unlimited searches as a real user is not the same thing as running a bot against the service.
Re: Post-mortem for last week's incident at Kagi
#99I’m a paid user of Kagi, and having experienced downtime made me realize how much I took Google’s reliability for granted. Google has never gone down on me, maybe once in the last two decades. Losing access to your search engine is quite crippling. I LOVE Kagi, that’s why I pay for it, but experiencing downtime in my second month was quite off-putting. I love post-mortems, but I hope to never read them. :)
That said, I hope this experience makes Kagi even more resilient and reliable.
Re: Post-mortem for last week's incident at Kagi
#100As much as people gush over Kagi on HN, I still have yet to actually try it because I cannot for the life of me get authentication to work. Even after immediately resetting my password, I get an "incorrect email or password" or "try again later" error on the login page. I've tried at least 3 times over the last few months with the same results each time. If such a fundamental part of a web service company's website i…
Push on - it’s a great service.