Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

171–180 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#171

Post by wiether »

tsegers wrote: post #124
sedatk wrote: post #99

I’m a paid user of Kagi, and having experienced downtime made me realize how much I took Google’s reliability for granted. Google has never gone down on me, maybe once in the last two decades. Losing access to your search engine is quite crippling. I LOVE Kagi, that’s why I pay for it, but experiencing downtime in my second month was quite off-putting. I love post-mortems, but I hope to never read them. :) That said,…

As another paying user of Kagi I wonder what prevented you from using another search engine for the six hours that Kagi was unavailable. Search engines are not like your email provider or ISP in that you're locked in.

> I wonder what prevented you from using another search engine

Well, seeing that Kagi was down, I tried to switch to Google by doing "!g xxx" which gave me the same result than my previous "xxx" search.

I took me a few seconds to realize how stupid I was and then typing "google.com" in the address bar.

Re: Post-mortem for last week's incident at Kagi

#172

Post by bostik »

rwiggins wrote: post #106
WhitneyLand wrote: post #71

Earlier quoted context omitted.

I was wondering the same thing. They would have some writes indirectly due to searches, say if someone chooses to block a search result. They’re also going to have some history and analytics surely. But yeah it’s not obvious what should cause per search write lock contention…

You know, in retrospect, I think Kagi expects O(thousands) searches per month per user, so doing per-user usage accounting in the DB is fine -- thanks to row-level locking. Well, at least until you get a user who does 60k "in a short time period"... :-)

It's the outliers and "surely nobody could be THAT awful" that kill you. Every time.

Re: Post-mortem for last week's incident at Kagi

#173

Post by rurp »

dmoy wrote: post #162
lamontcg wrote: post #151

Earlier quoted context omitted.

I hate it when button mashers are in charge of incident response. Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.

To be fair, if your rollbacks are cheap (1-3 button pushes), safe, and automated, it doesn't really hurt to try a rollback while you look for other causes.

Not to mention the times someone has said that an issue couldn't possibly have been caused by a recent change they made, only to find out that it was.

Re: Post-mortem for last week's incident at Kagi

#174

Post by pbronez »

sedatk wrote: post #99

I’m a paid user of Kagi, and having experienced downtime made me realize how much I took Google’s reliability for granted. Google has never gone down on me, maybe once in the last two decades. Losing access to your search engine is quite crippling. I LOVE Kagi, that’s why I pay for it, but experiencing downtime in my second month was quite off-putting. I love post-mortems, but I hope to never read them. :) That said,…

100%. This outage and the (unrelated) bug on the new mobile safari extension were jarring. I definitely rely on snappy & ubiquitous to Kagi!

Re: Post-mortem for last week's incident at Kagi

#175

Post by dotnet00 »

mightyham wrote: post #152
freediver wrote: post #87

Earlier quoted context omitted.

Have you tried contacting Kagi support to help debug the issue?

No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.

It's understandable that such a basic seeming issue would negatively impact your opinion of the service, but it's also worth considering that such a basic issue must surely be some sort of unique edge case if the vast majority of other people are claiming to be happily using the service (which implies being able to log into their accounts).

Of course you don't owe Kagi anything so you don't have to reach out to support, but just something to consider before questioning someone's competence.

Re: Post-mortem for last week's incident at Kagi

#176

Post by sedatk »

tsegers wrote: post #124
sedatk wrote: post #99

I’m a paid user of Kagi, and having experienced downtime made me realize how much I took Google’s reliability for granted. Google has never gone down on me, maybe once in the last two decades. Losing access to your search engine is quite crippling. I LOVE Kagi, that’s why I pay for it, but experiencing downtime in my second month was quite off-putting. I love post-mortems, but I hope to never read them. :) That said,…

As another paying user of Kagi I wonder what prevented you from using another search engine for the six hours that Kagi was unavailable. Search engines are not like your email provider or ISP in that you're locked in.

The thing is, I never thought Kagi was down and thought that it must be a problem with my configuration or connection. That was how much I trusted in Kagi. I didn’t spend whole downtime online though.

Re: Post-mortem for last week's incident at Kagi

#178

Post by dghlsakjg »

mightyham wrote: post #152
freediver wrote: post #87

Earlier quoted context omitted.

Have you tried contacting Kagi support to help debug the issue?

No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.

It is a beta project.

You would expect this to work, but you also shouldn't be surprised that a beta project isn't perfect.

If you need the ability to login reliably and a search engine that never goes down, stick to google.

If you want to help a new entrant with a product that reliably outperforms Google search get their product battle-ready, then give them a bit of a chance to make it right.

Re: Post-mortem for last week's incident at Kagi

#179

Post by jabradoodle »

zbentley wrote: post #170
jabradoodle wrote: post #167

Earlier quoted context omitted.

If your service is returning 5xx, that is the the definition of a server error, of course that is degraded service. Instead we have pointless dashboards that are green an hour after everything is broken. Returning 4xx on a client error isn't hard and is usually handled largely by your framework of choice. Your argument is a strawman

> Returning 4xx on a client error isn't hard and is usually handled largely by your framework of choice. > Your argument is a strawman That's....super not true. Malformed requests with gibberish (or, more likely, hacker/pentest- generated) headers will cause e.g. Django to return 5xx easily. That's just the example I'm familiar with, but cursory searching indicates reports of similar failures emitted by core framewor…

Usually handled by the framework, you may have to write some code, I'd expect my saas provider to write code so that I know whether their service is available or not.

Re: Post-mortem for last week's incident at Kagi

#180

Post by throwaway_673 »

JohnMakin wrote: post #49
blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”

This is unfortunately very true.

Example:

At my work place we were made aware of some instability issues and the first place to look is of course the logs. It had apparently been happening for a while, but the most important parts of the system were not affected noticeably. This was early morning and we didn't have much traffic.

We noticed a lot of network connection errors for SQS, so one of the team members started blaming Amazon.

My response was that if SQS had these instabilities, we would have noticed posts on HN or downtime on other sites. The developer ignored this and jumped into the code to add toggles and debug log messages.

I spent 2 more minutes and found other cases where we had connection issues to different services. I concluded that the reason why we saw so many SQS errors was because it is called much more frequently.

The other developer pushed changes to prod which caused even more problems, because now the service could not even connect to the database.

He wrongly assumed that the older versions of the service, which was still running, was holding on to the connections and killed them in order to make room for his new version. We went from a system with instabilities, to a system that was down.

The reason was a network error that was fixed by someone in the ops team.

Post reply on HN