Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

161–170 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#161

Post by AndrewKemendo »

Gareth321 wrote: post #123
phyzome wrote: post #84

Earlier quoted context omitted.

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

This is exactly why those status pages are almost always a lie. Either they need to be fully automated without some middle manager hemming and hawing, or they shouldn’t be there at all. From a customer’s perspective, I’ve been burned so many times on those status pages that I ignore them completely. I just assume they’re a lie. So I’ll contact support straight away - the very thing these status pages were intended to…

The simple fix is to have a “last update: date time”

Or you can build a team to automate everything and force everyone and everything into a rigid update frequency which becomes a metric that applies to everyone and becomes the bane of the existence of your whole engineering organization

Re: Post-mortem for last week's incident at Kagi

#162

Post by dmoy »

lamontcg wrote: post #151
JohnMakin wrote: post #49

Earlier quoted context omitted.

Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”

I hate it when button mashers are in charge of incident response. Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.

To be fair, if your rollbacks are cheap (1-3 button pushes), safe, and automated, it doesn't really hurt to try a rollback while you look for other causes.

Re: Post-mortem for last week's incident at Kagi

#163

Post by lamontcg »

dmoy wrote: post #162
lamontcg wrote: post #151

Earlier quoted context omitted.

I hate it when button mashers are in charge of incident response. Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.

To be fair, if your rollbacks are cheap (1-3 button pushes), safe, and automated, it doesn't really hurt to try a rollback while you look for other causes.

Most of my production experience precedes CI/CD as a concept by a number of years. You still don't know though if the rollback fixed it or if just bouncing the software alone fixed some kind of brownout, and you may not be able to replicate what happened and get stuck unsure of what the bug was and be in a state where you don't know if you can roll forward again or not. Then a developer throws a bit of spackle on the code to address what the problem "might" be -- then when you roll forward people assume that fixed it, which it might not, which is how cargo cults get formed.

Re: Post-mortem for last week's incident at Kagi

#164

Post by JohnMakin »

dmoy wrote: post #162
lamontcg wrote: post #151

Earlier quoted context omitted.

I hate it when button mashers are in charge of incident response. Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.

To be fair, if your rollbacks are cheap (1-3 button pushes), safe, and automated, it doesn't really hurt to try a rollback while you look for other causes.

Code changes should be trivial to roll back. Infra changes can be very difficult to roll back, however.

Re: Post-mortem for last week's incident at Kagi

#165

Post by xigoi »

_cenw wrote: post #144
MyFedora wrote: post #141

> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. Disclose that it's fair use upfront. Leveraging unlimited searches =/= Abuse. Never bait and switch unlimited to fair use, nobody reads fine print. Customers who pay for unlimited expect the service to scale the infrastructure to handle whatever they throw at it. I think that's a reasonable ex…

the presence of a paid search API should be enough of a hint that a $10 account trying to read back the entire index is not going to be tolerated. https://help.kagi.com/kagi/api/search.html though the lack of alerting and the lack of consideration for bad actors in general (just look for the kagi feedback thread on suicide results, you can kagi it) seems pretty consistent

What does this have to do with the suicide debate?

Re: Post-mortem for last week's incident at Kagi

#166

Post by ok_dad »

dylan604 wrote: post #94
DiggyJohnson wrote: post #92

Earlier quoted context omitted.

You definitely do not have to take stupid ideas like that seriously. Simply your life.

A little more credit to people for making a funny and a little less assumptions that people are that boneheaded would make the world a better place to be sure. I know for one, I rather enjoy being able to see the humor and having a laugh rather than getting all enraged and venting on the internet to show some sort of self perceived superiority.

Nah the ones spewing self perceived superiority are the folks who would call interns red shirts or make other off color jokes that end up hurting others and don’t even know it.

Re: Post-mortem for last week's incident at Kagi

#167

Post by jabradoodle »

darkwater wrote: post #138
jabradoodle wrote: post #135

Earlier quoted context omitted.

You need to validate your inputs and return 4xx

Yeah and you also shall not write bugs in your code. Real world has bugs, even trivial ones.

If your service is returning 5xx, that is the the definition of a server error, of course that is degraded service. Instead we have pointless dashboards that are green an hour after everything is broken.

Returning 4xx on a client error isn't hard and is usually handled largely by your framework of choice.

Your argument is a strawman

Re: Post-mortem for last week's incident at Kagi

#168

Post by tetha »

callalex wrote: post #105
tetha wrote: post #36

Earlier quoted context omitted.

Mh, I work quite a bit in the OPs-side and monitoring and observability are part of my job, for a bit of time now too. I'll say: Effective observability, monitoring and alerting of complex systems is a really hard problem. Like, you look at a graph of a metric, and there are spikes. But... are the spikes even abnormal? Are the spikes caused by the layer below, because our storage array is failing? Are the spikes caus…

Running voodoo analysis on graph spikes is indeed a fool’s errand. What you really need is load testing on every component of your system, and alerts for when you approach known, tested limits. Of course this is easier said than done and things will still be missed, but I’ve done both approaches and only one of them had pagers needlessly waking me in the middle of the night enough to go on sleepless swearing rants to…

Yeh. Or, during a complex investigation, you need to setup a hypothesis explaining these spikes in order to eventually establish causality surrounding these spikes. And once you have causality, you can start fixing.

For example, I've had a disagreement with another engineer there during a larger outage. We eventually got to the idea: If we click that button in the application, the database dies a violent death. Their first reaction was: So we never click that button again. My reaction was: We put all attention on the DB and click that button a couple of times.

If we can reliably trigger misbehavior of the system, we're back in control. If we're scared, we're not in control.

Re: Post-mortem for last week's incident at Kagi

#169

Post by mightyham »

saagarjha wrote: post #158
mightyham wrote: post #152

Earlier quoted context omitted.

No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.

I feel like it is a lot more absurd to not reach out to the resources that are equipped specifically to handle problems like yours.

My point is that this is a service that I was curious about but have no real need for. I'm mostly satisfied by using Google or DDG. As a customer, it's pretty absurd that the onus is on me to spend a fairly significant amount of time contacting support in order to simply evaluate a niche product. Furthermore, the company is a tech company, so the fact that their authentication is bugged, seems like more than enough of a reason to not spend any more of my time evaluating their service. I literally can't think of any web based products I currently use which have, at any point, had bugged authentication.

Re: Post-mortem for last week's incident at Kagi

#170

Post by zbentley »

jabradoodle wrote: post #167
darkwater wrote: post #138

Earlier quoted context omitted.

Yeah and you also shall not write bugs in your code. Real world has bugs, even trivial ones.

If your service is returning 5xx, that is the the definition of a server error, of course that is degraded service. Instead we have pointless dashboards that are green an hour after everything is broken. Returning 4xx on a client error isn't hard and is usually handled largely by your framework of choice. Your argument is a strawman

> Returning 4xx on a client error isn't hard and is usually handled largely by your framework of choice.

> Your argument is a strawman

That's....super not true. Malformed requests with gibberish (or, more likely, hacker/pentest- generated) headers will cause e.g. Django to return 5xx easily.

That's just the example I'm familiar with, but cursory searching indicates reports of similar failures emitted by core framework or standard middleware code for Rails, Next.js, and Spring.

Post reply on HN