Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-mortem for last week's incident at Kagi

status.kagi.com

Post reply on HN
Print view View original

151–160 of 186 posts

Re: Post-mortem for last week's incident at Kagi

#151

Post by lamontcg »

JohnMakin wrote: post #49
blantonl wrote: post #24

At first, by what turned out to be a complete coincidence, the incident occurred at precisely the same time that we were performing an infrastructure upgrade to our VMs with additional RAM resources I can assure you that these "coincidences" happen all the time, and will cause you to question your very existence when you are troubleshooting them. And if you panic while questioning your very existence, you'll invariab…

Completely agree. I’ve triaged many outages with varying degrees of severity in my career so far, and the worst ones were always caused by someone panic-jumping onto some red herring rather than coming up with a sensible explanation as to why that would be a fix other than “it happened at the same time.” I have a saying I really like to throw around, which is “if you don’t know why/how you fixed it, you may not have”

I hate it when button mashers are in charge of incident response.

Things like when traffic ramps up in the morning and the site falls over and someone in charge blames the deployment the night before and screams "rollback", and then it eventually turns out to have nothing at all to do with the deployment.

Re: Post-mortem for last week's incident at Kagi

#152

Post by mightyham »

freediver wrote: post #87
mightyham wrote: post #74

As much as people gush over Kagi on HN, I still have yet to actually try it because I cannot for the life of me get authentication to work. Even after immediately resetting my password, I get an "incorrect email or password" or "try again later" error on the login page. I've tried at least 3 times over the last few months with the same results each time. If such a fundamental part of a web service company's website i…

Have you tried contacting Kagi support to help debug the issue?

No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.

Re: Post-mortem for last week's incident at Kagi

#153

Post by sjsdaiuasgdia »

TeeWEE wrote: post #110
phyzome wrote: post #84

Earlier quoted context omitted.

As an engineer on call, I have been in this conversation so many times: "Hey, should we go red?" "I don't know, are we sure it's an outage, or just a metrics issue?" "How many users are affected again?" "I can check, but I'm trying to read stack traces right now." "Look, can we just report the issue?" "Not sure which services to list in the outage" ...and so on. Basically, putting anything up on the status page is a…

Connect your status page to actual metrics and decide a treshold for downtime. Boom you’re done.

Stage 1: Status is manually set. There may be various metrics around what requires an update, and there may be one or more layers of approval needed.

Problems: Delayed or missed updates. Customers complain that you're not being honest about outages.

Stage 2: Status is automatically set based on the outcome of some monitoring check or functional test.

Problems: Any issue with the system that performs the "up or not?" source of truth test can result in a status change regardless of whether an actual problem exists. "Override automatic status updates" becomes one of the first steps performed during incident response, turning this into "status is manually set, but with extra steps". Customers complain that you're not being honest about outages and latency still sucks.

Stage 3: Status is automatically set based on a consensus of results from tests run from multiple points scattered across the public internet.

Problems: You now have a network of remote nodes to maintain yourself or pay someone else to maintain. The more reliable you want this monitoring to be, the more you need to spend. The cost justification discussions in an enterprise get harder as that cost rises. Meanwhile, many customers continue to say you're not being honest because they can't tell the difference between a local issue and an actual outage. Some customers might notice better alignment between the status page and their experience, but they're content, so they have little motivation to reach out and thank you for the honesty.

Eventually, the monitoring service gets axed because we can just manually update the status page after all.

Stage 4: Status is manually set. There may be various metrics around what requires an update, and there may be one or more layers of approval needed.

Re: Post-mortem for last week's incident at Kagi

#154

Post by eklavya »

lesam wrote: post #131
tnolet wrote: post #127

Earlier quoted context omitted.

Oh yes, the amount of times I rolled back a change during an outage that had nothing to do with the outage... It's a critical skill for engineers: being able to critically reason, debug and "test in isolation" any changes that address outages. Much harder than it seems and typically a "senior" skill to have.

Assuming you have a tested rollback strategy, I’m all for rolling back first and asking questions later. It at least can often eliminate the rolled back change as the cause, even if it doesn’t fix the problem.

That's essentially on call 101, you try to mitigate as fast as possible and root cause later when you are not half asleep. Surprised to see some comments here.

Re: Post-mortem for last week's incident at Kagi

#155

Post by lopkeny12ko »

karlshea wrote: post #98
lopkeny12ko wrote: post #95

> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. As such, we removed searching capability from the offending accounts. Don't advertise something as unlimited if it's not actually unlimited.

Unlimited searches as a real user is not the same thing as running a bot against the service.

What difference does it make? So would it be "legitimate use" if I sat at a computer and manually clicked 100 times to do something, while it is "abuse" for me to write a script that automates exactly the same thing? Is it only considered abuse if the action isn't tedious and tiring for me as a user?

Re: Post-mortem for last week's incident at Kagi

#157

Post by returningfory2 »

mightyham wrote: post #152
freediver wrote: post #87

Earlier quoted context omitted.

Have you tried contacting Kagi support to help debug the issue?

No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.

I'm confident that account login is working for most people. Potentially something weird has happened with you account and it requires manual intervention. This is exactly what support is for - handling weird situations.

Re: Post-mortem for last week's incident at Kagi

#158

Post by saagarjha »

mightyham wrote: post #152
freediver wrote: post #87

Earlier quoted context omitted.

Have you tried contacting Kagi support to help debug the issue?

No and I think it's kind of absurd that I would have to reach out to support staff so that I can get something as basic as account log in working correctly.

I feel like it is a lot more absurd to not reach out to the resources that are equipped specifically to handle problems like yours.

Re: Post-mortem for last week's incident at Kagi

#159

Post by saagarjha »

lopkeny12ko wrote: post #155
karlshea wrote: post #98

Earlier quoted context omitted.

Unlimited searches as a real user is not the same thing as running a bot against the service.

What difference does it make? So would it be "legitimate use" if I sat at a computer and manually clicked 100 times to do something, while it is "abuse" for me to write a script that automates exactly the same thing? Is it only considered abuse if the action isn't tedious and tiring for me as a user?

Yes. This is why a buffet that offers you unlimited food requires you to actually eat it, and not just pack it into boxes to take home.

Re: Post-mortem for last week's incident at Kagi

#160

Post by saagarjha »

WhereIsTheTruth wrote: post #126

If 60k requests hurts your infra, then your infra is poorly built and you cheap out on resources Also straight out blocking the account instead of rate-limiting? Yeah, poor infrastructure Kagi is aimed at power users, they should do and provide better

You spend extra on resources and people lambast you for your bloated and expensive stack. You run it lean and people like you say you’re cheaping out.
Post reply on HN