Post-mortem for last week's incident at Kagi • Hacker News Board

Re: Post-mortem for last week's incident at Kagi

#181

Post by laeri » Thu, Jan 18, 2024, 5:02 PM UTC

darkwater wrote: post #122

laeri wrote: post #117

Earlier quoted context omitted.

A malformed request should not lead to a 500, they should be handled and validated.

Well, in the real world it might. It should trigger a bug creation and a fix to the code, but not an incident. Now all of a sudden to decide this you need more complex and/or specific queries in your monitoring system (or a good ML-based alert system), so complexity is already going up.

Query input validation is nearly a solved problem. If you don't I would argue this is an incident if in this case 500's are returned.

Re: Post-mortem for last week's incident at Kagi

#182

Post by laeri » Thu, Jan 18, 2024, 5:05 PM UTC

zbentley wrote: post #170

jabradoodle wrote: post #167

Earlier quoted context omitted.

If your service is returning 5xx, that is the the definition of a server error, of course that is degraded service. Instead we have pointless dashboards that are green an hour after everything is broken. Returning 4xx on a client error isn't hard and is usually handled largely by your framework of choice. Your argument is a strawman

> Returning 4xx on a client error isn't hard and is usually handled largely by your framework of choice. > Your argument is a strawman That's....super not true. Malformed requests with gibberish (or, more likely, hacker/pentest- generated) headers will cause e.g. Django to return 5xx easily. That's just the example I'm familiar with, but cursory searching indicates reports of similar failures emitted by core framewor…

If input validation is not present in your framework of choice then the framework clearly has problems.

If you do not validate your inputs properly I am not sure what you are doing when you have a user facing applications of this size. Validating inputs is the lowest hanging fruit for preventing hacking threats.

Re: Post-mortem for last week's incident at Kagi

#183

Post by vfxdev » Thu, Jan 18, 2024, 8:59 PM UTC

This incidcent also shows that no security audit was ever performed on this platform, as a DoS vulnerability would've been one of the first findings there. Makes you wonder what other vulnerabilities are in their infrastructure...

Re: Post-mortem for last week's incident at Kagi

#184

Post by therealdrag0 » Fri, Jan 19, 2024, 7:55 AM UTC

eep_social wrote: post #59

fragmede wrote: post #35

Earlier quoted context omitted.

This is easy for me to say in hindsight, but a graph of queries per user, for the top 100 accounts would have lit up the issue like a Christmas tree. But it's the kind of chart you stuff on the full page of charts that are usually not interesting. So many problems have been avoided by "huh, that looks weird" on a dashboard. In a more targeted search, asking Splunk to show out where the traffic is coming from on a map…

If daily volume is ~400k a sharp 60k spike should show even on a straight rps dashboard. I suspect that didn’t exist because most cloudy SaaS tools don’t seem to put any emphasis on traffic over a particular unit time, like in the Cloudflare site dashboard one must hover over the graph and subtract timestamps to find out if the number of request is per hour, minute, etc. Splunk is similarly bad — the longer the perio…

For splunk you can set unit by setting span like this | timechart span=30s count by X

Re: Post-mortem for last week's incident at Kagi

#185

Post by lanstin » Sat, Jan 20, 2024, 5:54 AM UTC

bostik wrote: post #172

rwiggins wrote: post #106

Earlier quoted context omitted.

You know, in retrospect, I think Kagi expects O(thousands) searches per month per user, so doing per-user usage accounting in the DB is fine -- thanks to row-level locking. Well, at least until you get a user who does 60k "in a short time period"... :-)

It's the outliers and "surely nobody could be THAT awful" that kill you. Every time.

I once had a stock alert product running on a backend I wrote. One person signed up for alerts for every single Nasdaq ticker there was. We didn’t expect that.

Re: Post-mortem for last week's incident at Kagi

#186

Post by nijave » Sun, Jan 21, 2024, 11:32 AM UTC

_cenw wrote: post #144

MyFedora wrote: post #141

> While we do offer unlimited searches to our users, such volume was a clear abuse of our platform, against our terms of use. Disclose that it's fair use upfront. Leveraging unlimited searches =/= Abuse. Never bait and switch unlimited to fair use, nobody reads fine print. Customers who pay for unlimited expect the service to scale the infrastructure to handle whatever they throw at it. I think that's a reasonable ex…

the presence of a paid search API should be enough of a hint that a $10 account trying to read back the entire index is not going to be tolerated. https://help.kagi.com/kagi/api/search.html though the lack of alerting and the lack of consideration for bad actors in general (just look for the kagi feedback thread on suicide results, you can kagi it) seems pretty consistent

Not necessarily. It's common to charge business users a different amount (>0) than personal use