An Alternative Approach to Rate Limiting

Post by nullnilvoid » Thu, Apr 13, 2017, 12:13 AM UTC

irgeek wrote: post #26

In response, we implemented a shadow ban: On the surface, the attackers continued to receive a 200 HTTP response code, but behind the scenes we simply stopped sending document invitations after they exceeded the rate limit. And right there they broke the service for legitimate users. Totally unacceptable collateral damage IMHO.

Post by vacri » Thu, Apr 13, 2017, 12:17 AM UTC

nullnilvoid wrote: post #31

irgeek wrote: post #26

In response, we implemented a shadow ban: On the surface, the attackers continued to receive a 200 HTTP response code, but behind the scenes we simply stopped sending document invitations after they exceeded the rate limit. And right there they broke the service for legitimate users. Totally unacceptable collateral damage IMHO.

The problem with rate limiting is to distinguish a normal user from a spammer. A normal user can send more requests than usual sometimes. If a normal user gets rate-limited by mistake, you are going to get lots of upset users.

Post by ioddly » Thu, Apr 13, 2017, 12:41 AM UTC

vacri wrote: post #32

irgeek wrote: post #26

In response, we implemented a shadow ban: On the surface, the attackers continued to receive a 200 HTTP response code, but behind the scenes we simply stopped sending document invitations after they exceeded the rate limit. And right there they broke the service for legitimate users. Totally unacceptable collateral damage IMHO.

Wow, that's really bad. I get shivers at the thought of trying to troubleshoot that as a client.

Post by Animats » Thu, Apr 13, 2017, 1:38 AM UTC

antirez wrote: post #24

Animats wrote: post #17

Earlier quoted context omitted.

This is why I've suggested, and used, fair queuing for rate limiting. Any source can make lots of requests, but they queue up by IP address. (In practice, you have queues organized by hashed IP address, which you service round-robin. That's how routers do fair queuing.) If one queue gets very long, then you start sending 429 errors for new adds to that queue. Otherwise, the requests from the heavy load source just ta…

Neat idea, but if you hash the same as the spammer?

Post by joneholland » Thu, Apr 13, 2017, 1:39 AM UTC

drchickensalad wrote: post #38

joneholland wrote: post #36

You know you can implement a token bucket that doesn't share state between your API servers, in about 10 lines of code, using just a in memory map. Your incoming requests should be balanced across all of the servers so you just derive the allowed throughput and divide by the number front ends....

You have to send all the traffic from one client to one server then right? Seems not without heavy drawbacks. Otherwise you can easily get them to hit their limit super early with bad luck.