31–40 of 48 posts
I wonder if anyone has switched algorithms after hitting real-world scaling issues with one of those? Curious if there are any “gotchas” that only show up at scale. I only have experience with fixed window rate limiting
We used leaky bucket IIRC and the issue I saw was that the distributed aspect of it was coded incorrectly and so depending on the node you hit you were rate-limited or not :facepalm:
I wonder if anyone has switched algorithms after hitting real-world scaling issues with one of those? Curious if there are any “gotchas” that only show up at scale. I only have experience with fixed window rate limiting
Something I’ve long wondered is why you never hear about rate limiting algorithms that are based on the cost to serve the request or algorithms that dynamically learn the capacity of the system and give everyone a fair share. In the field of router buffer management, there are algorithms like Stochastic Fair Blue, which does the latter, but is somewhat hard to apply to HTTP because you’d have to define a success/fail…
For situations where eventual consistency is good enough, you can run a task in a loop that tries every n seconds to update a quantity. But as you say that can also saturate, so what you really want is for the task to update, then wait m seconds and go again, where m is more than the time you expect the task to complete in (I don't think voluntary rate limiting on the client side gets enough column inches. Peer to peer you end up footguning yourself if you bite off more than you can chew, and if you start stalling on responses then you gum up the server as well.
Instead of storing the current number of tokens, you instead store when the bucket will be full. If you take a token from the bucket, you increment the timestamp accordingly by 1/rps. The only complication is it the filled timestamp was in the past, you have to first update it with the current timestamp to avoid overfilling.
What's even nicer is that it doubles as a throttle implementation rather than just a rate limiter. You know the bucket is empty if you compute empty_at=filled_at-(max_tokens/rps) which is still in the future. From that calculation you now know when it will have capacity again, so you can sleep accordingly. If you use a queue before the gcra, it then starts sowing down new connections rather than just dropping them.
You should still have a limit on the queue, but it's nice in that it can gracefully turn from token bucket into leaky bucket.
> Follow Sagyam's Blog's journey > By following, you'll have instant access to our new posts in your feed. > Continue with Google > More options As soon as I see this in a blog, I quit tab. Why do authors do this to themselves?
Earlier quoted context omitted.
It really looks AI generated
Yes, I usually prompt (Claude, GPT and Deepseek),on my rough vision, and take ideas from all of them. They never quite get it right on their own. But for a code that's deploy and forget, AI generated code is good enough.
Earlier quoted context omitted.
- I like bullet points, they are easy to read. - "Working" I wanted to keep things consistent. - Content getting cut was a limitation of iframe. Most blogging platform don't allows you to embed another page. This was best I could do given the limitation. - I do use AI to bounce ideas, but a lot of effort went into getting the apps working as intended.
Why "Working?" It's unclear what that means. Is it supposed to say, "How it works"?
No mention of CGNAT which caused me many problems at a previous role?
> Follow Sagyam's Blog's journey > By following, you'll have instant access to our new posts in your feed. > Continue with Google > More options As soon as I see this in a blog, I quit tab. Why do authors do this to themselves?
- Medium which paywalls the article and forces you to sign up just to read.
- Substack has same problem, it's great for funneling people to your paid newsletter but there is a sign up banner as soon the page loads.
- Build your own and miss out on the social aspect and there's no proof if the numbers are real.
My favourite algorithm is generic cell rate algorithm (GCRA). It works like token bucket in this post. The implementation is dead simple and requires no background tasks and needs very minimal state. Instead of storing the current number of tokens, you instead store when the bucket will be full. If you take a token from the bucket, you increment the timestamp accordingly by 1/rps. The only complication is it the fill…