Rate limiter for LLMs outperforms exponential backoff
1–2 of 2 posts
Re: Rate limiter for LLMs outperforms exponential backoff
#2rateLLMiter is a Python rate limiter that smoothes out requests to LLM APIs to get faster, more consistent performance. It uses a ticket bucket algorithm rather than the usual exponential backoff.