Retries – An interactive study of request retry methods
1–10 of 56 posts
Re: Retries – An interactive study of request retry methods
#2Thanks for sharing!
I’m the author of this post, and happy to answer any questions :)
Re: Retries – An interactive study of request retry methods
#3Really nice animations, I especially liked the demonstration of the effect that after some servers will "explode", any server that will be restarted will automatically be DoS'ed until we'll throw a bunch of extra temporary servers into the system. Thanks.
Re: Retries – An interactive study of request retry methods
#4Really nice animations, I especially liked the demonstration of the effect that after some servers will "explode", any server that will be restarted will automatically be DoS'ed until we'll throw a bunch of extra temporary servers into the system. Thanks.
Yeah! An insidious problem that’s not obvious when you’re picking a retry interval.
I had fun with the details of the explosion animation. When it explodes, the number of requests that come out is the actual number of in-progress requests.
Re: Retries – An interactive study of request retry methods
#5A must-read (or rather: must-see) for anyone who thinks exponential backoff is overrated.
Re: Retries – An interactive study of request retry methods
#6Remember to limit the exponential backoff interval if you are not limiting the number of retries
Re: Retries – An interactive study of request retry methods
#7The animations are so cool!!!
In general the phenomena is known as _metastable failure_ that could be triggered when there are more things to do during failure than normal run.
With retry, the client do more work within the same amount of time, compared to doing nothing or doing exponential backoff.
Re: Retries – An interactive study of request retry methods
#8Thanks for sharing! I’m the author of this post, and happy to answer any questions :)
There's a subtle insight that could be added to the post if you consider worth it, and it's something that's actually there already, but difficult to realize: Clients in your simulation have an absolute maximum number of retries.
I noticed this mid-read, when looking at one of the animations with 28 clients, that they would hammer the server but suddenly go into wait state, without apparent reason.
Later in the final animation with debug mode enabled, the reason becomes apparent for those who click on the Controls button:
Retry Strategy > Max Attempts = 10
It makes sense, because in the worst case when everything goes wrong, a client should reach a point where it desists and just aborts with a "service not available" error.
Re: Retries – An interactive study of request retry methods
#9This still isn't what I'd call "safe". Retries are amazing at supporting clients in handling temporary issues, but horrible for helping them deal with consistently overloaded servers. While jitter & exponential backoff help with the timing, they don't reduce the overall load sent to the service.
The next step is usually local circuit breakers. The two easiest to implement are terminating the request if the error rate to the service over the last is greater than x%, and terminating the request (or disabling retries) if the % of requests that are retries over the last is greater than x%.
i.e. don't bother sending a request if 70% of requests have errored in the last minute, and don't bother retrying if 50% of the requests we've sent in the last minute have already been retries.
Google SRE book describes lots of other basic techniques to make retries safe.
Re: Retries – An interactive study of request retry methods
#10This still isn't what I'd call "safe". Retries are amazing at supporting clients in handling temporary issues, but horrible for helping them deal with consistently overloaded servers. While jitter & exponential backoff help with the timing, they don't reduce the overall load sent to the service. The next step is usually local circuit breakers. The two easiest to implement are terminating the request if the error rate…
Totally! Thanks for bringing those up. I tried to keep the scope specifically on retries and client-side mitigation. There's a whole bunch of cool stuff to visualise on the server-side, and I'm hoping to get to it in the future.