Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Retries – An interactive study of request retry methods

encore.dev

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Retries – An interactive study of request retry methods

#31

Post by samwho »

korm wrote: post #29
cowsandmilk wrote: post #23

Earlier quoted context omitted.

Your response makes it sound like you think circuit breakers are server side and not related to retries. They are not; they are a client-side mitigation that are a critical part of a mature retry library.

The client can track its own error rate to the service, but it would need information from a server to get the overall health of the service, which is what the author probably means. Furthermore the load balancer can add a Retry-After header to have more control over the client's retries.

I think I've misunderstood what circuit breakers are for years! I did indeed think they were a server-side mechanism. The original commenter's description of them is great, you can essentially create a heuristic based on the observed behaviour of the server and decide against overwhelming it further if you think it's unhealthy.

TIL! Seems like it can have tricky emergent behaviour. I bet if you implement it wrong you can end up in very weird situations. I should visualise it. :)

Re: Retries – An interactive study of request retry methods

#32

Post by lclarkmichalek »

samwho wrote: post #31
korm wrote: post #29

Earlier quoted context omitted.

The client can track its own error rate to the service, but it would need information from a server to get the overall health of the service, which is what the author probably means. Furthermore the load balancer can add a Retry-After header to have more control over the client's retries.

I think I've misunderstood what circuit breakers are for years! I did indeed think they were a server-side mechanism. The original commenter's description of them is great, you can essentially create a heuristic based on the observed behaviour of the server and decide against overwhelming it further if you think it's unhealthy. TIL! Seems like it can have tricky emergent behaviour. I bet if you implement it wrong you…

I mean, they can and should be both. Local decisions can be cheap, and very simple to implement. But global decisions can be smarter, and more predictable. In my experience, it's incredibly hard to make good decisions in pathological situations locally, as you often don't know you're in a pathological situation with only local data. But local data is often enough to "do less harm" :)

Re: Retries – An interactive study of request retry methods

#33

Post by fooey »

sesm wrote: post #24

Summary of the article: use exponential backoff + jitter for retry intervals. What author didn’t mention: sometimes you want to add jitter to delay the first request too, if the request happens immediately after some event from server (like server waking up). If you don’t do this, you may crash the server, and if your exponential backoff counter is not global you can even put server into cyclic restart.

Yup, classic Thundering Herd Problem

Re: Retries – An interactive study of request retry methods

#34

Post by cratermoon »

I worked at a company with a self-inflicted wound related to retries.

At some point in the distant (internet time) past, a sales engineer, or the equivalent, had written a sample script to demonstrate basic uses of the API. As many of you quickly guessed, customers went on a copy/paste rampage and put this sample script into production.

The script went into a tight loop on failure, naively using a simple library that did not include any back-off or retry in the request. I'm not deeply familiar with how the company dealt with this situation. I am aware there was a complex load balancing system across distributed infrastructure, but also, just a lot of horsepower.

Lesson for anyone offering an API product: don't hand out example code with a self-own, because it will become someone's production code.

Re: Retries – An interactive study of request retry methods

#35

Post by joshka »

For a lot of things, retry once and only once (at the outermost layer to avoid multiplicative amplification) is more correct. At a large enough scale, failing twice is often significantly (like 90%+) correlated with the likelihood of failing a third time regardless of backoff / jitter. This means that the second retry only serves to add more load to an already failing service.

Re: Retries – An interactive study of request retry methods

#36

Post by davidw »

I have been thinking about queueing theory lately. I don't have the math abilities to do anything deep with it, but it seems like even basic applications of certain things could prove valuable in real world situations where people are just kind of winging it with resource allocation.

Re: Retries – An interactive study of request retry methods

#37

Post by tomwt »

joshka wrote: post #35

For a lot of things, retry once and only once (at the outermost layer to avoid multiplicative amplification) is more correct. At a large enough scale, failing twice is often significantly (like 90%+) correlated with the likelihood of failing a third time regardless of backoff / jitter. This means that the second retry only serves to add more load to an already failing service.

Retrying end-to-end instead of stepwise greatly reduces the reliability of a process with a reasonable number of steps.

That being said, processes should ideally be failing in ways which make it clear whether an error is retryable or not.

Re: Retries – An interactive study of request retry methods

#38

Post by samwho »

samwho wrote: post #13
codebeaker wrote: post #11

Earlier quoted context omitted.

What technology did you use for the animations? I've a bunch of itches I'd like to scratch that would be improved by having some canvas animated explainers or UI but I never clicked with anything. D3 back in the day. A rudimentary look in the source code showed a element but I'm not up to date enough with web standards to guess where to look for that in your JS bundle to guess at the framework!

It uses PixiJS ( https://pixijs.com/ ) for the 2D rendering and GSAP3 ( https://gsap.com/ ) for the animation. The blocks are custom HTMl elements ( https://developer.mozilla.org/en-US/docs/Web/API/Web_compone... ) which I use to encapsulate the logic. I've been thinking about creating a separate repo to house the source code of posts I've finished so people can see it. I don't like all the bundling and minification…

I've uploaded the code for all of my visualisation posts here: https://github.com/samwho/visualisations.

Enjoy! :)

Re: Retries – An interactive study of request retry methods

#39

Post by sroussey »

whenlambo wrote: post #25
sesm wrote: post #24

Summary of the article: use exponential backoff + jitter for retry intervals. What author didn’t mention: sometimes you want to add jitter to delay the first request too, if the request happens immediately after some event from server (like server waking up). If you don’t do this, you may crash the server, and if your exponential backoff counter is not global you can even put server into cyclic restart.

If you can crash the server with an improperly timed request, then you have a much bigger problem than client-side stuff.

True, but you can imagine something like a websocket to all clients getting reset and everyone re-connecting, re-authenticating, and getting a new payload.

Re: Retries – An interactive study of request retry methods

#40

Post by xer »

joshka wrote: post #35

For a lot of things, retry once and only once (at the outermost layer to avoid multiplicative amplification) is more correct. At a large enough scale, failing twice is often significantly (like 90%+) correlated with the likelihood of failing a third time regardless of backoff / jitter. This means that the second retry only serves to add more load to an already failing service.

Correct. It's also the case that human generated requests will lose their relevance within seconds, a quick retry is all it's worth. As for machine generated requests a dead letter queue would make more sense, poor engineered backend services would OOM and well-engineered would load shed, if the requests are queued on the application servers they are doomed to be lost anyway.
Post reply on HN