Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotence now prevents pain later

ericlathrop.com

Post reply on HN
Print view View original

41–50 of 133 posts

Re: Idempotence now prevents pain later

#41

Post by alex_young »

tigger0jk wrote: post #35

> 1. Query the database to find all dormant accounts with a balance, which haven't been charged the fee this month. > 2. Charge each of these accounts a fee > 3. Setup a cron job to run this every hour Note that if this job ever runs successfully, but takes more than an hour, you will double-count. Can easily happen if the box running these crons is overloaded. One fix is to automatically halt the job after 55 minute…

Sounds like a good reason to use a pidfile or mutex so you can eliminate the possibility of any concurrent jobs.

Re: Idempotence now prevents pain later

#42

Post by adrusi »

jayd16 wrote: post #29
bob1029 wrote: post #24

I feel like immutability and a 100% leak-proof layer of domain methods which in turn manage domain mutations would ultimately bring more value than explicitly adding idempotence throughout. If I have an idempotent method like "CreateCustomerRecord", this can cause a lot of pain for audit features and other aspects of the domain model if it is internally making determinations about whether to actually create or silent…

This can create false negatives when a request must be retried due to network failure but actually succeeded because the failure was during the response. Idempotency is great for "debouncing" requests. If you want to tell difference between identical requests that are different transactions, add a unique transaction id of some kind.

There's a place for idempotency tokens. They're relatively easy to retrofit onto old systems, and occasionally they are the best way to go about making changes idempotent, but they should be a red flag – an indication that you should step back and see if maybe you can redesign an API to make idempotency a natural guarantee rather than something you artificially strap on with a token. As a rule of thumb, I would always mention the idea of adding an idempotency token, and prompt for alternatives, with all stakeholders present.

(In this particular example, I agree that an idempotency token is probably the way to go, as otherwise callers would need to somehow distinguish between their callers trying to create a duplicate account, versus something going wrong with them resulting in duplicate requests. I just have too often seen developers conflate "idempotency" with "use an idempotency token)

Re: Idempotence now prevents pain later

#43

Post by Animats »

elcomet wrote: post #26
Animats wrote: post #15

Earlier quoted context omitted.

It's a basic property of HTTP "GET". Or it's supposed to be. "GET" is not supposed to change server site state. That's what "POST" is for. This matters if there's a cache in the middle, since caches tend to assume that GET requests are idempotent and can be served from cache. Cloudflare assumes that. POST requests have to go through to the real server.

Get is idempotent because it's the identity function. It does not change the state of the data. So it's a trivial case of idempotency. A more interesting function is PUT (idempotent) vs POST (not idempotent)

GET is supposed to be idempotent. But there are sites and APIs where URL parameters make something happen on a GET.

Re: Idempotence now prevents pain later

#44

Post by Pxtl »

I handle the deployment pipeline at my office and I've got a hard rule that every step in the CI chain has to be idempotent. Considering how many points of failure there can be in CI, being able to retry without undoing the mess of the previous deployment is a lifesaver.

Re: Idempotence now prevents pain later

#45

Post by totetsu »

dang wrote: post #2

Possibly related past threads: What Is Idempotence? - https://news.ycombinator.com/item?id=19570815 - April 2019 (51 comments) Idempotence: What is it and why should I care? - https://news.ycombinator.com/item?id=17804617 - Aug 2018 (73 comments) You know how HTTP GET requests are meant to be idempotent? - https://news.ycombinator.com/item?id=16964907 - May 2018 (304 comments) Implementing Stripe-Like Idempotency Key…

When the same thread is appears on HN twice does it get the same results?

Re: Idempotence now prevents pain later

#46

Post by thrwyoilarticle »

totetsu wrote: post #45
dang wrote: post #2

Possibly related past threads: What Is Idempotence? - https://news.ycombinator.com/item?id=19570815 - April 2019 (51 comments) Idempotence: What is it and why should I care? - https://news.ycombinator.com/item?id=17804617 - Aug 2018 (73 comments) You know how HTTP GET requests are meant to be idempotent? - https://news.ycombinator.com/item?id=16964907 - May 2018 (304 comments) Implementing Stripe-Like Idempotency Key…

When the same thread is appears on HN twice does it get the same results?

This kind of thing is allowed according to the FAQs. I think the links are just additional reading.

Re: Idempotence now prevents pain later

#47

Post by josephcsible »

thrwyoilarticle wrote: post #46
totetsu wrote: post #45

Earlier quoted context omitted.

When the same thread is appears on HN twice does it get the same results?

This kind of thing is allowed according to the FAQs. I think the links are just additional reading.

I think you missed a joke there.

Re: Idempotence now prevents pain later

#49

Post by [deleted] »

jayd16 wrote: post #23
brsg wrote: post #20

Idempotency is a pretty critical concept in system design, and I think most developers have run into issues related to it even if they aren't directly familiar with the term. To give another simple example as the OP - Suppose you have a product that relies on time series data. For demo purposes you might create a curated data set to present to clients, but the presenter doesn't want to show data from 2019 as the "mos…

I don't think B is technically idempotent either. Change still occurs but with minimal difference. You cannot cache the results and use them again next week. An idempotent change would be to pass in the current time instead of checking system time. In this case, as long as the input is the same, the result is the same. You could use cached results, but most likely you want to use new inputs.

[deleted]

Re: Idempotence now prevents pain later

#50

Post by eric4smith »

Years ago when I was a green programmer filled with p*ss and vinegar - I had to write a system to send a few million emails every day.

That’s when I came across Qmail and studied it’s design. Qmail was designed so well that it literally absolutely would not send an email twice.

But my scheduler did.

To send a few hundred thousand emails more than once is not good is not good for your reputation lol.

It taught me a valuable lesson about Idempotency from an impressionable stage of my career and now, with creating workers running under Sidekiq, Oban and other job schedulers that run very frequently, this has become quite useful.

At one point I used to lean on Redis to “help” me be more idempotent. But with the high throughput of some systems I was working on, even redis with its atomic nature of put and get and handy dandy sets and lists, would have race conditions.

Now I hardly touch redis because my skills in making sure things get processed just once have improved greatly. And I can just use PostgreSQL and my code.

Being Idempotent from the beginning is now the center of my life.

Post reply on HN