Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotency

berkansasmaz.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Idempotency

#31

Post by jkubicek »

My day-job is Data Engineering. We've carefully built our ETL patterns/tools/data structures to be idempotent. It's amazing how freeing this is. Dealing with cascading failures, intermittent failures, delays, re-runs and time-zones are all os much more manageable when we know that just re-running any given task will do the right thing.

It's also amazing how hard of a concept idempotent data pipelines are for some folks to understand, and how much effort they'll put into _avoiding_ idempotency.

Re: Idempotency

#32

Post by The_Colonel »

l33t7332273 wrote: post #21
koterpillar wrote: post #2

Reading this article is completely safe even if you already understand idempotency.

I want to meet the article that isn’t safe to read even if you’re familiar with its content.

Well, it's conceivable that you could end up creating a separate copy of the information in your brain's neural network.

Now imagine you keep getting interrupted in the middle, but really want to read it all, thus gradually filling the whole of your brain capacity.

Re: Idempotency

#33

Post by kroolik »

djbusby wrote: post #11
kroolik wrote: post #5

Come here to learn more details and maybe find answers to my questions. Unfortunately, the article is very modest explaining how to work with idempotency. Missing key answers like what about two requests in flight, how to store and query requests status reliably. The article mention requests should be ACID, but the diagram has caching storage. Would redis/dynamodb work here?

Redis has SETEX that will set if not existing. Can be used as a lock.

True, generic test-and-set ops can be used. But what about if there is already a request in-flight? Do you wait or fail? What is the experience working with either? What is the experience invalidation the cache key? And similar other questions :) So more practical than theoretical side to idempotency

Re: Idempotency

#34

Post by markerz »

l33t7332273 wrote: post #21
koterpillar wrote: post #2

Reading this article is completely safe even if you already understand idempotency.

I want to meet the article that isn’t safe to read even if you’re familiar with its content.

I can’t find it anymore, but there was someone who had some home automation where they could open or close their garage door from the internet. However it would occasionally open automatically. Eventually they realized it happens whenever they opened their browser. Apparently, they visited the page often enough for it to be favorited on their homepage. Unfortunately, it was written to act on any request, including GET requests and so whenever they opened his browser to the home page, it would fetch the favicon or preview and trigger the automation.

Re: Idempotency

#35

Post by charcircuit »

l33t7332273 wrote: post #21
koterpillar wrote: post #2

Reading this article is completely safe even if you already understand idempotency.

I want to meet the article that isn’t safe to read even if you’re familiar with its content.

The curse of being familiar with its content is that you know when the author of the article is giving bad advice, explaining wrong about how things work, or having an opposite opinion on a polarizing topic. If you are not familiar with the subject these things won't bother you.

Re: Idempotency

#36

Post by charcircuit »

kevinventullo wrote: post #18
yawboakye wrote: post #15

pet peeve: idempotency is badly named. idempotent literally translated is ‘same power.’ two or more things having the same power doesn’t necessarily mean they produce same, single result. if that were the case we wouldn’t need the special handling for idempotent instructions/commands/actions to ensure duplicates are not created.

I’m not totally sure, but in math “idempotent” elements are those for which x^2 = x. And so in software engineering it refers to operations for which doing them twice has the same effect as doing them once. Maybe this is what is meant by “same power”?

It refers to functions where f(x) = x.

Re: Idempotency

#37

Post by fluoridation »

markerz wrote: post #34
l33t7332273 wrote: post #21

Earlier quoted context omitted.

I want to meet the article that isn’t safe to read even if you’re familiar with its content.

I can’t find it anymore, but there was someone who had some home automation where they could open or close their garage door from the internet. However it would occasionally open automatically. Eventually they realized it happens whenever they opened their browser. Apparently, they visited the page often enough for it to be favorited on their homepage. Unfortunately, it was written to act on any request, including GE…

That's not unsafe, just insecure.

Re: Idempotency

#38

Post by linepogl »

And yet, the idempotent solution in the article is not idempotent.

The author proposes a solution that needs three steps: query a cache, execute, update the cache. These are not atomic and, therefore, not thread-safe. As a result, if a second request arrives before the first is finished, the operation will be executed again.

Re: Idempotency

#39

Post by Kim_Bruning »

jkubicek wrote: post #31

My day-job is Data Engineering. We've carefully built our ETL patterns/tools/data structures to be idempotent. It's amazing how freeing this is. Dealing with cascading failures, intermittent failures, delays, re-runs and time-zones are all os much more manageable when we know that just re-running any given task will do the right thing. It's also amazing how hard of a concept idempotent data pipelines are for some fol…

I can attest. I tried explaining Idempotency to people building an industrial laser controller once.

Idempotent calls might be used redundantly to ensure that a particular state or setting exists at a particular time.

I don't really care that a particular setting was already set yesterday or an hour ago.

Conversely, it's probably a really a good idea to be sure it's on that particular setting right now, before I fire the laser!

Re: Idempotency

#40

Post by eximius »

jay-barronville wrote: post #14
CSMastermind wrote: post #3

One of the things I think I did right in my career is working on payment and billing systems early on. It does teach you how to build at least semi-reliable software because people tend to get really mad when you screw with their money. It also is a good introduction to regulation and compliance. Idempotency was is one of the concepts you get taught on day one in that field (as kind of demonstrated by that being the…

Interestingly, I didn’t have to be taught about idempotency early on in my career; I’m pretty sure I didn’t even know what that word meant. I’m a naturally curious person and I was always curious about the failure cases for the software I worked on—e.g., “What happens if this request is successful but times out before returning a response to the client?” I honestly assumed every programmer asked these types of questi…

Hopefully many do, but it is much easier to talk about when you learn the terminology.
Post reply on HN