Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotence: What is it and why should I care?

cloudingmine.com

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Idempotence: What is it and why should I care?

#61

Post by pgt »

hardwaresofton wrote: post #57
ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

You will enjoy the fact-based database Datomic, which is the reason Rich Hickey made Clojure: https://www.youtube.com/watch?v=Cym4TZwTCNU

Datalog is a much, much, much query language better than SQL: http://www.learndatalogtoday.org/

Re: Idempotence: What is it and why should I care?

#62

Post by genericid »

rectangletangle wrote: post #22
ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

This is solid advice. Another common trick is to disable the submit event when the submit button is clicked for the first time, preventing two requests from firing, when the user double clicks. Then re-enable the event, if the request fails. Ideally this is done in addition to server side nonce validation, and not as the only preventative measure, because browser differences, or network issues could cause a double re…

This is basically client-side validation.

Re: Idempotence: What is it and why should I care?

#63

Post by yjftsjthsd-h »

tw1010 wrote: post #58

I read the title as impedance, then I read the article and kept reading impedance, and was about to write a comment mentioning that that's not at all impedance he's talking about, it's idempotency, but then I read the title again and now I can't understand why I read it as the former all that time.

That's okay, I read "impotence" and couldn't believe that anyone would need it explained as to why they should care.

Re: Idempotence: What is it and why should I care?

#64

Post by haswell »

rhacker wrote: post #11
kazinator wrote: post #6

Earlier quoted context omitted.

This is about idempotent operations , which are basically state changes that have the same affect if executed multiple times as if executed once: $ chmod a+x file $ chmod a+x file This is linked to mathematical idempotence in the sense that an operation is a function which takes some inputs i and the state of the world S and produces a new state S': S' = f(S, i). So then if f(f(S, i), i) = f(S, i) then f is idemponen…

Posting this to drive people crazy :) (only kidding) How about state changes as a side effect, with a near meaningless result. LockAccount('user') The account is now locked. LockAccount('user') Error: The account was already locked! However, the internal state of the user after the second operation remains the same. We also cannot use any of the information in the result as something meaningful as shown in these math…

For HTTP resources, this is addressed in RFC7231 - essentially, servers may implement non-idempotent behavior such as logging for methods otherwise considered idempotent.

https://tools.ietf.org/html/rfc7231#section-4.2.2

Re: Idempotence: What is it and why should I care?

#65

Post by SmirkingRevenge »

drej wrote: post #44

I work in data engineering and fixating on idempotence has been one of the best things I've ever done. Now whenever we build a new job or we review an existing one, the first question (well, second, first being 'do we actually need this?') is usually 'is this idempotent?' Saves SO much hassle. Processes fail, nodes disconnect, OOM kills stuff, these things happen on a daily basis in larger systems, be ready for that.

Amen. And if you combine idempotent operations with atomic operations wherever possible, it gets even better!

Re: Idempotence: What is it and why should I care?

#66

Post by hood_syntax »

pgt wrote: post #61
hardwaresofton wrote: post #57

Earlier quoted context omitted.

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

You will enjoy the fact-based database Datomic, which is the reason Rich Hickey made Clojure: https://www.youtube.com/watch?v=Cym4TZwTCNU Datalog is a much, much, much query language better than SQL: http://www.learndatalogtoday.org/

I take issue with some of Rich Hickey's technical opinions, but Datomic seems pretty cool. I knew Datalog has been around for a while, but had never looked into it further.

Re: Idempotence: What is it and why should I care?

#67

Post by hood_syntax »

jen729w wrote: post #53
digitaLandscape wrote: post #27

Earlier quoted context omitted.

I don't like this because in practice sites usually fail to re-enable the submit button if something goes wrong. Just let the user submit multiple requests if they want to retry, don't take that away from them, just make it harmless.

Newbie programmer here, but even I am already implementing finite state machines which handily fix this sort of issue. Any sort of UI without them now feels archaic.

Agree 100%. If you don't have explicit state transitions in a UI, you're asking for trouble. I think some people don't realize how much harder they make it on themselves by not setting clear boundaries; they see the up front cost and balk, when it saves a non-trivial amount of headache in the future, not to mention reducing mental overhead when you've got the structure hashed out.

Re: Idempotence: What is it and why should I care?

#68

Post by hardwaresofton »

SmirkingRevenge wrote: post #60
hardwaresofton wrote: post #57

Earlier quoted context omitted.

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

No need to write the distributed log yourself, you just need Kafka. You basically just independently conceived of, what is becoming a pretty popular architecture these days - event driven systems based on distributed logs. The source of truth in your system becomes the idempotent distributed log of events (rather than your rdbms or data warehouse) which ought to be "replayable", allowing you to audit or even potentia…

I think I wasn't clear enough about why this solution isn't just a distributed log/event sourcing approach:

- The description above was intended to NOT specify any tech it depends on -- Kafka is the best queueing and long-term-log-storage engine I've ever heard of, but the point is that it shouldn't matter. Kafka doesn't "write" the distributed log, it just stores it.

- I don't want an Event Sourcing solution. It's often combined with CQRS, but the thing is, I don't think you actually need events (I might be wrong) -- you just need the commands, and as long as what they do never changes, you can skip a bunch of problems with event sourcing -- you can more easily compress stuff if you don't allow the semantics of operations to change. Also there's stuff about how you handle things like schema changes, etc -- in event sourcing it's often "recalculate the whole DB", but I think that should not be the solution -- In this system I'm proposing, I'd literally ensure I could reflect the migration as operations in the log, and do it that way.

What you're describing is a Kafka-log based event sourcing system, and it's close to what I mean, but not quite the same -- I want the systems to be able to able to trivially cluster, for example, by hitting an endpoint like `/replicate`, which does nothing but stream log messages out as they hit one particular server. In the system you describe I think the equivalent thing would be a tiny bit harder to achieve since you'd need to be able to listen to another subscriber's messages.

I'm also completely aware that I might go about building this and realize I just took the long way around to the system you're describing but I don't think I am just yet.

Re: Idempotence: What is it and why should I care?

#69

Post by hardwaresofton »

pgt wrote: post #61
hardwaresofton wrote: post #57

Earlier quoted context omitted.

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

You will enjoy the fact-based database Datomic, which is the reason Rich Hickey made Clojure: https://www.youtube.com/watch?v=Cym4TZwTCNU Datalog is a much, much, much query language better than SQL: http://www.learndatalogtoday.org/

I do remember Datomic and I think it's a great tool but I fell out of love with the Clojure ecosystem and JVM-based languages as a whole and don't think I'll be getting back into it/them.

I do remember wanting to check out Datomic (I believe after seeing a talk on how it was being used at a bank in southern america?[0]), but I found it unreasonably hard to find and download/experiment with the community edition -- compare this to something like Postgres which is much more obvious, more F/OSS compliant (I understand that they need to make money) and Datomic doesn't really look that appealing to me these days.

At this point in my learning of software craftmanship I can't do non-statically type-checked/inferenced languages anymore -- I almost never use JS without Typescript for example. Typed clojure was in relatively early stages when I was last actively using clojure, and I'm sure it's not bad (probably way more mature now), but it's a staple in other languages like Common Lisp (the declare form IIRC). The prevailing mood the clojure community seemed to be against static type checking and I just don't think I can jive with that anymore.

Thinking this way right now Datomic wouldn't be a good fit for me pesonally but I believe that it is probably high quality paradigm.

[EDIT] - I found the talk: https://www.youtube.com/watch?v=7lm3K8zVOdY

[0]: https://www.datomic.com/nubanks-story.html

Re: Idempotence: What is it and why should I care?

#70

Post by digitaLandscape »

jen729w wrote: post #53
digitaLandscape wrote: post #27

Earlier quoted context omitted.

I don't like this because in practice sites usually fail to re-enable the submit button if something goes wrong. Just let the user submit multiple requests if they want to retry, don't take that away from them, just make it harmless.

Newbie programmer here, but even I am already implementing finite state machines which handily fix this sort of issue. Any sort of UI without them now feels archaic.

Meh. The logic still applies: on a bad network, the first request can hang for half a minute when a retry would get lucky and go through immediately.

Don't handcuff your users for no benefit, just because you can't write a well-behaved backend.

Post reply on HN