Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotence: What is it and why should I care?

cloudingmine.com

Post reply on HN
Print view View original

71–77 of 77 posts

Re: Idempotence: What is it and why should I care?

#71

Post by pmarreck »

flukus wrote: post #31
pmarreck wrote: post #5

One of the practical applications of this concept I've found is that I try to write idempotent database migrations (so rerunning the migration, which is a common necessity while you're developing it, but is also useful if problems occur, won't error). So in essence both the "up" and the "down" migrations are idempotent and warn if they are not (and why).

Database migrations are inherently stateful so I'm not a fan of indompodence here, it can leave the schema in some arbitrary states. I much prefer tools like flyway ( https://flywaydb.org/ ) that are more deterministic, each migration will only be run once so you're going from known state to known state.

I've had situations where a migration that ran locally just fine, failed in staging and/or production (config-level stuff running into hosted DB access rights, etc.). Those situations are a mess to untangle without idempotent migrations.

Also, minor niggle but Flyway isn't database-agnostic, you'd have to use the SQL of whatever DB you happen to be using (although if you code in Java I guess you could use ORM commands)

Re: Idempotence: What is it and why should I care?

#72

Post by beaconstudios »

hardwaresofton wrote: post #57
ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

what you've just described is very similar to the data management side of the platform I'm building. I can confirm that given event sourcing, idempotent mutation, and only adding new API endpoints, you're a long way towards what I would consider the ideal online software architecture. So many problems with API maintenance just go away with this approach.

Re: Idempotence: What is it and why should I care?

#73

Post by pgt »

hardwaresofton wrote: post #69
pgt wrote: post #61

Earlier quoted context omitted.

You will enjoy the fact-based database Datomic, which is the reason Rich Hickey made Clojure: https://www.youtube.com/watch?v=Cym4TZwTCNU Datalog is a much, much, much query language better than SQL: http://www.learndatalogtoday.org/

I do remember Datomic and I think it's a great tool but I fell out of love with the Clojure ecosystem and JVM-based languages as a whole and don't think I'll be getting back into it/them. I do remember wanting to check out Datomic (I believe after seeing a talk on how it was being used at a bank in southern america?[0]), but I found it unreasonably hard to find and download/experiment with the community edition -- co…

Yes, Datomic is the killer app for Clojure [^1]. Have a look at Datascript[^2] and Mozilla's Mentat[^3], which is basically an embedded Datomic in Rust.

Hickey's Spec-ulation keynote is probably his most controversial talk, but it finally swayed me toward dynamic typing for growing large systems: https://www.youtube.com/watch?v=oyLBGkS5ICk

The Clojure build ecosystem is tough. Ten years ago, I could not have wrangled Clojure with my skillset - it's a stallion. We early adopters are masochists, but we endure the pain for early advantages, like a stable JavaScript target, immutable filesets and hot-reloading way before anyone else had it.

Is it worth it? Only if it pays off. I think ClojureScript and Datomic are starting to pay off, but it's not obvious for who - certainly not very ever organisation.

React Native? I tore my hair out having to `rm -rf ./node-modules` every 2 hours to deal with breaking dependency issues.

Whenever I try to use something else (like Swift), I crawl back to Clojure for the small, consistent language. I don't think Clojure is the end-game, but a Lisp with truly immutable namespaces and data structures is probably in the future.

[^1]: In 2014 I wrote down "Why Clojure?" - http://petrustheron.com/posts/why-clojure.html [^2]: https://github.com/tonsky/datascript [^3]: https://github.com/mozilla/mentat

Re: Idempotence: What is it and why should I care?

#74

Post by pgt »

hardwaresofton wrote: post #69
pgt wrote: post #61

Earlier quoted context omitted.

You will enjoy the fact-based database Datomic, which is the reason Rich Hickey made Clojure: https://www.youtube.com/watch?v=Cym4TZwTCNU Datalog is a much, much, much query language better than SQL: http://www.learndatalogtoday.org/

I do remember Datomic and I think it's a great tool but I fell out of love with the Clojure ecosystem and JVM-based languages as a whole and don't think I'll be getting back into it/them. I do remember wanting to check out Datomic (I believe after seeing a talk on how it was being used at a bank in southern america?[0]), but I found it unreasonably hard to find and download/experiment with the community edition -- co…

Also look at Magic, an experimental typed JVM Lisp with immutable namespaces: https://github.com/mikera/magic

Re: Idempotence: What is it and why should I care?

#75

Post by Groxx »

hardwaresofton wrote: post #57
ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

I'm curious about how you anticipate handling new APIs / how this approach helps ensure consistency for people who aren't on the new APIs. Seems like if

    a -> b -> c
becomes

    a    b -> c
     \-> x
then C won't be aware of the new stuff happening in X... unless it's 100% compatible with everything B does, including every log it produces, at the moment it starts receiving traffic, which seems unlikely. In the time until C updates to read from X:

    a    b    c
     \-> x -->^
isn't C (and its consumers) operating on an "inconsistent view of the database", as produced by A?

Re: Idempotence: What is it and why should I care?

#76

Post by hardwaresofton »

Groxx wrote: post #75
hardwaresofton wrote: post #57

Earlier quoted context omitted.

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

I'm curious about how you anticipate handling new APIs / how this approach helps ensure consistency for people who aren't on the new APIs. Seems like if a -> b -> c becomes a b -> c \-> x then C won't be aware of the new stuff happening in X... unless it's 100% compatible with everything B does, including every log it produces, at the moment it starts receiving traffic, which seems unlikely. In the time until C updat…

I think I mentioned it earlier, but the idea is that the commands are immutable -- API growth happens through accretion only. New APIs must handle a superset of old ones.

Realistically, this is basically the same as how it's handled in most APIs today -- until you can guarantee (or choose to strictly enforce) that no one use a particular API, it just stays.

In addition to this, new instances use completely different databases, but rely on the replaying the stream of commands that got the old instance there to catch up (and new commands as they come in).

Re: Idempotence: What is it and why should I care?

#77

Post by Groxx »

hardwaresofton wrote: post #76
Groxx wrote: post #75

Earlier quoted context omitted.

I'm curious about how you anticipate handling new APIs / how this approach helps ensure consistency for people who aren't on the new APIs. Seems like if a -> b -> c becomes a b -> c \-> x then C won't be aware of the new stuff happening in X... unless it's 100% compatible with everything B does, including every log it produces, at the moment it starts receiving traffic, which seems unlikely. In the time until C updat…

I think I mentioned it earlier, but the idea is that the commands are immutable -- API growth happens through accretion only. New APIs must handle a superset of old ones. Realistically, this is basically the same as how it's handled in most APIs today -- until you can guarantee (or choose to strictly enforce) that no one use a particular API, it just stays. In addition to this, new instances use completely different…

To be clear, I'm interpreting "APIs" as "a method". So adding a new method is equivalent to adding a new service. If you mean for methods to never increase in number, only flexibility, then yea - I think I follow, this all makes sense. Then new stuff is truly new and disjoint from others, and there's no migration to worry about.

---

Also, since you're mentioning "replaying the stream of commands", I think this means "consistent" is strictly bound to "... at the point in time it has read to, from API X"? Then yea, switching APIs / methods is fine, you just delay the readers. It's event sourcing in a nutshell - there are undeniable benefits between any two "services", it's a compelling design.

I was interpreting it more in a system-wide sense with a large number of services, which is where I don't have a good feel for event sourcing - consumers of C and [others] are not "up to date" with what A has done until they read all data derived from all sources from the same minimum A-timestamp. So without a vector clock (probably) it's generally unsafe to consume from C and Q until they're both up to date, because C is missing stuff from A that Q already handled. Building something that maintains correctness and usefulness in the face of this seems extremely difficult or constrained, unless you accept unbounded delays (in practice: likely weeks of dev time in some cases).

---

And last but not least: CRDTs solve pretty much all of this without synchronization of any kind, yea. Are they still a pain to design? Or have we developed relatively-repeatable strategies nowadays? I haven't kept up much here, sadly.

---

I'll probably have to reread this all a couple times to make sure I'm not totally off somewhere irrational, sorry! Yours was a rather dense comment to comprehend, and I'm not sure I'm following correctly. Event sourcing has been interesting to me for quite a while, but I've never really developed a feel for how to build large, multi-developer(-team) systems out of it and it sounds like you might have an idea.

Post reply on HN