Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotence: What is it and why should I care?

cloudingmine.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Idempotence: What is it and why should I care?

#51

Post by enraged_camel »

ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

I apologize for the noob question, but... is it a good idea to have the client be in charge of generating UUIDs? When dealing with idempotence, they have to be truly random, not just pseudorandom, right? I looked into this recently, and my understanding (from the cursory amount of research I did) was that the UUID generation method depends on the random number generator used by the browser, which can vary a lot.

Why not generate the UUID on the server and send it to the client along with the page request (if using SSR)? I.e. the server generates the UUID, sticks it in a database field, then sends it to the client. When the client responds with the UUID, you can check that against the database to make sure its valid.

Re: Idempotence: What is it and why should I care?

#52

Post by filmor »

tutfbhuf wrote: post #17
idempotent wrote: post #4

A function f is idempotent if: f(f(x)) = f(x) The "absolute value" function is idempotent: abs(abs(-42)) = abs(-42) The "squared" function is not: sq(sq(7)) != sq(7)

In category theory we also say h is idempotent iff h^2 = h whereas h is an endomorphism.

The "morphism" property is completely unneeded for idempotence and the "endo" part is implied :)

Re: Idempotence: What is it and why should I care?

#53

Post by jen729w »

digitaLandscape wrote: post #27
rectangletangle wrote: post #22

Earlier quoted context omitted.

This is solid advice. Another common trick is to disable the submit event when the submit button is clicked for the first time, preventing two requests from firing, when the user double clicks. Then re-enable the event, if the request fails. Ideally this is done in addition to server side nonce validation, and not as the only preventative measure, because browser differences, or network issues could cause a double re…

I don't like this because in practice sites usually fail to re-enable the submit button if something goes wrong. Just let the user submit multiple requests if they want to retry, don't take that away from them, just make it harmless.

Newbie programmer here, but even I am already implementing finite state machines which handily fix this sort of issue.

Any sort of UI without them now feels archaic.

Re: Idempotence: What is it and why should I care?

#54

Post by rimliu »

js2 wrote: post #19

At 13:45 in this interview from 2003 with Sergey Brin and Larry Page, you can listen to them try to explain idempotentence on the air to Terry Gross and her NPR audience: http://www.npr.org/2003/10/14/167643282/google-founders-larr...

And in 2005 Google came up with Google Accelerator, which has shown many sloppy programmer what happens when you use HTTP method which is supposed to be idempotent—GET—for others purposes. What happened that GA crawled all the links it found for prefetching and some of them were "delete" links in admin interfaces. I think that was the biggest push not to use GETs to modify data :)

Re: Idempotence: What is it and why should I care?

#55

Post by tialaramex »

r_c_a_d wrote: post #46
akira2501 wrote: post #29

Earlier quoted context omitted.

> So...what's special about Ohms, Kelvins, and ångström Nothing other than misguided thinking in the early versions of the standard. The other problems with these special symbols is that if you call tolower() or similar on them they'll return the "normal" character they're based off of. So toupper(tolower(char)) != char.

Does tolower() or toupper() even make sense with general unicode characters? I wouldn't expect it to... but I've never really thought about it before :-)

Mostly, we're used to defining tolower() and toupper() to return either a lower or upper case variant if one exists, otherwise you get back what you put in. For most Unicode codepoints no such variants exist and so you just get back whatever you fed in. Some of the alphabets have uppercase/ lowercase, but obviously most writing systems don't do this.

However, lower(upper(X)) is not defined to be the same as lower(X), and there's no promise that meddling with a string transforming with lower() or upper() does what you hoped because that isn't how language actually works (e.g. in English the case sometimes marks proper nouns so "May" is the Prime Minister of the UK, but "may" is just an auxiliary verb).

Where standards tell you something is case-insensitive, but it's also allowed to be Unicode rather than ASCII, you can and probably should "case crush" it with tolower() and then never worry about this problem. In a few places you have to be careful because a standard says something in particular is case-insensitive, but not everything that goes in that slot is case-insensitive. For example MIME content type names like "text/plain", "TEXT/PLAIN" and "Text/Plain" are case-insensitive, but

multipart/mixed; boundary="ABCDEFGHIJKL" multipart/mixed; boundary="abcdefghijkl" multipart/mixed; boundary="AbcDefGhiJkl"

... declare three different boundary tokens, and none of them matches the sequence abCdeFghIjkL.

Re: Idempotence: What is it and why should I care?

#56

Post by chaboud »

enraged_camel wrote: post #51
ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

I apologize for the noob question, but... is it a good idea to have the client be in charge of generating UUIDs? When dealing with idempotence, they have to be truly random, not just pseudorandom, right? I looked into this recently, and my understanding (from the cursory amount of research I did) was that the UUID generation method depends on the random number generator used by the browser, which can vary a lot. Why…

You can do a mix of the two to reduce server traffic (especially for abandoned sessions) and user latency.

Give the client a client ID that you generate that should be properly random... Have them submit that ID along with their self-generated ID.

Re: Idempotence: What is it and why should I care?

#57

Post by hardwaresofton »

ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but:

- Express all operations as log messages (ez pz distribution)

- Ensure all operations are idempotent

- Record the operations (this is the log you can distribute if you please)

- Disallow API code modification, only allow accretion/use of new API endpoints.

- All APIs that come up have their own databases, a bit of the CQRS model here (but without events -- just the actions performed)

- When you need to stand up new API servers, start the new ones (with handling code for old operations completely unchanged) next to the old ones, and update the http-server code (like request handlers) to output the new commands. Older servers that don't understand the new commands will ignore (or redirect), and new servers that do understand will process and add to the distributed log. New nodes just stream the replications of the already existing nodes and no one spends any time with an inconsistent view of the database

Of course, writing to a distributed log is slow (pick whichever consensus algo you want, you either have durability with a quorum or best-effort without), but this only is a huge deal if you're doing lots of writes, and for most web applications, that's not what's happening, the vast majority is reads.

CRDTs might even fit in here, because if you want a multi-master setup, you could literally keep the log as a set (I'm not quite sure how truncation of super old records would want) keyed by transaction ID -- Assuming the same request doesn't go to multiple servers, their logs should be easily combinable at the end of the day -- API1 is gonna see events A B and E, API2 might see C and F, and API3 will likely see D G and H.

Honestly everything I've described here is really more like moving the coordination/distributed log problem to the application level (up until now all this action would just happen @ the Postgres/DB level), but I'm not yet convinced it's a terrible idea.

I haven't found the time to actually try to make what I'm describing here a thing but would love to hear thoughts

Re: Idempotence: What is it and why should I care?

#58

Post by tw1010 »

I read the title as impedance, then I read the article and kept reading impedance, and was about to write a comment mentioning that that's not at all impedance he's talking about, it's idempotency, but then I read the title again and now I can't understand why I read it as the former all that time.

Re: Idempotence: What is it and why should I care?

#59

Post by qoi2ijds0 »

rectangletangle wrote: post #22
ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

This is solid advice. Another common trick is to disable the submit event when the submit button is clicked for the first time, preventing two requests from firing, when the user double clicks. Then re-enable the event, if the request fails. Ideally this is done in addition to server side nonce validation, and not as the only preventative measure, because browser differences, or network issues could cause a double re…

IMO disabling buttons causes more trouble than it's worth - you can never guarantee the user hasn't clicked twice (because don't forget the code that disabling the button in the first place is relying on an event firing that tells you the button was clicked - this doesn't mean 2 events can't be queued before the event handler is called), and then you need a whole chunk of code around re-enabling the button depending on what has happened after the fact which is then a big source of bugs.

Re: Idempotence: What is it and why should I care?

#60

Post by SmirkingRevenge »

hardwaresofton wrote: post #57
ageitgey wrote: post #18

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

So I think this is actually the secret to creating actually dependable, no-downtime transitioning endpoints. It's just an idea that has been rolling around in my head but: - Express all operations as log messages (ez pz distribution) - Ensure all operations are idempotent - Record the operations (this is the log you can distribute if you please) - Disallow API code modification, only allow accretion/use of new API en…

No need to write the distributed log yourself, you just need Kafka.

You basically just independently conceived of, what is becoming a pretty popular architecture these days - event driven systems based on distributed logs.

The source of truth in your system becomes the idempotent distributed log of events (rather than your rdbms or data warehouse) which ought to be "replayable", allowing you to audit or even potentially recreate your application state (databases, etc), at any point in time.

Transitioning to a new version of an api/service means adding another subscriber to the event stream (and perhaps reprocessing some or all of its history), that runs in parallel with the old version - then when everything looks good you can quietly disable the old version.

Post reply on HN