Live data from Hacker News

Idempotence: What is it and why should I care?

cloudingmine.com

41–50 of 77 posts

Re: Idempotence: What is it and why should I care?

#41

Idempotence can be very helpful when one strives for resiliency. Suppose you have an application that processes tasks that (among others) call an external API. Both external API call and task processing can fail independently. Moreover, task processing can fail after API call succeeded. If you the external API is idempotent, you can simply retry on any task processing failure, no matter when it happened. It can simpl…

One thing to note about this - idempotency is a very simple concept, but the implementation is actually quite hard. Case-in-point: the kafka producer API recently gained support for "idempotent/transactional capabilities": https://kafka.apache.org/documentation/#upgrade_11_exactly_o... . That is to say exactly-once persistence by using an idempotency key.

Everything looks bulletproof unless you take a step back. To connect with the example you gave: just having the kafka producer guarantee that one call to send() is idempotent is not enough. Your application needs to be able to be idempotent - e.g. if the same RPC/web request has to be retried on a different server. You need to be able to pass an idempotency key TO the kafka API - which you currently cannot do. The API currently allows for either a global(ish) lock or duplicated messages - so it's not quite idempotent.

Idempotency needs to be end-to-end, otherwise it doesn't work. Unfortunately that's very rarely the case - almost nobody tries to idempotently make XHR requests to their servers. In effect it's almost always easier to de-duplicate idempotently on read rather than attempt to write idempotently. It's a really hard simple problem with lots of corner cases.

Re: Idempotence: What is it and why should I care?

#42

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

This is solid advice. Another common trick is to disable the submit event when the submit button is clicked for the first time, preventing two requests from firing, when the user double clicks. Then re-enable the event, if the request fails. Ideally this is done in addition to server side nonce validation, and not as the only preventative measure, because browser differences, or network issues could cause a double re…

disabling the submit button is a nice UI feature to indicate idempotency, but it's not the same thing as the function actually being idempotent. both are important.

Re: Idempotence: What is it and why should I care?

#43

This is vital if you are designing apis or clients that deal with charging a user money. It should be literally impossible for a user to accidentally get charged twice due to a flakey connection if you design correctly. The trick is to have the client generate a random 'idempotency key' (a uuid) to start each logical transaction and have the server use that id to prevent double charges of the same transaction. By alw…

Good point, but I'd add that not only when charging money, but any operation that can have side effects if executed more than once (emails, external calls, etc.).

Having idempotent queue consumers/listeners can help with overall systems resilience (someone already mentioned this below). It helps as much as you don't have to worry when restarting a queue broker and/or apps with consumers.

It adds complexity, but I think it pays off in the long run.

Re: Idempotence: What is it and why should I care?

#44
I work in data engineering and fixating on idempotence has been one of the best things I've ever done. Now whenever we build a new job or we review an existing one, the first question (well, second, first being 'do we actually need this?') is usually 'is this idempotent?' Saves SO much hassle. Processes fail, nodes disconnect, OOM kills stuff, these things happen on a daily basis in larger systems, be ready for that.

Re: Idempotence: What is it and why should I care?

#45
post #21

Reminds me of the Spotify unicode username issue[1]. Where a function assumed to be idempotent (think tolower(username)) actually wasn't with certain unicode inputs. Allowing account takeovers. [1] https://labs.spotify.com/2013/06/18/creative-usernames/

spotify> For example it is hard to see the difference between Ω and Ω even though one is obviously a Greek letter and the other is a unit for electrical resistance and in unicode they indeed have different code points This surprised me, because the correct Ohm symbol is in fact the Greek letter, so why does Unicode have a special code point for it? Unicode also does this for Kelvin, where the correct symbol is a capi…

Unicode was originally proposed as a universal character set to replace all existing character sets. For it to have any chance of acceptance it had to be possible to convert from JIS/whatever into Unicode then back again without any loss of information. So if there were any daft duplicates in legacy character sets those had to be duplicated in Unicode. I don't know if that explains those three physical units, but that's what I'd guess happened.

Re: Idempotence: What is it and why should I care?

#46
post #21

Earlier quoted context omitted.

spotify> For example it is hard to see the difference between Ω and Ω even though one is obviously a Greek letter and the other is a unit for electrical resistance and in unicode they indeed have different code points This surprised me, because the correct Ohm symbol is in fact the Greek letter, so why does Unicode have a special code point for it? Unicode also does this for Kelvin, where the correct symbol is a capi…

> So...what's special about Ohms, Kelvins, and ångström Nothing other than misguided thinking in the early versions of the standard. The other problems with these special symbols is that if you call tolower() or similar on them they'll return the "normal" character they're based off of. So toupper(tolower(char)) != char.

Does tolower() or toupper() even make sense with general unicode characters? I wouldn't expect it to... but I've never really thought about it before :-)

Re: Idempotence: What is it and why should I care?

#48
Pure idempotency isn't usually desirable. In any important database table where this can be an issue, you want two timestamps. 1 for when the row was created and 1 for when it was last updated. The upsert should change the ts_updated value, and the attempt should be logged.

If you want genuine pure idempotent interactions, you can't do that, and you have to rely on unique constraints in the database and swallow that particular error, so that nothing about the universe of the application state changes and nothing gets logged.

But that is a pretty garbage way to do things. As with many things, some moderation and flexibility are a good idea.

Instead of focusing on the exact meaning of the word, we should focus on making sure that nothing bad happens if someone does something twice. That's what the operational concept of idempotence is.

It's relatively easy to get something to happen at least one time, and it's slightly less easy to get something to happen at most one time. Getting something to happen exactly one time is really, really hard. That's why we should care: most of the things we want to happen will happen more than once in any nontrivial system.

Purity in concept isn't important. Safety in the sense that nothing bad happens the second or third or nth time around is.

Side note about the Stripe blog linked in the current top post by ageitgey: that's not a useful solution. You can't trust a uuid created outside the context of the uniqueness that needs a guarantee. That's one of the fundamental problems of distributed systems. And it's one I would think Stripe should know better than to espouse since, after all, they did hire aphyr.

You need something closer to home, not something received from a relatively untrusted source. Before you gripe at me and tell me that uuids are, in fact, uuids, let me explain. There are all kinds of situations that can force a client to regen a uuid.

A gas station pump resets because of a blink in power. It remembers all the information about the transaction except for the uuid because the programmer was smart and wanted that to be, well, unique. And he wanted the pump to be smart and retry the failed attempt. Same txn; different uuid.

The user hits the refresh button in the middle of the txn, but the rest of the form data is cached. Not the uuid.

The corner store owner who hates people who use credit/debit cards in general in Queens gets pissed because it takes more than two seconds to process, and pulls the power plug to reset it. Yeah, POS units should clear after that, but they don't always.

User's phone switches from cell service to wifi. Forces a refresh in the app in the middle of a txn. It's still listening for the same transaction response and when it times out tries again with a different uuid.

These are real scenarios that we have to deal with. Trusting a client to only ever retry with the same uuid is not safe. So when I say that you need a uuid in the same context, I mean the uuid created in your database when it receives an auth request, inside the SQL transaction used to create the entry for that attempt.

Everything else about the transaction is used to fingerprint it with the local uuid as the upsert key. Once that is in place, then you can start to have some duplicate txn safety. That's the beginning of your hell, but it's a better one than trusting client devices.

I'm not trying to take a potshot at Stripe. I have tons of respect for them, but that particular article is misplaced. You can't just pass uuids around and think that you're safe because they are actually unique. I wish it were that simple.

It most definitely is not, as the person said, "the trick."

Re: Idempotence: What is it and why should I care?

#49
post #46

Earlier quoted context omitted.

> So...what's special about Ohms, Kelvins, and ångström Nothing other than misguided thinking in the early versions of the standard. The other problems with these special symbols is that if you call tolower() or similar on them they'll return the "normal" character they're based off of. So toupper(tolower(char)) != char.

Does tolower() or toupper() even make sense with general unicode characters? I wouldn't expect it to... but I've never really thought about it before :-)

What's worse, tolower() and toupper() are locale-dependent. In most locales,

  tolower("I") = "i"
but in Turkish,

  tolower("I") = "ı"
Same in the other direction, because there is also a large I with dot.

Re: Idempotence: What is it and why should I care?

#50

Pure idempotency isn't usually desirable. In any important database table where this can be an issue, you want two timestamps. 1 for when the row was created and 1 for when it was last updated. The upsert should change the ts_updated value, and the attempt should be logged. If you want genuine pure idempotent interactions, you can't do that, and you have to rely on unique constraints in the database and swallow that…

> you want two timestamps. 1 for when the row was created and 1 for when it was last updated. The upsert should change the ts_updated value, and the attempt should be logged.

If timestamps are worth recording they're worth recording in the normal way you'd record business data. Associate a timestamp with when the user made their request, not on when your server happened to process it.

> A gas station pump resets because of a blink in power. It remembers all the information about the transaction except for the uuid because the programmer was smart and wanted that to be, well, unique. And he wanted the pump to be smart and retry the failed attempt. Same txn; different uuid.

Don't do that. If it's a transaction ID, generate the ID for the transaction and keep it with the data for the transaction.

> The user hits the refresh button in the middle of the txn, but the rest of the form data is cached. Not the uuid.

Again, keep the ID with the data it goes with.

> Trusting a client to only ever retry with the same uuid is not safe. So when I say that you need a uuid in the same context, I mean the uuid created in your database when it receives an auth request, inside the SQL transaction used to create the entry for that attempt.

That's an unscalable approach. If you can afford to run everything on a single database with monotonically increasing time then sure, knock yourself out, it makes everything easier and is good enough for a lot of cases. But in these days of client-side UI, like it or not you are working on a distributed system where the client and server are separate nodes, and distributed system techniques are your best hope of getting sensible behaviour.

Post reply on HN