Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotency is easy until the second request is different

blog.dochia.dev

Post reply on HN
Print view View original

91–100 of 244 posts

Re: Idempotency is easy until the second request is different

#91

Post by jerf »

pvillano wrote: post #78
xlii wrote: post #48

Don't fix other people problems. If idempotent key was seen then send back response. Clients intention is outside the scope. If contract says "idempotency on key" the idempotent response on key. If contract says "idempotent on body hash" then response on body hash (which might or might not include extra data). APIs are contracts. Not the pinky promise of "I'll do my best guess"

I would rather do more work myself to make an API as fool proof as possible, than hope everyone else is perfect, and lose data when they're not.

That just leads to bigger fools. I don't just mean that as clever wordplay, but as a serious point. No matter how sloppy you make your API someone else will use it even more sloppily. Now you've got an enormous sloppy surface you can't properly contain or maintain, and people still transgress its boundaries even so.

The robustness principle has its times and places but the general consensus that it should be applied everywhere to everything was a big mistake. The default should be that you are very rigid and precise and only apply the robustness principle in those times and places it applies, and I'm perfectly comfortable waiting to deploy something precise and find out that this was one of them. The vast majority of APIs is not the time and place for the robustness principle. It's the time and place for careful precision on exactly what is provided, and detailed and description error messages, logging, and metrics for when the boundaries are transgressed.

Re: Idempotency is easy until the second request is different

#92

Post by randallsquared »

zarzavat wrote: post #77
randallsquared wrote: post #76

Earlier quoted context omitted.

No, the sensible policy is to have the code operate idempotently for every request with an idempotent method. This is a design decision, not something you slap on top afterward with a special key.

I believe you need to read the article. The article is about the Idempotency-Key header. https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...

I skimmed the article earlier, but going back and looking, it doesn't appear that the article mentions the spec at all, or links to it. In fact, the first paragraph is literally "People talk about..."

Some help for others to understand the history of this (which apparently Stripe, Paypal, Dwolla, and others use): https://github.com/mdn/content/issues/41497 There are links to the RFC and prior art.

That aside, my first impulse is to say that the server should specify that the key includes a hash of the important parts of the request, checked on receipt, so that only the key itself need be stored. However, FF's implementation apparently(?) adds the header automatically to POST and PATCH if it's not already present, which means that it's not able to comply with such a decision, and the RFC (currently expired) recommends using a UUID, so.

I'm guessing the original motivation of this is "Browser JS might not be able to send a PUT, or proxies may not handle a PUT correctly".

Re: Idempotency is easy until the second request is different

#94

Post by ordu »

pvillano wrote: post #83
ordu wrote: post #37

Well, it is "reality has a surprising amount of detail"[1] all over again. Or rather a good specific example for it. [1] http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...

Thanks for letting me read that again. I once wrote about inherent, irreducible complexity and how we try to deal with it. The draft has sections on how complexity can be hidden, spread out, localized, passed off, or recreated from scratch. Unfortunately, people are now using LLMs to pile complexity on the simplest of tasks, and my essay isn't really worth finishing.

> people are now using LLMs to pile complexity on the simplest of tasks, and my essay isn't really worth finishing.

Isn't the opposite true? The more people are messing with complexity, the more they could benefit from a model of a complexity? And if they generate complexity with external tools, then maybe a theoretical take on that will be the only way for them to learn? I mean, we learn these things through struggle and pain, but if all of that becomes an LLM problem, than you just stop learning? But at some point complexity will strike back, at some point there will be as much of it, that LLM will be no help.

OTOH, if LLM still win, and skills of managing complexity will be lost in future generations, if we are at the peak of our skills of dealing with complexity, than shouldn't we try our best to imprint our hard won lessons into a history? Maybe for some later generations the tide will turn and they would write textbooks on complexity, and with your article you'll get your portrait in a textbook, and each bored pupil will decorate it with mustaches? You have a chance to immortalize yourself. xD

Or maybe you can become someone like Ramanujan for math? Someone who honed obsolete math skills to an unimaginable level? Maybe a time will come, when students will pour over Ramanujan works, because his skills became useful again, and they try to find out how Ramanujan thought?

...

Sorry, I just couldn't resist. Seriously, it is hard to predict with LLMs, maybe we will not need intellect or any intellectual skills at all after AGI.

Re: Idempotency is easy until the second request is different

#95

Post by NovemberWhiskey »

The real problem is that sticking an idempotency key onto an operation doesn’t make it idempotent.

It may improve efficiency where a protocol doesn’t assure exactly-once delivery of messages, but it cannot help you with problems other than deduplication of identical messages.

Creating a payment is not an idempotent operation. If the economics of the operation can differ when the “idempotency” key remains the same then you’ve just created a foot-gun in your API.

You can document that you’re going to ignore “duplicate” requests that share an idempotency key but that’s just user-hostile. The system as a whole is broken as designed.

Re: Idempotency is easy until the second request is different

#96

Post by kaoD »

randallsquared wrote: post #64
mrkeen wrote: post #54

Earlier quoted context omitted.

In what sense is GET naturally idempotent? The GET/POST split is the defence (even it's only advisory). GET-only means every time you hit the back button during an order flow, you might double-order.

GET is not supposed to make changes on the server. The usual idempotent verbs for making changes are PUT and DELETE. One thing that's confusing, here, is that idempotency only applies for the same request, but the article implies that idempotency is about whether the request contains a specific "idempotency key". Don't do that, and this problem evaporates.

> Don't do that, and this problem evaporates.

Don't do that, and you solved nothing.

Either I'm missing what you mean, or half the comments here are missing the point of idempotency.

Let's say your server received this request twice within one minute:

    {
      items: [ { id: 123, amount: 1 } ],
      creditCardInfo: { ... }
    }
How can you tell from the server if that's a retry (think e.g. some reverse proxy crashed and the first request timed out, but the payment already went through to the user's CC)... or if the user just trying to purchase another item 123 because they forgot they needed 2?

There is simply no way to make the requests idempotent without an idempotency key. The only way to tell both situations apart is to key the requests by some UID. The HTTP verb is irrelevant.

Did I misunderstand what you meant?

Re: Idempotency is easy until the second request is different

#97

Post by Pxtl »

stickfigure wrote: post #89

This is all way too much. If you see a duplicate idempotency key, skip the replay and always return 409. This becomes a client problem. Clients already need to help enforce idempotent contracts; "check for conflict response" is not an onerous imposition. I've built multiple ecommerce APIs with this approach and they work great. No heroic measures required. You can often satisfy this contract with a unique constraint;…

[dead]

Re: Idempotency is easy until the second request is different

#98

Post by mainde »

amiga386 wrote: post #82
mainde wrote: post #81

Earlier quoted context omitted.

IMHO it's more: fix problems, or at least mitigate them, regardless whose problem it is. I've been in this situation, a clientside bug meant that different requests arrived with the same idempotency key. In my case, updating the client would have taken weeks, in the best case scenario. Updating the backend to check for a matching request body would have taken minutes, maybe hours. It took me a surprising amount of ar…

Yeah but don't let them reify it. Ideally you already send client version in requests (or have an API version prefix). Add the workaround only for legacy clients. Next client version must distinguish itself from predecessor and must not require the bodge to work.

Well.. it was ~6 years and ~10 billion payments ago, the clients have been fixed but the "hack" is still there, it has caused no harm as far as I can tell. Worst case scenario it's useless, best case scenario it prevents regressions.

The issue with things that client must not do is that they might still do them, and users don't care whose fault it is. It's important to have auxilliary mechanisms to mitigate these.

Re: Idempotency is easy until the second request is different

#100

Post by halestock »

stickfigure wrote: post #89

This is all way too much. If you see a duplicate idempotency key, skip the replay and always return 409. This becomes a client problem. Clients already need to help enforce idempotent contracts; "check for conflict response" is not an onerous imposition. I've built multiple ecommerce APIs with this approach and they work great. No heroic measures required. You can often satisfy this contract with a unique constraint;…

But that's not idempotent? If I'm a client and I don't know if the original request went through, getting a 409 on any subsequent requests tells me nothing about whether the original request was successful or not.
Post reply on HN