Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotency is easy until the second request is different

blog.dochia.dev

Post reply on HN
Print view View original

211–220 of 244 posts

Re: Idempotency is easy until the second request is different

#211

Post by scott_w »

stickfigure wrote: post #185
scott_w wrote: post #179

Earlier quoted context omitted.

Not in the payments world. If you’re 99% done but only the bookkeeping failed, then it’s likely that money is changing hands and you need to deal with that fact. Payments are not an atomic infrastructure and you cannot magic that into reality.

Payments are multistep but each of the steps needs to be atomic. The "create payment" operation must be transactional and the communication channel between you and the processor must be idempotent so you don't inadvertently create multiple payments. The fact that payments have a settlement process is not relevant to this discussion.

> The "create payment" operation must be transactional and the communication channel between you and the processor must be idempotent so you don't inadvertently create multiple payments.

Yes, I agree. You want to generate a token, persist it locally and use that to communicate with the payment gateway, so re-submissions use the same key and either error or return the transaction state.

> The fact that payments have a settlement process is not relevant to this discussion.

I wasn't talking about settlement, I was talking about the processing aspect. What I meant was: once you kickstart the process with the gateway, money is highly likely to change hands as a result. This means a process of:

1. POST /checkout

2. Create token

3. POST to payment gateway with token

4. Wait for gateway to return

5. Persist transaction/error

6. Return success/error

What is needed is to persist and return the token to the caller before contacting the payment gateway, to make a check + retry mechanism possible.

And yes, I've seen code that follows steps 1-6 exactly as I've described and, yes, all the problems you imagine would occur from those steps have occurred at one time or another.

Re: Idempotency is easy until the second request is different

#212

Post by GRiMe2D »

kelnos wrote: post #197
pdonis wrote: post #109

Earlier quoted context omitted.

What if the original request is still being processed when the retry comes in? That doesn't fall into either of your categories: the request isn't successful, but it hasn't failed either.

You still return 409, with error detail in the response body, because it is still true that the client sent multiple requests with the same idempotency key, regardless of the progress of the original request. Regardless, I think your assumption about how the request/response cycle should be working is wrong. For this kind of API and transaction, the server should be returning a response immediately: 202 Accepted. The…

Ah you see, you just essentially moved unique constraint to a client. It is no different generating an ID on client side and making a request. It is not idempotency.

Idempotency-Key should not replay the response (it depends, actually). But also it should not error 409. You need to be content aware before adding Idemmpotency Key header handling.

What will happen when the request is received and handled but during writing response body TCP connection dropped unexpectedly. And after second or two a connection reestablished. How two sides agree that previous request accepted and everything good to go? That's what Idempotency-Key header does.

Re: Idempotency is easy until the second request is different

#213

Post by anarazel »

ahoka wrote: post #207
stickfigure wrote: post #203

Earlier quoted context omitted.

There are a lot of ways to implement this, so I posted an example with one of the most common ways - a database which uses optimistic concurrency in serializable isolation level. Postgres is often configured this way, though it's not the only way it can be configured. With optimistic concurrency models, collisions are only detected at commit time. Two transactions can simultaneously update the same data; each update…

"a database which uses optimistic concurrency in serializable isolation level. Postgres is often configured this way, though it's not the only way it can be configured." It's not the default (read committed is) and I never saw serializable being set in actual production systems. You can do it, but then you have to be able to retry all of your transactions, including read. What if the task you do take 5 minutes? 30 mi…

> It's not the default (read committed is) and I never saw serializable being set in actual production systems.

It's not the common mode of deployment, but it's definitely in prod use.

> You can do it, but then you have to be able to retry all of your transactions, including read.

Pure read transactions shouldn't need to be retried in postgres due to serialization errors. You need to have read-write dependencies for that.

That's not to say that effectively read only transactions aren't affected by serializable, you do need to record the necessary metadata for the serialization logic to work.

FWIW, if you know your transaction is read only and long running, you can start a transaction with START TRANSACTION READ ONLY DEFERRABLE, which makes the start transaction slower, but then does not need to do any work related to serializable while the transaction is running.

Re: Idempotency is easy until the second request is different

#214

Post by stickfigure »

scott_w wrote: post #211
stickfigure wrote: post #185

Earlier quoted context omitted.

Payments are multistep but each of the steps needs to be atomic. The "create payment" operation must be transactional and the communication channel between you and the processor must be idempotent so you don't inadvertently create multiple payments. The fact that payments have a settlement process is not relevant to this discussion.

> The "create payment" operation must be transactional and the communication channel between you and the processor must be idempotent so you don't inadvertently create multiple payments. Yes, I agree. You want to generate a token, persist it locally and use that to communicate with the payment gateway, so re-submissions use the same key and either error or return the transaction state. > The fact that payments have a…

This is one possible API but it's not the dominant way payments are exposed by payments providers. Stripe, Amazon, Paypal, et al do this differently. They're fine.

Re: Idempotency is easy until the second request is different

#215

Post by stickfigure »

ahoka wrote: post #207
stickfigure wrote: post #203

Earlier quoted context omitted.

There are a lot of ways to implement this, so I posted an example with one of the most common ways - a database which uses optimistic concurrency in serializable isolation level. Postgres is often configured this way, though it's not the only way it can be configured. With optimistic concurrency models, collisions are only detected at commit time. Two transactions can simultaneously update the same data; each update…

"a database which uses optimistic concurrency in serializable isolation level. Postgres is often configured this way, though it's not the only way it can be configured." It's not the default (read committed is) and I never saw serializable being set in actual production systems. You can do it, but then you have to be able to retry all of your transactions, including read. What if the task you do take 5 minutes? 30 mi…

> I never saw serializable being set in actual production systems

Every major prod system I've worked on in the last 15 years ran in serializable, including my current charge which processes tens of billions of dollars annually. YMMV but this is quite common in serious production systems. Google's Spanner only runs in serializable.

It doesn't matter though. I could write the sequence out with a SELECT FOR UPDATE and the second request will block instead of retry. The client experience is the same; the "second" request blocks. @pdonis wanted an example so I picked one.

Re: Idempotency is easy until the second request is different

#216

Post by pdonis »

kelnos wrote: post #197
pdonis wrote: post #109

Earlier quoted context omitted.

What if the original request is still being processed when the retry comes in? That doesn't fall into either of your categories: the request isn't successful, but it hasn't failed either.

You still return 409, with error detail in the response body, because it is still true that the client sent multiple requests with the same idempotency key, regardless of the progress of the original request. Regardless, I think your assumption about how the request/response cycle should be working is wrong. For this kind of API and transaction, the server should be returning a response immediately: 202 Accepted. The…

By "request" I didn't necessarily mean the HTTP request sent by the client, and I don't think the post I was responding to did either. But I agree my use of terminology was ambiguous. Let me restate more clearly, and in a way that shows the issue under your request/response process:

An HTTP request comes in with a certain idempotency key. The server returns 202, as you say, and begins to process the database transaction.

While the server is still procesing the database transaction, a second HTTP request comes in with the same idempotency key. What response does this second HTTP request get? The original transaction that the first HTTP request triggered hasn't succeeded and hasn't failed, so it doesn't fall into either of the categories in the post I responded to.

Your answer is that the second HTTP request gets a 409, which makes sense to me, although others are objecting to it.

Re: Idempotency is easy until the second request is different

#217

Post by pdonis »

stickfigure wrote: post #203
pdonis wrote: post #193

Earlier quoted context omitted.

> The second request starts a transaction and starts processing, everything looks good - no dups. So when the second request comes in, even though it has the same idempotency key as the first request, the server doesn't check to see if there's already a request received with that idempotency key? That would seem to defeat the whole purpose of idempotency keys. > On retry, the second transaction detects the duplicate.…

There are a lot of ways to implement this, so I posted an example with one of the most common ways - a database which uses optimistic concurrency in serializable isolation level. Postgres is often configured this way, though it's not the only way it can be configured. With optimistic concurrency models, collisions are only detected at commit time. Two transactions can simultaneously update the same data; each update…

> There are a lot of ways to implement this

Sure, I get that. What I don't get is why you would be using idempotency keys as part of the implementation if you're going to go ahead and start a second transaction when you get a duplicate request, and not even check the idempotency key, and let your database tell you you've got a duplicate when you try to commit the second transaction. This subthread is specifically about implementations that use idempotency keys, since that's what the article is about.

Re: Idempotency is easy until the second request is different

#218

Post by stickfigure »

kelnos wrote: post #197
pdonis wrote: post #109

Earlier quoted context omitted.

What if the original request is still being processed when the retry comes in? That doesn't fall into either of your categories: the request isn't successful, but it hasn't failed either.

You still return 409, with error detail in the response body, because it is still true that the client sent multiple requests with the same idempotency key, regardless of the progress of the original request. Regardless, I think your assumption about how the request/response cycle should be working is wrong. For this kind of API and transaction, the server should be returning a response immediately: 202 Accepted. The…

> You still return 409

No no no no no.

You have multiple clients submitting the same business operation simultaneously. One must succeed, the others must fail. If you're using the 409 approach ("notify client that request is redundant") you must not send a 409 code until the work is complete.

The client must interpret 200 and 409 as success cases. 200 means "it was done" and 409 means "it was already done". Clients looping (say, processing durable queue messages) can stop when they receive these responses.

If the work is not complete, you can't return 409, or clients will think the work is done. You will lose messages.

Re: Idempotency is easy until the second request is different

#219

Post by pdonis »

stickfigure wrote: post #218
kelnos wrote: post #197

Earlier quoted context omitted.

You still return 409, with error detail in the response body, because it is still true that the client sent multiple requests with the same idempotency key, regardless of the progress of the original request. Regardless, I think your assumption about how the request/response cycle should be working is wrong. For this kind of API and transaction, the server should be returning a response immediately: 202 Accepted. The…

> You still return 409 No no no no no. You have multiple clients submitting the same business operation simultaneously. One must succeed, the others must fail. If you're using the 409 approach ("notify client that request is redundant") you must not send a 409 code until the work is complete. The client must interpret 200 and 409 as success cases. 200 means "it was done" and 409 means "it was already done". Clients l…

> You have multiple clients submitting the same business operation simultaneously.

It doesn't have to be multiple clients. It could be the same client, not having received a response to its first request and deciding to re-send the request again.

Re: Idempotency is easy until the second request is different

#220

Post by saltcured »

hn_throwaway_99 wrote: post #173
saltcured wrote: post #168

Earlier quoted context omitted.

Idempotence already requires some client thinking if you want to conform to HTTP specs. I.e. idempotent DELETE with proper protocol behavior requires that one request see the 200 OK or 204 No Content and the other sees 404 Not Found, because the delete has already happened. It would be misleading to say 200 OK to both, because that answer means the resource was there when the request arrived. Honestly, the whole HTTP…

> The cleanest mapping in the spirit of HTTP would be that you do multiple round trips. A POST creates a new idempotence context, a bit like "start a transaction". The new URI is the key for coordinating state change and allowing restart/recovery. I think that gave me "Enterprise Java Beans PTSD". I.e. an over-engineered solution that adds complexity for both the client and server in the name of some sort of "protoco…

FWIW, I'm not particularly fond of HTTP. But there is PTSD in both directions. Doing random things ignoring (or subverting) "protocol purity" often create disastrous effects when they haven't considered how the larger system will behave when you have various middleware bits that are essentially obeying different protocols while superficially claiming to use an interoperable standard.

When I let myself ruminate, it irks me that we all let HTTP become the defacto "internet protocol" just because of firewalls. Because there was a cargo cult idea that HTTP is benign and so one of few ports allowed almost everywhere, we do stupid contortions to squeeze every protocol through an HTTP tunnel.

These short-sighted acts of laziness accumulate into HTTP everywhere. And of course, the firewall is nearly pointless when "everything" is going through that one hole anyway.

Post reply on HN