Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

APIs, robustness, and idempotency

stripe.com

Post reply on HN
Print view View original

11–20 of 52 posts

Re: APIs, robustness, and idempotency

#11

Post by pbreit »

brandur wrote: post #7
pbreit wrote: post #3

One thing about Stripe's API I have mixed feelings about is the liberal versioning. My experiencing with 100s of payment integrations is that they get done once and hopefully never touched again. I know most of Stripes updates are "additive" such that they are backwards compatible if coded liberally, but it can be confusing. Same with Lob.

(I work at Stripe.) API versioning is definitely a debatable subject, and I don't think that anyone at Stripe would claim that the current state of affairs is perfect by any means, but it's one that we think provides a good compromise between the stability of client integrations and our own ability to iterate on the API's design and make progress. The classic problem with web APIs is that unless you have a good versi…

Funny you mention that, I can't recall ever seeing a proper move from /v1 to /v2!

Re: APIs, robustness, and idempotency

#12

Post by smallnamespace »

brandur wrote: post #8
vfaronov wrote: post #6

Curious that they don’t mention HTTP conditional requests [1] even in passing. This mechanism is typically used for slightly different things, but you can, for example, make a PATCH request “idempotent” (in their sense) by adding an If-Match header to it. I’d say that Idempotency-Key itself may be considered a precondition and used with status codes 412 [2] and 428 [3]. By the way, WebDAV extended this mechanism with…

(I wrote this.) It's always a bit of a fine line as to what makes the final cut in this sort of article (I tried to stay on message without getting too off track), but HTTP conditional requests are definitely something that could have been a good fit. I should point out though that using `ETag`/`If-Match` generally has a slightly different use on updates compared to Stripe's `Idempotency-Key`. A server sends back an…

If you interpreted 'current state of a resource' somewhat liberally, where the resource is the abstract state of a particular versioned/timestamped request, does an ETag then really become the same as an Idempotency-Key?

Or put another way, is the only difference that ETags are generally computed based on the server's stored state of a particular resource so it's possible to have multiple clients with the same ETag, while the 'resource' backing an Idempotency-Key is the entire state of a particular user that encompasses all the user's resources?

Re: APIs, robustness, and idempotency

#13

Post by pbreit »

gkoberger wrote: post #5
pbreit wrote: post #3

One thing about Stripe's API I have mixed feelings about is the liberal versioning. My experiencing with 100s of payment integrations is that they get done once and hopefully never touched again. I know most of Stripes updates are "additive" such that they are backwards compatible if coded liberally, but it can be confusing. Same with Lob.

How is this confusing? You get locked into the version, and never have to worry about things breaking. If Stripe releases new features, you also get the new features. For most people, you just never have to think about versioning.

I'm cool with Stripe's versioning for the most part. There have just been a few occasions where to get a non-breaking enhancement I've need to move to a breaking version. But not a huge deal.

Re: APIs, robustness, and idempotency

#14

Post by Johnie »

brandur wrote: post #9

I authored this article and just wanted to leave a quick note on here that I'm more than happy to answer any questions, or debate/discuss the finer points of HTTP and API semantics ;) An ex-colleague pointed out to me on Twitter today that there are other APIs out there that have developed a concept similar to Stripe's `Idempotency-Key` header, the "client tokens" used in EC2's API for example [1]. To my knowledge th…

Have you considered using the resource identifier as an idempotency key? Basically, have the client generate the ID (UUID, but namespaced on the server side [client+date]). This eliminates the need for the client to generate the idempotency key as well as the eliminates the need for the server to maintain an idempotency repository?

Re: APIs, robustness, and idempotency

#15

Post by majewsky »

pbreit wrote: post #11
brandur wrote: post #7

Earlier quoted context omitted.

(I work at Stripe.) API versioning is definitely a debatable subject, and I don't think that anyone at Stripe would claim that the current state of affairs is perfect by any means, but it's one that we think provides a good compromise between the stability of client integrations and our own ability to iterate on the API's design and make progress. The classic problem with web APIs is that unless you have a good versi…

Funny you mention that, I can't recall ever seeing a proper move from /v1 to /v2!

OpenStack Keystone is at /v3 already. But yeah, it really took about 3-4 years until most consuming OpenStack services updated.

Re: APIs, robustness, and idempotency

#16

Post by [deleted] »

Johnie wrote: post #14
brandur wrote: post #9

I authored this article and just wanted to leave a quick note on here that I'm more than happy to answer any questions, or debate/discuss the finer points of HTTP and API semantics ;) An ex-colleague pointed out to me on Twitter today that there are other APIs out there that have developed a concept similar to Stripe's `Idempotency-Key` header, the "client tokens" used in EC2's API for example [1]. To my knowledge th…

Have you considered using the resource identifier as an idempotency key? Basically, have the client generate the ID (UUID, but namespaced on the server side [client+date]). This eliminates the need for the client to generate the idempotency key as well as the eliminates the need for the server to maintain an idempotency repository?

[deleted]

Re: APIs, robustness, and idempotency

#17

Post by Hydraulix989 »

I'm shocked at how few HTTP libraries on GitHub properly handle exponential backoff, let alone retries.

Do the Internet a favor, and file an issue with your favorite HTTP library asking them to implement exponential backoff.

I haven't found anything in JS that does this properly though. Do people really just write apps that crap out upon the first HTTP request failure?

The best library I have come across is actually SquareUp's OkHttp (the payment processing companies seem to be the only ones getting this right).

Re: APIs, robustness, and idempotency

#18

Post by brandur »

smallnamespace wrote: post #12
brandur wrote: post #8

Earlier quoted context omitted.

(I wrote this.) It's always a bit of a fine line as to what makes the final cut in this sort of article (I tried to stay on message without getting too off track), but HTTP conditional requests are definitely something that could have been a good fit. I should point out though that using `ETag`/`If-Match` generally has a slightly different use on updates compared to Stripe's `Idempotency-Key`. A server sends back an…

If you interpreted 'current state of a resource' somewhat liberally, where the resource is the abstract state of a particular versioned/timestamped request, does an ETag then really become the same as an Idempotency-Key? Or put another way, is the only difference that ETags are generally computed based on the server's stored state of a particular resource so it's possible to have multiple clients with the same ETag,…

So I think that you could absolutely patch a system that's quite similar to `Idempotency-Key` into `ETag`, but you might be pushing the original concept far enough base to the point where you're not gaining much by doing so.

Using `If-Match` is essentially indicating that you want to make a request conditionally as long as the server's state matches a nonce that you're holding. Presumably that nonce was handed to you already by the server on an initial request that you already executed.

You can hand Stripe's API an `Idempotency-Key` on the first request that you make against it. Furthermore, you'd never say that the first request made in this way was meant to be conditional, even if subsequent retries (after an initial failure) might be.

I think it wouldn't be a problem in practice to just retrofit `ETag` to do the same thing, but doing so is (arguably) semantically wrong, and I think there's something to be said for the clarity that just using your own header with an obvious name like `Idempotency-Key`.

I'm open to be persuaded though :)

Re: APIs, robustness, and idempotency

#19

Post by brandur »

majewsky wrote: post #15
pbreit wrote: post #11

Earlier quoted context omitted.

Funny you mention that, I can't recall ever seeing a proper move from /v1 to /v2!

OpenStack Keystone is at /v3 already. But yeah, it really took about 3-4 years until most consuming OpenStack services updated.

For every one successful major version revamp there may be a dozen that never changed :)

Even amongst those that did do a major API revision, I suspect that you're usually left with the no-win situation of either leaving users lingering on your previous version ~forever, or enforcing a deprecation schedule and annoying a lot of people (Twitter's V1 retirement for example).

Re: APIs, robustness, and idempotency

#20

Post by brandur »

Johnie wrote: post #14
brandur wrote: post #9

I authored this article and just wanted to leave a quick note on here that I'm more than happy to answer any questions, or debate/discuss the finer points of HTTP and API semantics ;) An ex-colleague pointed out to me on Twitter today that there are other APIs out there that have developed a concept similar to Stripe's `Idempotency-Key` header, the "client tokens" used in EC2's API for example [1]. To my knowledge th…

Have you considered using the resource identifier as an idempotency key? Basically, have the client generate the ID (UUID, but namespaced on the server side [client+date]). This eliminates the need for the client to generate the idempotency key as well as the eliminates the need for the server to maintain an idempotency repository?

I think that would be a fine alternative to the system we have.

I don't have a perfect history of events, but I suspect that our current design is basically the result of two things:

1. Idempotency keys as a concept were introduced quite a bit later than the API was originally conceived, so it made sense to make them an optional augmentation to existing integrations.

2. Our resource IDs have a fairly specific formats including a prefix (i.e. `acct_` for an account or a `ch_`). It would still be possible to generate this client-side, but it's a little extra trouble.

Post reply on HN