Idempotence now prevents pain later

And then the question adds state, "as of yyyy/mm/dd, which customers were more than $30 into credit?"

Edit: Even mentioning the rule to someone who asked can bring out the haters.

Particularly in this case O^2 = O makes more sense to me than f(f(x)) = f(x). And I just naturally think about A B - B A = 0 rather than f(g(x)) - g(f(x)) = 0 for commutativity.

"Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notification and save it to the list.

"Send a report at midnight". Naive way: have a cron entry that runs at midnight or run a loop that checks the time, and if midnight, sends the report. Idempotent way: cross-check against a list of reports that should have been sent, and if missing, send it. Run this check every few minutes.

The nice thing about this is it doesn't stop you from also doing the event-based stuff (for "real-time" notifications, for example), so long as it's the secondary approach from design point of view. Ironically when everything is working fine, the event-driven stuff will actually be doing all the work -- but as soon as there's a failure (like, the system happens to reboot at 23:59) you'll be grateful to build this way.

I could write paragraphs about the nuance, different approaches, failure modes and the trade-offs with all this -- suffice to say treating events as optional in event-based systems has served me well.

A couple of key aspects are:

1. The payments API must be idempotent for the Cron to be idempotent. Otherwise they can't safely retry timeout failures.

2. The operations and have to be (to loosely use the term) atomic.

For #1, a payment processor should explicitly state idempotency semantics. Look at Stripe [1] for example. They require you to pass Idempotency Key header in charge request. Clients must carefully choose this key to fit the needs for their business flow and use case. For the author since they charge monthly the idempotency key must be such that it should be constant for a given customer and given month. So something like "customerid-YYYY-MM" makes sense. Now no matter which day of the month cron runs the customer is guaranteed to charge at most once.

As for #2 most of data stores now a days support atomic operations so it should be reasonably straight forward. Get-and-set in Redis, start/end transaction with "for update" in MySQL etc.,

Note that #2 is optional if you don't mind spurious retries to Stripe.

Stepping back from the details it can be noticed that for a system/operation to be idempotent all its downstream dependencies should support idempotent operations. In this example, if Stripe weren't idempotent then you have to resort to manual fixes if charge operation times out.

Another observation is, if a downstream system can assure idempotency then don't bother making your operation idempotent. Failed payment? Retry. No need to store if a payment was attempted or not. Make sure the idempotency-key generation logic is robust.

PS: Source - I've been in payments industry for a while so spend a lot of time thinking about and implementing idempotency. The latest one being when my team worked on payments API at Uber. We included a idempotency in the glossary [2] section ;-)

[1] https://stripe.com/docs/api/idempotent_requests [2] https://developer.uber.com/docs/payments/glossary