Possibly way OT but this reminds me of a question I had. Lets say I have a task that creates a daily invoice that failed to run for X days, but when it's run again it needs to go back and create X invoices from when it was last run. Is there a name / concept for that?
Idempotence now prevents pain later
61–70 of 133 posts
Re: Idempotence now prevents pain later
#62Earlier quoted context omitted.
The article didn't really go into the details of the implementation but my mental image was basically your second proposal: the deduction of the fee should happen in the same database transaction as the updating of a "fee-last-charged" timestamp, relying on the database for thread safety and allowing the cron job itself to be stateless (and inherently protected against simultaneous execution of multiple instances of…
This could still lead to a double-charge, though. If some rogue deploy script creates fifty of the cron job, and weirder things happen every day, one of those could be checking the stale state during the transaction of another one. A classic data race. Eliminating this problem is left as an exercise for the interested reader...
The way to fix that is to remove state altogether.
1. Balance is not a simple variable, but the sum of all credits and debits to an account
2. A fee is a charge record in your database
3. This fee has a database constraint that you can have only one record per month
Re: Idempotence now prevents pain later
#63Earlier quoted context omitted.
The article didn't really go into the details of the implementation but my mental image was basically your second proposal: the deduction of the fee should happen in the same database transaction as the updating of a "fee-last-charged" timestamp, relying on the database for thread safety and allowing the cron job itself to be stateless (and inherently protected against simultaneous execution of multiple instances of…
This could still lead to a double-charge, though. If some rogue deploy script creates fifty of the cron job, and weirder things happen every day, one of those could be checking the stale state during the transaction of another one. A classic data race. Eliminating this problem is left as an exercise for the interested reader...
At least one of the two cron jobs will fail to commit its transaction though.
Re: Idempotence now prevents pain later
#64I had only ever encountered idempotence in the context of system management (Ansible, Puppet, Chef, etc) Article made me think it's actually applicable to other management as well
PUT requests are intended to be idempotent, which is one of the things that distinguishes them from POST requests. This (in my experience) is most non-CS-backgrounded software developers' exposure to idempotence, but it actually has tons of value pretty much wherever you can apply it. The ability to (sometimes accidentally) do a thing twice and have it leave no unintended consequence is huge. UPSERTs can be idempoten…
In the last couple years I've come to realize that nearly all POSTs should actually be PUTs, sometimes with a client-provided uniqueness token in the URL (like a UUID) that becomes part of the newly-created resource's ID.
Now that I think about it, all the situations where I'm fine using POST are also idempotent, such as changing an object's metadata (technically PUT or PATCH), or sending a batch of HTTP requests to an endpoint bundled as a single request.
Re: Idempotence now prevents pain later
#65Possibly related past threads: What Is Idempotence? - https://news.ycombinator.com/item?id=19570815 - April 2019 (51 comments) Idempotence: What is it and why should I care? - https://news.ycombinator.com/item?id=17804617 - Aug 2018 (73 comments) You know how HTTP GET requests are meant to be idempotent? - https://news.ycombinator.com/item?id=16964907 - May 2018 (304 comments) Implementing Stripe-Like Idempotency Key…
When the same thread is appears on HN twice does it get the same results?
It's idempotent except it calls a RNG.
Re: Idempotence now prevents pain later
#66This is closely related to commutativity, which is another useful property of a system, especially a distributed one, where events commonly arrive out of order.
Re: Idempotence now prevents pain later
#67What if it takes more than an hour to get through the initial list of customers?
Re: Idempotence now prevents pain later
#68Possibly related past threads: What Is Idempotence? - https://news.ycombinator.com/item?id=19570815 - April 2019 (51 comments) Idempotence: What is it and why should I care? - https://news.ycombinator.com/item?id=17804617 - Aug 2018 (73 comments) You know how HTTP GET requests are meant to be idempotent? - https://news.ycombinator.com/item?id=16964907 - May 2018 (304 comments) Implementing Stripe-Like Idempotency Key…
When the same thread is appears on HN twice does it get the same results?
Yes, it always results in at least one “this has been posted before” post.
Re: Idempotence now prevents pain later
#69I had to solve this for charges and interest and I solved it in a more elegant way than described (I think).
Using mutation iterator, last update time and knowing the rate of change over time you can back apply any any calculations that have been missed regardless of how long it’s been since your cron job ran.
I setup a server to run the job on startup and 1 time a day at midnight. This also allows multiple different charges and interests to be applied to the same account. I also set it up to apply this process every time an account was accessed to make sure the accounts were up to date.
Re: Idempotence now prevents pain later
#70It's 2021 and we still don't have scalable computer systems with continuations support - that's why his cronjob does not work reliably in the first manner if that script crashed or the machine reboots, it will run again.
The idempotence is nice and I use it as much as possible, but, his case is very simplistic. Imagine you have a complex data migration process, which crashes in the middle - how you safely continue from where you crashed or safely roll back?