Idempotence now prevents pain later • Hacker News Board

Re: Idempotence now prevents pain later

#121

Post by gregmac » Fri, Apr 09, 2021, 3:17 PM UTC

lathiat wrote: post #118

gregmac wrote: post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

The downside of this approach bares talking about. If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. And it can also be an O(n^2) problem that is fast enough to make it into production and slow enough down the line with more data to wreak havoc. This was common when I was writing puppet. It was easy to have your entire puppet run take minutes…

> If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it.

Definitely! And on top of that, you probably want to apply some business logic to what you're doing.

To expand on the example of "Report should run at midnight":

If you run the check regularly, you can constrain the date range you're checking to just since the last time you checked, and store this date either in memory or persisted depending on your use case, technology, and how you want system failures/restarts handled.

Similarly, if the system was down for several days, it might not be relevant to send all missing reports, but instead just the last one.

On the other hand, if "daily reports" are required for auditing purposes, it probably does make sense to send all of them. But you have to be careful so on the very first run it doesn't go "Hey, there are 18,726 missing reports since Jan 1, 1970!"

It's also important to write the report itself to be idempotent. You can't write it so it reports to "current" time with the assumption it runs at exactly midnight. Instead you have to actually constrain the dates to exactly what you want. Even aside from the idempotent invocation I'm talking about, as your system gets big you'll get thousands of reports that all need to run at "midnight" and that just isn't practical: maybe they run all sequentially, and some will not actually execute until several minutes after midnight.

Re: Idempotence now prevents pain later

#122

Post by TeMPOraL » Fri, Apr 09, 2021, 3:18 PM UTC

BatteryMountain wrote: post #110

unixhero wrote: post #108

Earlier quoted context omitted.

Thank you, this will blow my mind. I will take this path. Real time systems is the field which is where I want to be.

To add to this part: "My main epiphany was that realtime applications do not exist. Rather they cannot exist." Most of us think our applications are realtime (never mind the OS schduling cpu time for your code, so inherently not realtime) but this is only because it behaves as expected, while there is not too much load nor too much data. The moment you start having significant computational load or start having too m…

> An interesting spin on realtime systems are computer games (...) Ponder that.

My younger self did that, quite a lot, thank you. I never felt comfortable with various approach of intermixing input handling, physics/game logic and rendering - whether to run them sequentially, decouple some or all of them, whether the physics should be "look-ahead" or "look-behind"... I ended up picking the pattern that leads to the most stable and deterministic behavior (look-behind physics in lockstep with input and on fixed update rate, rendering independent and done variable-rate), and stopped thinking about it.

So thanks for mentioning it in this context, it makes me more comfortable to hear from someone else that "now" in games is ill-defined on philosophical level, and I shouldn't have worried about it as much as I did.

Re: Idempotence now prevents pain later

#123

Post by DharmaPolice » Fri, Apr 09, 2021, 3:22 PM UTC

MereInterest wrote: post #115

DharmaPolice wrote: post #111

Earlier quoted context omitted.

I'm probably being stupid here but the majority of updates only involve a limited number of columns. How would you translate a statement like 'UPDATE customers SET surname='Smith' WHERE customer_id = 1234' into an UPSERT?

I would argue that your SQL statement should match the information that a user put in. It would be pretty rare for a user to directly input a single field change and their id. More likely, there would be some "personal information" submission, where all fields are available to edit. I would have a separate "customer_personal_info" table that gets UPSERTed with the entire contents on the submission. One impact of this…

>I would argue that your SQL statement should match the information that a user put in.

I'm skeptical this is desirable even just from a security perspective.

I think you're considering the role of the database in a narrower context than they're used. Certainly the databases I work with, it's a minority of updates (actually a very small minority) that are performed as a result of a user inputting something into a form in the sense you're talking (i.e. updating an entity). Think of how many updates are of the form update this order to change status to shipped or product stock available to X. You wouldn't want to be obligated to pass the entire row of something to an application so it could pass back a single changed column.

This is not to even get into the scenario where an update is applied to a view (which might be a limited result from one or more tables). In that sort of scenario it's not even clear what an UPSERT would even do.

Re: Idempotence now prevents pain later

#124

Post by nikolay » Fri, Apr 09, 2021, 4:08 PM UTC

stevage wrote: post #81

nikolay wrote: post #70

It's 2021 and we still don't have scalable computer systems with continuations support - that's why his cronjob does not work reliably in the first manner if that script crashed or the machine reboots, it will run again. The idempotence is nice and I use it as much as possible, but, his case is very simplistic. Imagine you have a complex data migration process, which crashes in the middle - how you safely continue fr…

SOmetimes you can write the migrations as a set of idempotent operations: "If table doesn't have column X, add it. If row Y doesn't exist in table Z, add it." etc.

Well, I did not mean a sequential one and only involving the database, but, let's say, you have forks (i.e. parallel executions) and merges (waiting to sme dependent tasks to complete), etc. What if execution halts in the middle of a step and you don't really have transaction support outside of the DB? I typically would use a workflow engine, but rollback is not something workflow engines support either.

Re: Idempotence now prevents pain later

#125

Post by JamesSwift » Fri, Apr 09, 2021, 4:48 PM UTC

bob1029 wrote: post #24

I feel like immutability and a 100% leak-proof layer of domain methods which in turn manage domain mutations would ultimately bring more value than explicitly adding idempotence throughout. If I have an idempotent method like "CreateCustomerRecord", this can cause a lot of pain for audit features and other aspects of the domain model if it is internally making determinations about whether to actually create or silent…

The state machine is great in a vacuum but it falls apart when someone trips over the power cord at exactly the wrong time. Thats the benefit of idempotent design, that it is resilient to real world failure modes that dont typically get modeled into the systems you describe.

Re: Idempotence now prevents pain later

#126

Post by JamesSwift » Fri, Apr 09, 2021, 4:54 PM UTC

lathiat wrote: post #118

gregmac wrote: post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

The downside of this approach bares talking about. If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. And it can also be an O(n^2) problem that is fast enough to make it into production and slow enough down the line with more data to wreak havoc. This was common when I was writing puppet. It was easy to have your entire puppet run take minutes…

Curious to hear what ways and strategies you considered and landed on.

Re: Idempotence now prevents pain later

#127

Post by mateo411 » Fri, Apr 09, 2021, 9:06 PM UTC

catlifeonmars wrote: post #66

This is closely related to commutativity, which is another useful property of a system, especially a distributed one, where events commonly arrive out of order.

Also associativity, which is more than two events occurring in any order.

Interesting I know floating point arithmetic is not associative, but I think it is communitative.

Re: Idempotence now prevents pain later

#128

Post by mandeepj » Sat, Apr 10, 2021, 12:34 AM UTC

gregmac wrote: post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

> "Send a report at midnight". Naive way: have a cron entry that runs at midnight or run a loop that checks the time, and if midnight, sends the report. Idempotent way: cross-check against a list of reports that should have been sent, and if missing, send it. Run this check every few minutes.

Ha! You are biased toward the word "Idempotent way". Above - you have totally different steps under "Idempotent way". So, your process is different for each of them. There's no tech differentiator and it should be - as that's how you implement "Idempotent way"

Re: Idempotence now prevents pain later

#129

Post by 91aintprime » Sat, Apr 10, 2021, 5:08 AM UTC

gregmac wrote: post #121

lathiat wrote: post #118

Earlier quoted context omitted.

The downside of this approach bares talking about. If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. And it can also be an O(n^2) problem that is fast enough to make it into production and slow enough down the line with more data to wreak havoc. This was common when I was writing puppet. It was easy to have your entire puppet run take minutes…

> If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. Definitely! And on top of that, you probably want to apply some business logic to what you're doing. To expand on the example of "Report should run at midnight": If you run the check regularly, you can constrain the date range you're checking to just since the last time you checked, and stor…

suppose you serve clients who want their reports to be generated at 02:30 local time. you could try to run a cron container for every time zone :bleh:, but with daylight savings, the 2:30 event can fire twice, or not at all!

instead you simply see if there's a report for yesterday, and if there isn't make one.

the edge cases are still there, see replies

Re: Idempotence now prevents pain later

#130

Post by nanis » Sat, Apr 10, 2021, 10:01 AM UTC

lloeki wrote: post #112

gregmac wrote: post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

I cannot upvote this enough. One of the experiences I had was applying it to provisioning with idempotent shell scripts, combined with a trivial tool like https://github.com/lloeki/apply Creating a VM in DO would add the basic SSH keys, then just run the thing against the new IP. Boom, VM ready. Want to make existing VMs config up to date? Apply. Boom. The whole thing ridiculously scaled up in a totally unexpected wa…