Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotence now prevents pain later

ericlathrop.com

Post reply on HN
Print view View original

111–120 of 133 posts

Re: Idempotence now prevents pain later

#111

Post by DharmaPolice »

MereInterest wrote: post #87
stevage wrote: post #80

Totally agree. I wish UPSERT had been invented earlier, or was a more canonical part of SQL than INSERT and UPDATE for this reason. Also, in web front end programming, there are lots of cases where X needs to cause Y to happen, but Z also needs to cause Y to happen. It's much much simpler if Y is idempotent, rather than X checking if Z has already happened etc.

I'm going to make an absolutist statement for which I will probably be corrected shortly. For any given table, interactions should either be done entirely with INSERT or be done entirely with UPSERT. The two should never mix on a single table, and UPDATE should never be used. My reasoning is that if a table represents the current state of the world, then any state changes should be made with UPSERT in order to bring…

I'm probably being stupid here but the majority of updates only involve a limited number of columns. How would you translate a statement like 'UPDATE customers SET surname='Smith' WHERE customer_id = 1234' into an UPSERT?

Re: Idempotence now prevents pain later

#112

Post by lloeki »

gregmac wrote: post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

I cannot upvote this enough.

One of the experiences I had was applying it to provisioning with idempotent shell scripts, combined with a trivial tool like https://github.com/lloeki/apply

Creating a VM in DO would add the basic SSH keys, then just run the thing against the new IP. Boom, VM ready. Want to make existing VMs config up to date? Apply. Boom.

The whole thing ridiculously scaled up in a totally unexpected way, saving hundreds of hours of headaches and mistakes.

Why not ansible/puppet? Check out the rationale in the readme.

Re: Idempotence now prevents pain later

#113

Post by chmod775 »

> 1. Query the database to find all dormant accounts with a balance, which haven't been charged the fee this month.

> 2. Charge each of these accounts a fee.

There's still a race condition between 1. and 2. if you don't atomically check the account hasn't been charged yet as you're charging it.

With long running processes like these it's also pretty much guaranteed to happen if you accidentally run two instances of this process at the same time.

Re: Idempotence now prevents pain later

#115

Post by MereInterest »

DharmaPolice wrote: post #111
MereInterest wrote: post #87

Earlier quoted context omitted.

I'm going to make an absolutist statement for which I will probably be corrected shortly. For any given table, interactions should either be done entirely with INSERT or be done entirely with UPSERT. The two should never mix on a single table, and UPDATE should never be used. My reasoning is that if a table represents the current state of the world, then any state changes should be made with UPSERT in order to bring…

I'm probably being stupid here but the majority of updates only involve a limited number of columns. How would you translate a statement like 'UPDATE customers SET surname='Smith' WHERE customer_id = 1234' into an UPSERT?

I would argue that your SQL statement should match the information that a user put in. It would be pretty rare for a user to directly input a single field change and their id. More likely, there would be some "personal information" submission, where all fields are available to edit. I would have a separate "customer_personal_info" table that gets UPSERTed with the entire contents on the submission.

One impact of this is that is forces the changes to be atomic. If an object-oriented interface updates a database whenever properties are changed, then it results in many single-column changes. In the example below, if there is a network failure between the two commands, then changing the name from "Jane Doe" to "John Smith" could result in an unintended intermediate state of "John Doe".

   customer_obj.set_forename("John")
   customer_obj.set_surname("Smith")
(Again, I am by no means an expert, and in part posted this so that I can be corrected if/where I am wrong.)

Re: Idempotence now prevents pain later

#116

Post by pc86 »

I really don't want to go on a totally off-topic tangent, but I feel like this needs to be brought up.

> We need to charge dormant customers a monthly fee so that we don't have to keep their money on our accounting books forever.

That's absolutely and objectively not the reason this happens. Keeping their money "on [your] books" has zero cost associated with it outside of the escheatment[0] process, which can be completely automated. I worked at a company that had to do a decent amount of escheatment for almost every US state, Canada, and Mexico, and for a total company size of under ten people the entire organization probably spent 45 minutes working on it in any given month. It's a trivial amount of time.

This happens to that the company can turn it's customers' money into its money. I don't know enough about to escheatment process to know for sure but I'd imagine regular debits like this would also get around the need to return the money to the estate after a given period of time. So someone puts $500 in their account, gets hit by a bus the next day, and the company gets to take a little bit every month/quarter/year until it's all theirs.

It's a pretty scummy thing to do and I'm surprised the author just takes it at face value that they "have to" do it.

[0] https://en.wikipedia.org/wiki/Escheat#Transfer_agents_and_es...

Re: Idempotence now prevents pain later

#118

Post by lathiat »

gregmac wrote: post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

The downside of this approach bares talking about. If you compute the full desired state every few minutes it can be quite expensive depending on how/what/why you are doing it. And it can also be an O(n^2) problem that is fast enough to make it into production and slow enough down the line with more data to wreak havoc.

This was common when I was writing puppet. It was easy to have your entire puppet run take minutes - and want to run every few minutes - while consuming a LOT of cpu time including on a centralised server and not just the end host.

There are of course ways and strategies to this and not every situational falls afoul of it. But it is something you very much need to be aware of.

I otherwise whole heartedly endorse this.

Re: Idempotence now prevents pain later

#119

Post by cassianoleal »

BatteryMountain wrote: post #110
unixhero wrote: post #108

Earlier quoted context omitted.

Thank you, this will blow my mind. I will take this path. Real time systems is the field which is where I want to be.

To add to this part: "My main epiphany was that realtime applications do not exist. Rather they cannot exist." Most of us think our applications are realtime (never mind the OS schduling cpu time for your code, so inherently not realtime) but this is only because it behaves as expected, while there is not too much load nor too much data. The moment you start having significant computational load or start having too m…

Going a little bit further, there's the time between the frame displaying on the screen and it reaching your retina, then the time between that and it reaching your brain in the form of electro-chemical signals, then the time between that and your brain decoding it.

By the time you perceive anything, and even more so by the time you react to it, it's already in the past.

There is no "now", it's all an approximation.

Post reply on HN