Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Idempotence now prevents pain later

ericlathrop.com

Post reply on HN
Print view View original

11–20 of 133 posts

Re: Idempotence now prevents pain later

#11

Post by contravariant »

sdenton4 wrote: post #6
omarhaneef wrote: post #4

I'm assuming a lot of people click on it to see what the word Idempotence means. From the article: "Idempotence is the property of a software that when run 1 or more times, it only has the effect of being run once." And the example is, instead of a chron job just running a process once a month or on some other schedule, it runs more frequently but checks if the change has already been made. (From the latin Idem which…

Mathematically, it's x^2 = x, which implies x^n = x for all positive integers n. Nilpotence (x^2 = 0) is also very helpful some times: it's a process which is self-reversing. Like the discrete Fourier transform (if you set up the constants properly).

Careful, for nilpotence the power doesn't have to be 2.

Also you may be confusing it with x^n = 1 (which I'm not sure how to name, 'root of unity' perhaps). This would be the case for the Fourier transform (with n=4).

If x^2 = 0 then applying the Fourier transform twice would null your function, which isn't the case.

Re: Idempotence now prevents pain later

#12

Post by pdpi »

sdenton4 wrote: post #6
omarhaneef wrote: post #4

I'm assuming a lot of people click on it to see what the word Idempotence means. From the article: "Idempotence is the property of a software that when run 1 or more times, it only has the effect of being run once." And the example is, instead of a chron job just running a process once a month or on some other schedule, it runs more frequently but checks if the change has already been made. (From the latin Idem which…

Mathematically, it's x^2 = x, which implies x^n = x for all positive integers n. Nilpotence (x^2 = 0) is also very helpful some times: it's a process which is self-reversing. Like the discrete Fourier transform (if you set up the constants properly).

X^2 is a weird way to describe it. A function f is idempotent if f(f(x)) = f(x).

Re: Idempotence now prevents pain later

#13

Post by void_mint »

smiley1437 wrote: post #7

I had only ever encountered idempotence in the context of system management (Ansible, Puppet, Chef, etc) Article made me think it's actually applicable to other management as well

PUT requests are intended to be idempotent, which is one of the things that distinguishes them from POST requests. This (in my experience) is most non-CS-backgrounded software developers' exposure to idempotence, but it actually has tons of value pretty much wherever you can apply it. The ability to (sometimes accidentally) do a thing twice and have it leave no unintended consequence is huge.

UPSERTs can be idempotent as well. "If this doesn't exist, create it, and if it does, update it to match this state", implies that running it twice will leave no unintended side effects.

Re: Idempotence now prevents pain later

#14

Post by bobbylarrybobby »

omarhaneef wrote: post #4

I'm assuming a lot of people click on it to see what the word Idempotence means. From the article: "Idempotence is the property of a software that when run 1 or more times, it only has the effect of being run once." And the example is, instead of a chron job just running a process once a month or on some other schedule, it runs more frequently but checks if the change has already been made. (From the latin Idem which…

When writing a Jupyter notebook, always try to make your cells idempotent. You’ll save yourself a lot of headache down the line.

Re: Idempotence now prevents pain later

#15

Post by Animats »

smiley1437 wrote: post #7

I had only ever encountered idempotence in the context of system management (Ansible, Puppet, Chef, etc) Article made me think it's actually applicable to other management as well

It's a basic property of HTTP "GET". Or it's supposed to be. "GET" is not supposed to change server site state. That's what "POST" is for. This matters if there's a cache in the middle, since caches tend to assume that GET requests are idempotent and can be served from cache. Cloudflare assumes that. POST requests have to go through to the real server.

Re: Idempotence now prevents pain later

#16

Post by throwawayboise »

omarhaneef wrote: post #4

I'm assuming a lot of people click on it to see what the word Idempotence means. From the article: "Idempotence is the property of a software that when run 1 or more times, it only has the effect of being run once." And the example is, instead of a chron job just running a process once a month or on some other schedule, it runs more frequently but checks if the change has already been made. (From the latin Idem which…

One of my first jobs was at an investment bank. They had a lot of programs that ran overnight, in a batch fashion. Everything had to be done before the markets opened the next morning. The term they used for idempotency was "free rerun." Being able to rerun any program with no special setup work was a high priority.

The value in programs being a "free rerun" was that every so often the program would barf on a bad bit of data in a record.

The programming environemnt was interpreted BASIC, so if an error occurred the program would print a message on the console and drop to an interactive prompt.

The operators running the batch schedule would see this and call the programmer on call for that night. You'd log in (over dial up at this time) and attach to the process, look at the error, figure out what went wrong, either correct the data or (more likely) skip the record and deal with it the next day. It was more important to have the programs finish on time; individual issues could be dealt with later.

Often you could just start up the program from where it left off, but if things were more screwed up it was important to be able to re-run it without any negative consequence.

Edit: this was ~30 years ago, so my point is that it's not any kind of new idea or something that wasn't recognized long ago.

Re: Idempotence now prevents pain later

#17

Post by treve »

omarhaneef wrote: post #4

I'm assuming a lot of people click on it to see what the word Idempotence means. From the article: "Idempotence is the property of a software that when run 1 or more times, it only has the effect of being run once." And the example is, instead of a chron job just running a process once a month or on some other schedule, it runs more frequently but checks if the change has already been made. (From the latin Idem which…

> And the example is, instead of a chron job just running a process once a month or on some other schedule, it runs more frequently but checks if the change has already been made.

As a property, I think it's even nicer if a script can literally fully run twice and for the outcome to be the same if it only ran once (so skipping the 'did I run before?' check).

Even though this check is useful in general, if you can define your data in such a way if it did somehow run, that this is not destructive / creates incorrect data, it makes the system more robust.

Of course this is not always possible though. For example, if the process results in an email being sent, you need an explicit check to not do that twice.

Re: Idempotence now prevents pain later

#18

Post by Justsignedup »

I've done this exact thing many times before...

I can honestly say that Eric is 100% right with his approach. It always leads to less headaches, more flexibility (oh trust me, someone is always gonna have a "but... there's like a special thing that I sometimes have to do" and it breaks some assumptions.

In any case... yeah... let's just say any time you have to be worried "did we already schedule this", really think "can this never care if it was or not? Should be always safe to schedule it again"

Re: Idempotence now prevents pain later

#19

Post by creata »

pdpi wrote: post #12
sdenton4 wrote: post #6

Earlier quoted context omitted.

Mathematically, it's x^2 = x, which implies x^n = x for all positive integers n. Nilpotence (x^2 = 0) is also very helpful some times: it's a process which is self-reversing. Like the discrete Fourier transform (if you set up the constants properly).

X^2 is a weird way to describe it. A function f is idempotent if f(f(x)) = f(x).

It's not that weird. People often write iterated composition as f^k, and this is especially true with matrices, where composition and multiplication mean the same thing.

Re: Idempotence now prevents pain later

#20

Post by brsg »

Idempotency is a pretty critical concept in system design, and I think most developers have run into issues related to it even if they aren't directly familiar with the term.

To give another simple example as the OP - Suppose you have a product that relies on time series data. For demo purposes you might create a curated data set to present to clients, but the presenter doesn't want to show data from 2019 as the "most recent"

Naturally, you decide to write a script. Do you

A) Write as script that moves the data forward by 1 week explicitly, and simply run this once per week or

B) Write a script that compares the current date to the data and moves it forward as much as it needs

At first glance, these two approaches work the same, but what if (A) triggers twice? What if it runs once every 6 days by mistake? (B) is idempotent however - subsequent executions won't change the state. It's usually impossible to predict all of the ways that software breaks, but designing with idempotency in mind eliminates a lot of them.

Post reply on HN