Live data from Hacker News

Idempotence now prevents pain later

ericlathrop.com

101–110 of 133 posts

Re: Idempotence now prevents pain later

#101
post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

well today i just learned something very important, thanks! How can I find more wisdom like this?

I wish I didn't have to be so meme'ey, but to answer your question;

By reading more Hacker News ;-)

Re: Idempotence now prevents pain later

#102
post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

well today i just learned something very important, thanks! How can I find more wisdom like this?

Stub your toe a few times until enlightenment forces its way into your field of view.

Seriously though, if you want learn a different way of building applications where loads of these things pop up, look into Actor Model. Even if you are not going to use Actor Model, reading up about it will teach you a lot about "realtime" applications... My main epiphany was that realtime applications do not exist. Rather they cannot exist. Because our reality is not realtime (or our perception of it is always delayed because our brains/awareness is slow) - we can never truly capture the present moment. Data we see is always in the past. Reality is eventually consistent (from our perspective). All things in our reality runs concurrently, stateless and in their own little time bubble. You can also read up on the concepts behind Event Sourcing.

Anyway, it is a huge rabbit hole, but reading up on the ideas behind Actor Model might just blow your mind. I'm convinced it will make a comeback because it is the natural way our reality works and Actor Model mimics it the closest, and it will also work better for multi-core cpu's. For non-tech reading, try The Power of Now by Eckhart Tolle. Then try to reflect on (relational databases) + (actor model) + (event sourcing) + (power of now) + (brain in a vat philosophy) + (try to to figure out at what speed the universe "renders", how fast our brains can think, how much delay there is, what limits do our body's sensors limits impose) + (try to figure out how closely Object Oriented programming can mimic reality (hint: all programming are tiny simulations that try to mimic some tiny part of our reality)) + (read up on the history of time keeping and the huge rabbit hole of calendars, and how most programming languages attempts to handle time (badly for the most part, it's a legit rabbit hole on it's own)).

After a year or two of hard reading, reflection and teachings (and wrecking your head), you may have good insight into the nature of programming and the fallacies of trying to build realtime systems, or trying to mimic our reality in a computer in general. It is insanely complex, we as programmers merely touch a grain of sand of the beach of complexity. We have no idea how simple our most convoluted systems are in the face of nature, yet we convince ourselves that we KNOW how things work and that we are in control at all times.

Good luck!

Re: Idempotence now prevents pain later

#103

Earlier quoted context omitted.

Have you used a iterator as the token? I had data that could be accessed and mutated from multiple different sources at the same time. With just a token and blocking the data caused race condition (cross server race conditions/deadlocks are just the worst). I solved this by giving an iterator with every read and a write required the same iterator back with the changes. If 2 servers try to write at the same time the f…

vector clocks?

I’m not familiar with Vector clocks. If it’s the same concept it’s purely by chance.

Re: Idempotence now prevents pain later

#104
post #51

Googler opinions are my own. I work on payments stuff. Payments at Google defined some specs for payment companies to build to, which allows easy onboarding as a form of payment on google's platform. We have a page talking about idempotency and expected behavior. https://developers.google.com/standard-payments/reference/id... I don't think it's that different from other payment companies, like adyen or worldpay, but…

Sorry if it's a bit off topic but are you required by your employer to tell us you work from them and whether your opinion represents them or not? I read such disclaimers a lot about people working in your company, but extremely rarely for any other company.

Another tech giant engineer here. All too often random opinionated comments get taken way out of context and misconstrued as something pertaining to the company they work for. Separate from any NDA concerns, no one wants to have some reporter quoting them as "Google insiders reveal targeting of Republicans". I can't speak for Google employees, but I certainly don't have to preface anything with "this is my own opinion", but I certainly won't say anything that implies that I'm voicing some stance or opinion of my employer.

Re: Idempotence now prevents pain later

#105
post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

This is basically like going from imperative to declarative.

Describe the final end/desired state and let the machine figure out how to get there, progressing through the work until it's caught up even across restarts or other boundaries.

Re: Idempotence now prevents pain later

#106
post #23
post #20

Idempotency is a pretty critical concept in system design, and I think most developers have run into issues related to it even if they aren't directly familiar with the term. To give another simple example as the OP - Suppose you have a product that relies on time series data. For demo purposes you might create a curated data set to present to clients, but the presenter doesn't want to show data from 2019 as the "mos…

I don't think B is technically idempotent either. Change still occurs but with minimal difference. You cannot cache the results and use them again next week. An idempotent change would be to pass in the current time instead of checking system time. In this case, as long as the input is the same, the result is the same. You could use cached results, but most likely you want to use new inputs.

Yes but this doesn't matter really. What's important is that running the script twice will still give you right values. From a practical point of vue it is the same as idempotency, which is what matters.

Re: Idempotence now prevents pain later

#107
post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

Apparently if you want spare cash, little books of this stuff could sell for $5-10. Just became aware myself, setting up to write again. That’s to say, this comment probably just saved or made me money. I’ve some hands on but clearly not as much as you exhibit. Thanks. Target audience: Sr SE, reporting to next level up and perhaps offering tradesoffs, or advising peers or reports working on those systems and need a r…

If this worked as a business model then everyone would be doing it. Everyone also includes people who don’t actually know this stuff, but want to make a quick buck, giving out bad advice. And if you don’t already have the experience, then you don’t have the knowledge to weed out bad advice.

Re: Idempotence now prevents pain later

#108

Earlier quoted context omitted.

well today i just learned something very important, thanks! How can I find more wisdom like this?

Stub your toe a few times until enlightenment forces its way into your field of view. Seriously though, if you want learn a different way of building applications where loads of these things pop up, look into Actor Model. Even if you are not going to use Actor Model, reading up about it will teach you a lot about "realtime" applications... My main epiphany was that realtime applications do not exist. Rather they cann…

Thank you, this will blow my mind. I will take this path. Real time systems is the field which is where I want to be.

Re: Idempotence now prevents pain later

#109
post #88

I have worked on lots of software that involves event-driven actions, and apply this concept throughout. "Need to send a notification email when x condition becomes true". Naive way: during processing, check the condition and call the SendEmail() function. Idempotent way: Run a query that finds all x conditions, join to a list of notifications based on email+id+time, and only if there's no entry, send the notificatio…

This is basically like going from imperative to declarative. Describe the final end/desired state and let the machine figure out how to get there, progressing through the work until it's caught up even across restarts or other boundaries.

The outcome is the same. Except that it is not declarative, because 'you' are the machine who codes up that logic.

Re: Idempotence now prevents pain later

#110

Earlier quoted context omitted.

Stub your toe a few times until enlightenment forces its way into your field of view. Seriously though, if you want learn a different way of building applications where loads of these things pop up, look into Actor Model. Even if you are not going to use Actor Model, reading up about it will teach you a lot about "realtime" applications... My main epiphany was that realtime applications do not exist. Rather they cann…

Thank you, this will blow my mind. I will take this path. Real time systems is the field which is where I want to be.

To add to this part: "My main epiphany was that realtime applications do not exist. Rather they cannot exist."

Most of us think our applications are realtime (never mind the OS schduling cpu time for your code, so inherently not realtime) but this is only because it behaves as expected, while there is not too much load nor too much data. The moment you start having significant computational load or start having too much data (aka the cpu, memory or storage starts choking), or when you start to split data apart (aka distributed applications) then the problems with "realtime" applications starts to become obvious.

If something goes wrong, to fix the immediate problems would be to retry/rerun the simulation, but that might result in duplicating your intents, which results in the simulation and its state as invalid. Then you either (hopefully) have some mechanism to de-duplicate whatever happened more than once or you have to start you simulation from scratch after fixing the data manually.

Or just design the core of the system with Idempotency in-mind from the start so you have a good way to handle such scenarios. Embrace that most things can/should be handled in a eventually-consistent way. Very few things in life has to happen RIGHT NOW (good luck with that).

An interesting spin on realtime systems are computer games, where we refresh a view of the simulation every few milliseconds (aka 60fps). Every frame gets rendered based on the inputs (keyboard/mouse)from just-after the previous frame. So in effect, the current frame being drawn is just a reaction based on previous inputs. So let me ask you this: in a game engine like this, where would you assign the value of "now"? Right after the previous frame? When capturing user input? When rendering the new frame starts? When showing the new frame but before capturing the new inputs? Where is reality's "now" moment in a computer game? Let's ignore the delays between keyboard/mouse input and drawing from CPU/GPU to the physical screen and lets assume this is a single player FPS and not a multiplayer turn-based game. And lets ignore sound processing too - just think of the simplest of game loops - should we consider them as realtime? Ponder that.

Post reply on HN