Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When "letting it crash" is not enough

flawless.dev

Post reply on HN
Print view View original

31–40 of 84 posts

Re: When "letting it crash" is not enough

#31

Post by albertzeyer »

mark_undoio wrote: post #27
albertzeyer wrote: post #25

As it was mentioned, this sounds very much like time-traveling debuggers like RR, which also records such database. But I don't exactly understand: If you exactly recover the same state, don't you end up with the same faulty undesirable state? (That's what you also get with RR, by intention, to debug the problem.) Clearly, here this is not the intention, i.e. you don't want to recover exactly the same state. So, it m…

I guess it insulates you against bugs in the VM implementation, plus against (transient) failures of the host system.

Ah, right, that would be an option to define a clear boundary, i.e. to recover exactly the VM state, but not the OS native state. I was thinking about a native app here, where there is no VM.

But this would cover only some specific set up failures, namely where something goes wrong in the host, which you can easily recover by resetting its state and retrying again. This is only a very limited amount of failures. I guess most failures come from bugs in the user code, which would all be state within the VM.

Also, maybe your app depends on some resources from network, e.g. some NFS, or maybe some other remote server, or whatever, which you anyway cannot control. This is not easily recoverable then. (Tools like https://criu.org/Main_Page, which try to serialize an app state, to be able to recover it later, have the same problem.)

Re: When "letting it crash" is not enough

#32

Post by [deleted] »

vilhelm_s wrote: post #26
notfed wrote: post #20

> This brings me to a recent discovery I made, another approach to dealing with failure that completely blew my mind . It's commonly known under the name durable execution, and is so new that most developers never have heard of it. Some feedback: this paragraph lands like a Reddit post from a teenager, who just took a big drag off a joint, and thinks they've invented a new field of science. (And also hasn't completed…

They're not claming that they invented it, it's an existing term. This blogpost lists 14 different projects in the space: https://www.golem.cloud/post/the-emerging-landscape-of-durab...

[deleted]

Re: When "letting it crash" is not enough

#34

Post by treflop »

I’ve been doing this for a long time, except I just write basically a memoize function that I slap onto any old function.

It’s not a real error handling solution. I only use it for my quick and dirty projects.

Many errors are recoverable but the ones that are not fuck up this scheme because a human needed to have written a proper error recovery scheme that isn’t just “retry over and over from a previous point.”

At the end of the day, you gotta put in manual work for error handling. You can’t rely on your language or any library to create a generic error handler.

Re: When "letting it crash" is not enough

#35

Post by hedora »

sausagefeet wrote: post #15
hedora wrote: post #7

Earlier quoted context omitted.

Exactly Once is basically a solved problem. The CAP theorem says you can either get consistency or availability in a system that has network partitions. If you give up consistency, you end up corrupting data every once in a while. If you don’t, then you can have exactly once, but it might take a long time if there’s a network failure. NFSv3 solved this back in the 1980’s. (V2 and V1 may have, but I don’t know.) It di…

This comment has some errors in it: 1. Exactly Once is absolutely not a solved problem. CAP says nothing about 'exactly once', it's about design choices for your data. But you can do all sorts of side-effectful things when it comes setting the data. 2. Giving up consistency does not mean you "corrupt your data every once in awhile". I don't know why you would say that. Choosing availability means you have to make dec…

If your system provides strong consistency, it is possible to build exactly once processing over it. NFSv3 is an existence proof, and there are plenty of theoretical results showing it is possible. Roughly speaking, you run the job and install the result in the consistent store iff your output register is null. If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed, but the system in the article precludes that, as do many existing systems.

As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”.

Unless you layer consistency on top of that (which is not always possible), then, tautologically, you don’t get consistency. In particular, most intuitive application-level invariants are going to be violated in all sorts of bizarre ways that take many pages to explain.

In practice, eventual consistency is closer to corrupting than not corrupting, because most application developers aren’t going to follow this conversation, and will use the database wrong.

I’m sure you or I could model our program and the storage in TLA+ and confirm we’re correctly maintaining application state, but that doesn’t help most developers.

Also, it’s not economically efficient.

We’re talking about “only” getting 6 nines instead of 7 because we chose CP instead of AP, but as a side benefit the system is easier to maintain, and it took less than 1/10th as much to implement, and has orders of magnitude fewer implementation bugs.

Re: When "letting it crash" is not enough

#36

Post by nelsonic »

solatic wrote: post #6

Flawless sounds a lot like https://temporal.io/ . I'm wondering if it has the same scalability concerns - sticking everything in Postgres is fine at small-ish scale, but what happens when you outgrow Postgres, either because you have higher availability requirements (can't handle primary DB restarts) or because of the sheer volume of the workload?

What percentage of apps “outgrow” Postgres?

Re: When "letting it crash" is not enough

#37

Post by sausagefeet »

hedora wrote: post #35
sausagefeet wrote: post #15

Earlier quoted context omitted.

This comment has some errors in it: 1. Exactly Once is absolutely not a solved problem. CAP says nothing about 'exactly once', it's about design choices for your data. But you can do all sorts of side-effectful things when it comes setting the data. 2. Giving up consistency does not mean you "corrupt your data every once in awhile". I don't know why you would say that. Choosing availability means you have to make dec…

If your system provides strong consistency, it is possible to build exactly once processing over it. NFSv3 is an existence proof, and there are plenty of theoretical results showing it is possible. Roughly speaking, you run the job and install the result in the consistent store iff your output register is null. If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notificati…

> If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed

If you are getting duplicate notifications, then you don't have an "exactly once" system, but definition. Exactly once systems are not possible, bu your own explanation. You can store data in a durable way such that applying it multiple times is safe, but that is not "exactly once". Your options are at-least once or at-most once. Again, by your own statement: if you have duplicate notifications in your system, you do not have "exactly once".

> As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”.

This is not data corruption, this is the semantics of an eventually consistent system. If you don't want those semantics, don't have an eventually consistent system, but that is entirely different than data corruption. Developers not understanding the system is different than corruption.

I agree that eventual consistent is probably not a good idea for most problems.

Re: When "letting it crash" is not enough

#38

Post by bkolobara »

MauranKilom wrote: post #28

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

Hi! I'm the author of the essay.

Durable execution is meant to complement your application. You will never want to model everything with it. It solves the problem of needing to decide how often to manually make snapshots of some important state, this becomes implicit. Workflows in flawless can still fail, you could call the `panic` function, or divide by zero. In the end it's arbitrary compute.

"External" state is one of the text book examples for using durable execution. If you are interacting with 5 different services and calling 5 different API endpoints, you sometimes want to have transactional behaviour. Leave all 5 systems in a consistent state after your interaction. You can't only call 2 and stop. Durable execution and patterns like saga [1] are one of the most straight forward ways (for me) to solve this.

In flawless specifically, I try to give enough context to the user why things failed. It's very easy to reconstruct the whole computation from the log. And let the user decide if they want to re-run the workflow. If you charge someone's credit card, but the call to extend their subscription fails (service down), you can't safely just re-run this. You have two choices, either you continue progressing and roll back the charge, or you fail and have someone manually look at it. In general, you want to use flawless in scenarios where the "called exactly once" guarantee is important. If you can just throw away the state and it's safe to re-run from the start, then you don't need flawless for this part of the app. The less state you have to care about, the better.

EDIT: The alternative would be to manually construct a state machine with a database. "Check if the credit card was charged. Call Stripe. I finished charging the credit card, save this information. Call the subscription service, it failed, restart everything. Check if the credit card was charged ...". And depending on your workflow, this can be a very complicated process where 90% of your code is just dealing with possible failures. Especially if failures happen on the edge of some calls it can become very tricky.

[1]: https://medium.com/cloud-native-daily/microservices-patterns...

Re: When "letting it crash" is not enough

#39

Post by baby »

Big problem with this approach is doing the same thing twice, which is a big no no in a lot of applications. For example, if you do something, then crash before you get the chance to update your durable execution object, then you’ll do it again after restarting.

We were dealing with such code in a TCB and the only way is to audit your code and ensure that any action that can only be done once at most saves that action before doing the action.

I think you have the same concepts in queues no? Some queues will repeat at most once, some at least once, something like that. I can’t remember.

Re: When "letting it crash" is not enough

#40

Post by zelphirkalt »

baby wrote: post #39

Big problem with this approach is doing the same thing twice, which is a big no no in a lot of applications. For example, if you do something, then crash before you get the chance to update your durable execution object, then you’ll do it again after restarting. We were dealing with such code in a TCB and the only way is to audit your code and ensure that any action that can only be done once at most saves that actio…

I guess you could then use something like undo-redo-logging or a database, if it is critical. But then you would need to store data and process it as well, writing the logic for that.
Post reply on HN