Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When "letting it crash" is not enough

flawless.dev

Post reply on HN
Print view View original

71–80 of 84 posts

Re: When "letting it crash" is not enough

#71

Post by dunham »

elbasti wrote: post #66

IMHO the beauty of "let it crash" is that you can code very tersely while maintaining data safety. In order for "let it crash" to work you need two things: 1. A tech stack that isolates crashes such that they do not affect the rest of the system. Example: you receive a malformed API response. The code responsible for parsing it crashes, but the rest of your application does not. 2. You use this to code in a declarati…

The erlang thing has bitten me in the past.

On my laptop, RabbitMQ would get into a state where it was unavailable and taking 100% CPU. I think it happened when I switched networks (moved laptop home) and the spinning was from repeatedly crashing. I never sorted the root cause, just wrote a script to kill and restart rabbitMQ. (The script to kill rabbit was named 'fudd'.)

Re: When "letting it crash" is not enough

#72

Post by conradfr »

MauranKilom wrote: post #28

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

About your first point, in a Beam app with a supervision tree you don't necessarily need to restore your entire app state or the state "it was in before", you can restart with just a "workable" state.

Re: When "letting it crash" is not enough

#73

Post by gregw2 »

Animats wrote: post #12

> .. durable execution, and is so new that most developers never have heard of it. It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system. Tandem's whole system concept was built around that. Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world object…

Good point about Tandem, but it wasn't limited to them.

Checkpoint/restart has also long been a high-end capability found in HPC (High Performance Computing) systems. I recall it even it made it onto some commercial UNIXes, although I don't remember whether it was bundled in or took the form of some additional layered software such as LSF (Load Sharing Facility) (example: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=admini... ).

There are some people who have tried to bring some checkpoint/restart to Linux. For example, see the CRIU project: https://criu.org/Checkpoint/Restore

I acknowledge that the original poster is talking about all this being done at the application level, not outside-the-app at the OS level...

For more references on the topic, see the wikipedia entry for Application checkpointing: https://en.wikipedia.org/wiki/Application_checkpointing

Re: When "letting it crash" is not enough

#74

Post by JohnFen »

JonChesterfield wrote: post #52
Animats wrote: post #12

> .. durable execution, and is so new that most developers never have heard of it. It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system. Tandem's whole system concept was built around that. Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world object…

There's a decent chance checkpoint/restart has been with us since the days of paper tape. It's moderately amusing to see "save state somewhere" described as so new people haven't heard of it.

I've been in the industry for so long now as to see this pattern repeat often enough that I've come to accept it as a sort of universal truth: nearly everything that people think is "new" is actually an echo and (hopefully) refinement of something that has come before.

I am hard-pressed to think of anything in the software world today that is actually, truly, new. We all stand on the shoulders of giants.

Re: When "letting it crash" is not enough

#75

Post by solatic »

nelsonic wrote: post #36
solatic wrote: post #6

Flawless sounds a lot like https://temporal.io/ . I'm wondering if it has the same scalability concerns - sticking everything in Postgres is fine at small-ish scale, but what happens when you outgrow Postgres, either because you have higher availability requirements (can't handle primary DB restarts) or because of the sheer volume of the workload?

What percentage of apps “outgrow” Postgres?

Very few. But there's a difference between having control over the codebase where you can start to split up the monolith, adopt event-based architectures down the line, i.e. having a strategy on how to deal with the problem if and when it becomes a problem, versus tightly coupling your business logic to a vendor that is tightly coupled to Postgres and therefore your options on how to switch away will be extremely fragile and expensive at best and straight up impossible at worst.

Re: When "letting it crash" is not enough

#77

Post by yencabulator »

MauranKilom wrote: post #28

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

This sort of architecture often shows up in actors, e.g. https://www.microsoft.com/en-us/research/publication/orleans...

In that world, what you're generally looking at is

local state + incoming message -> new state + outgoing messages

Outgoing messages are sent only after persisting, and will be retried until success. Unique message IDs are used for idempotency (also for incoming messages).

Important: the actor runs with no side effects -- that's what makes rerunning things safe. In that kind of world, side effects are often achieved via e.g. sending a message that updates a materialized view somewhere (see e.g. Kafka).

With this setup, the source of badness is often isolated to the incoming message, and after a few failures an incoming message can be moved into a "dead letter queue" for ops to look at. In many scenarios, this actually works remarkably well.

https://pmatseykanets.github.io/beanstalkd-docs/protocol/#bu...

Re: When "letting it crash" is not enough

#78

Post by yencabulator »

JonChesterfield wrote: post #52
Animats wrote: post #12

> .. durable execution, and is so new that most developers never have heard of it. It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system. Tandem's whole system concept was built around that. Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world object…

There's a decent chance checkpoint/restart has been with us since the days of paper tape. It's moderately amusing to see "save state somewhere" described as so new people haven't heard of it.

The challenge is purely in how to make it perform well.

Dumping the whole memory was a lot more viable when that was only a few kilobytes.

Re: When "letting it crash" is not enough

#80

Post by dnr »

solatic wrote: post #6

Flawless sounds a lot like https://temporal.io/ . I'm wondering if it has the same scalability concerns - sticking everything in Postgres is fine at small-ish scale, but what happens when you outgrow Postgres, either because you have higher availability requirements (can't handle primary DB restarts) or because of the sheer volume of the workload?

Temporal can also run on Cassandra, which scales much larger than Postgres (if you put in enough effort). It can also be replicated across regions for high availability. It's already running some pretty huge use cases.

(I work at Temporal)

Post reply on HN