Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Correctness Practices at Amazon Web Services

cacm.acm.org

Post reply on HN
Print view View original

41–50 of 143 posts

Re: Systems Correctness Practices at Amazon Web Services

#41

Post by jen20 »

supriyo-biswas wrote: post #18
sebstefan wrote: post #5

>Deterministic simulation. Another lightweight method widely used at AWS is deterministic simulation testing, in which a distributed system is executed on a single-threaded simulator with control over all sources of randomness, such as thread scheduling, timing, and message delivery order. Tests are then written for particular failure or success scenarios, such as the failure of a participant at a particular stage in…

Also somewhat similar to what Antithesis is doing.

Which is largely an extended version of what FoundationDB does (from many of the same people).

Re: Systems Correctness Practices at Amazon Web Services

#42

Post by hiddencost »

senthil_rajasek wrote: post #17
mlhpdx wrote: post #8

> 92% of catastrophic failures in tested distributed systems were triggered by incorrect handling of nonfatal errors This. If you take nothing else away from the article (which has a lot) take this: fail well, don’t fail poorly.

It would also be nice to list some "best practices" on how to handle non-fatal errors. I would be definitely interested to know of any sources.

One I see a lot is not being careful to use the correct error type / status code.

E.g. if you're in python and raise a value error when an API is rate limited, someone down stream from you is going to have a bad time.

Re: Systems Correctness Practices at Amazon Web Services

#43

Post by Marazan »

mlhpdx wrote: post #8

> 92% of catastrophic failures in tested distributed systems were triggered by incorrect handling of nonfatal errors This. If you take nothing else away from the article (which has a lot) take this: fail well, don’t fail poorly.

For me the most catastrophic situations happen when a fatal error is treated as a non-fatal error and suddenly instead of the system crashing the system starts promulgating nulls everywhere and into storage.

Re: Systems Correctness Practices at Amazon Web Services

#45

Post by carbondating »

sebstefan wrote: post #5

>Deterministic simulation. Another lightweight method widely used at AWS is deterministic simulation testing, in which a distributed system is executed on a single-threaded simulator with control over all sources of randomness, such as thread scheduling, timing, and message delivery order. Tests are then written for particular failure or success scenarios, such as the failure of a participant at a particular stage in…

This is what hermit does, although it's no longer actively developed.

https://github.com/facebookexperimental/hermit

Re: Systems Correctness Practices at Amazon Web Services

#46

Post by ad_hockey »

Supermancho wrote: post #27
mlhpdx wrote: post #8

> 92% of catastrophic failures in tested distributed systems were triggered by incorrect handling of nonfatal errors This. If you take nothing else away from the article (which has a lot) take this: fail well, don’t fail poorly.

How much effort should be put into "failing well"? I rather see the program crash than output a liability. Fail well is too broad to be useful, in my industry.

It gets tricky in a distributed system, or I suppose any server process. When the program crashes it just starts up again, and sometimes picks up the same input that caused the crash.

Typical example would be processing an event that you can't handle from a message queue. You don't want to crashloop, so you'd probably have to throw it away on a dead letter queue and continue processing. But then, is your system still correct? What happens if you later receive another event relating to the same entity, which depends on the first event? Or sometimes you can't even tell which entity the malformed or bug-triggering event relates to, and then it's a real problem.

Re: Systems Correctness Practices at Amazon Web Services

#47

Post by simonw »

S3 remains one of the most amazing pieces of software I've ever seen. That thing a few years ago where they just added strong read-after-write consistency to the whole system? Incredible software engineering. https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

Re: Systems Correctness Practices at Amazon Web Services

#48

Post by mjb »

Marazan wrote: post #44

Would I be right in saying Promela and SPIN are at a higher level than what is being described in the article?

I (one of the authors) did some distributed systems work with Promela about a decade ago, but it never felt like the right fit in the domain. It's got some cool ideas, and may be worth revisiting at some point.

Re: Systems Correctness Practices at Amazon Web Services

#49

Post by algorithmsRcool »

chubot wrote: post #28

One thing I wondered about the P language: It seems like in the early days, it was used at Microsoft to generate C code that’s actually used at runtime in the Windows USB stack? But now it is no longer used to generate production code? I asked that question here, which I think was the same question as in a talk: https://news.ycombinator.com/item?id=34284557 It seems like if the generated code is used in a kernel, it…

It looks like Coyote[0], which is used in azure, was an evolution of P# which was an evolution of P

[0]https://www.microsoft.com/en-us/research/wp-content/uploads/...

Re: Systems Correctness Practices at Amazon Web Services

#50

Post by skydhash »

Supermancho wrote: post #27
mlhpdx wrote: post #8

> 92% of catastrophic failures in tested distributed systems were triggered by incorrect handling of nonfatal errors This. If you take nothing else away from the article (which has a lot) take this: fail well, don’t fail poorly.

How much effort should be put into "failing well"? I rather see the program crash than output a liability. Fail well is too broad to be useful, in my industry.

In distributed systems, that means you either wants the whole system to crash (consistency) or that the node that crash is not critical for any operation and can be out until you clean out the state that makes it crash (a partition). The issue is when you fail to be in those two categories. Meaning some aspects are concurrent, but some are sequential.
Post reply on HN