Systems Correctness Practices at Amazon Web Services

Post by ajb » Sat, May 31, 2025, 5:56 AM UTC

nine_k wrote: post #109

amazingamazing wrote: post #21

> Deterministic simulation. Another lightweight method widely used at AWS is deterministic simulation testing, in which a distributed system is executed on a single-threaded simulator with control over all sources of randomness, such as thread scheduling, timing, and message delivery order. Tests are then written for particular failure or success scenarios, such as the failure of a participant at a particular stage i…

I suspect you can relatively easily obtain a completely deterministic machine by running QEMU in 100% emulation mode in one thread. But what you are after is controlled deterministic execution, and it's far harder. That is, making your multiple processes to follow a specific dance that triggers an interesting condition must be very involved, when seen from the level as low as CPU and OS scheduler. Hence a language-ag…

Post by [deleted] » Sat, May 31, 2025, 6:27 AM UTC

riknos314 wrote: post #108

simonw wrote: post #47

S3 remains one of the most amazing pieces of software I've ever seen. That thing a few years ago where they just added strong read-after-write consistency to the whole system? Incredible software engineering. https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

I had the distinct pleasure of working on S3 (Lifecycle) during the timeframe that the index team was working on the rearchitecture that enabled the read-after-write consistency. I can confidently say that as impressive as S3 is from the outside, it's at least that impressive internally, both in implementation, and organizational structure.

Post by smj-edison » Sat, May 31, 2025, 7:01 AM UTC

crvdgc wrote: post #78

amazingamazing wrote: post #21

> Deterministic simulation. Another lightweight method widely used at AWS is deterministic simulation testing, in which a distributed system is executed on a single-threaded simulator with control over all sources of randomness, such as thread scheduling, timing, and message delivery order. Tests are then written for particular failure or success scenarios, such as the failure of a participant at a particular stage i…

https://rr-project.org/ for languages that can be debugged by gdb.

Post by ninetyninenine » Sat, May 31, 2025, 7:34 AM UTC

sriram_malhar wrote: post #124

ninetyninenine wrote: post #91

Earlier quoted context omitted.

I find red dead redemption 2 more impressive. I don’t know why. It sounds stupid but S3 on the surface has the simplest api and it’s just not impressive to me when compared to something like that. I’m curious which one is actually more impressive in general.

That's the strangest comparison I have seen. What axis are you really comparing here? Better graphics? Sound?

Post by sylware » Sat, May 31, 2025, 9:47 AM UTC