Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Systems Correctness Practices at Amazon Web Services

cacm.acm.org

Post reply on HN
Print view View original

91–100 of 143 posts

Re: Systems Correctness Practices at Amazon Web Services

#91

Post by ninetyninenine »

SteveNuts wrote: post #81
benoau wrote: post #76

Earlier quoted context omitted.

Not just 20 years old - an almost flawless 20 years at massive scale.

It's funny that things that are pinnacles of human engineering exist like this where the general public has no idea it even exists, though they (most likely) use it every single day.

I find red dead redemption 2 more impressive. I don’t know why. It sounds stupid but S3 on the surface has the simplest api and it’s just not impressive to me when compared to something like that.

I’m curious which one is actually more impressive in general.

Re: Systems Correctness Practices at Amazon Web Services

#92

Post by tossandthrow »

simonw wrote: post #87
positisop wrote: post #68

Earlier quoted context omitted.

Google Cloud Storage had it for eons before S3. GCS comes across as a much better thought-out and built product.

S3 is probably the largest object store in the world. The fact that they can upgrade a system like that to add a feature as complex as read-after-write with no downtime and working across 200+ exabytes of data is really impressive to me.

I really do respect the engineering efforts.

But object stores are embarrassingly parallel, so if such a migration should be possible somewhere without down time, then it is definitely object stores.

Re: Systems Correctness Practices at Amazon Web Services

#93

Post by rthnbgrredf »

oblio wrote: post #11
skydhash wrote: post #4

Earlier quoted context omitted.

> Proving correctness is crucial in large systems. It could be good in smaller, but critical and widely used utilities like SSH and terminals.

Yeah, basically all the coreutils plus all the common extras (rsync, ssh, etc) could use stuff like this.

It should be feasible to rewrite the coreitils like ls, cd and cp in Lean 4 together with Cursor within days. Rsync and ssh are more complex though.

Re: Systems Correctness Practices at Amazon Web Services

#94

Post by john2x »

amazingamazing wrote: post #21

> Deterministic simulation. Another lightweight method widely used at AWS is deterministic simulation testing, in which a distributed system is executed on a single-threaded simulator with control over all sources of randomness, such as thread scheduling, timing, and message delivery order. Tests are then written for particular failure or success scenarios, such as the failure of a participant at a particular stage i…

There was a talk from Joe Armstrong about using property testing to test Dropbox.

Re: Systems Correctness Practices at Amazon Web Services

#95

Post by theflyinghorse »

positisop wrote: post #68
simonw wrote: post #47

S3 remains one of the most amazing pieces of software I've ever seen. That thing a few years ago where they just added strong read-after-write consistency to the whole system? Incredible software engineering. https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...

Google Cloud Storage had it for eons before S3. GCS comes across as a much better thought-out and built product.

Maybe. But Google do have a reputation which makes selecting them for infrastructure a risky endeavor

Re: Systems Correctness Practices at Amazon Web Services

#96

Post by Lwerewolf »

amazingamazing wrote: post #74
bit_razor wrote: post #73

Earlier quoted context omitted.

https://apple.github.io/foundationdb/testing.html https://www.youtube.com/watch?v=4fFDFbi3toc

Trust me, I love FDB, but that's not the same thing. The FDB team IIRC had to write their own programming language to do this. It's not a agnostic layer above the application. The problem with coupled tooling is that no one will use it. That's what is cool about antithesis. If they're able to complete their goal, that's basically what will be achieved.

I guess you meant to say "only the people working on the software coupled to the tooling will use it". It's not just FDB & Amazon that are using something like this, and it is a ridiculously powerful type of tool for debugging distributed systems.

Re: Systems Correctness Practices at Amazon Web Services

#97

Post by amazingamazing »

Lwerewolf wrote: post #96
amazingamazing wrote: post #74

Earlier quoted context omitted.

Trust me, I love FDB, but that's not the same thing. The FDB team IIRC had to write their own programming language to do this. It's not a agnostic layer above the application. The problem with coupled tooling is that no one will use it. That's what is cool about antithesis. If they're able to complete their goal, that's basically what will be achieved.

I guess you meant to say "only the people working on the software coupled to the tooling will use it". It's not just FDB & Amazon that are using something like this, and it is a ridiculously powerful type of tool for debugging distributed systems.

ah, yes.

Re: Systems Correctness Practices at Amazon Web Services

#98

Post by SteveNuts »

ninetyninenine wrote: post #91
SteveNuts wrote: post #81

Earlier quoted context omitted.

It's funny that things that are pinnacles of human engineering exist like this where the general public has no idea it even exists, though they (most likely) use it every single day.

I find red dead redemption 2 more impressive. I don’t know why. It sounds stupid but S3 on the surface has the simplest api and it’s just not impressive to me when compared to something like that. I’m curious which one is actually more impressive in general.

Simple to use from an external interface yes, the backend is wildly impressive.

Some previous discussion https://news.ycombinator.com/item?id=36900147

Re: Systems Correctness Practices at Amazon Web Services

#99

Post by we6251 »

mjb wrote: post #77
wwilson wrote: post #51

Earlier quoted context omitted.

DST was i̶n̶v̶e̶n̶t̶e̶d̶ popularized at FoundationDB a little over a decade ago, and has been quietly gathering steam ever since. If you’re interested in the technique, the FDB paper has some good info in section 4 and section 6.2: https://www.foundationdb.org/files/fdb-paper.pdf (Disclosure: I am an author.) I also gave a talk overview of it at Strange Loop back in 2015, but don’t have the youtube link handy. If you…

Hey Will. I'm a huge fan of the work you all are doing, and of FoundationDB, but I don't believe it's accurate that DST was invented at FoundationDB (or, maybe it was, but was also used in other places around the same time or before). For example, the first implementations of AWS's internal lock service (Alf) used DST as a key part of the testing strategy, sometime around 2009. Al Vermeulen was influential in introdu…

Also a huge proponent of Antithesis and their current work, but there definitely were some notable precedents at or around that time e.g. MODIST from 2009 (https://www.usenix.org/legacy/event/nsdi09/tech/full_papers/...), which similarly tried to build a "model checker for distributed systems".

As another interesting historical side note, I have wondered about the similarities between Antithesis and "Corensic", a startup spun out of UW around a similar time period (circa 2009). Apparently, they "built a hypervisor that could on-demand turn a guest operating system into deterministic mode" (see "Deterministic Multiprocessing" at https://homes.cs.washington.edu/~oskin/). My impression is that their product was not a significant commercial success, and the company was acquired by F5 Networks in 2012 (https://comotion.uw.edu/startups/corensic/).

Overall, I don't over-index on novelty, and think it is generally good for ideas to be recycled/revived/re-explored, with updated, modern perspectives. I believe that most rigorous systems designers/engineers likely converge to similar ideas (model checking, value of determinism, etc.) after dealing with these types of complex systems for long enough. But, it is nevertheless interesting to trace the historical developments.

Re: Systems Correctness Practices at Amazon Web Services

#100

Post by gamegoblin »

tossandthrow wrote: post #92
simonw wrote: post #87

Earlier quoted context omitted.

S3 is probably the largest object store in the world. The fact that they can upgrade a system like that to add a feature as complex as read-after-write with no downtime and working across 200+ exabytes of data is really impressive to me.

I really do respect the engineering efforts. But object stores are embarrassingly parallel, so if such a migration should be possible somewhere without down time, then it is definitely object stores.

Where would you make make the cut that takes advantage of object store parallelism?

That is, at what layer of the stack do you start migrating some stuff to the new strongly consistent system on the live service?

You can't really do it on a per-bucket basis, since existing buckets already have data in the old system.

You can't do it at the key-prefix level for the same reason.

Can't do both systems in parallel and try the new one and fall back to the old one if the key isn't in it, because opens up violations of the consistency rules you're trying to add.

Seems trickier than one might think.

Post reply on HN