Git at any scale
cursor.com
Git at any scale
1–10 of 128 posts
Re: Git at any scale
#2Related:
Cursor Origin Code Hosting
Re: Git at any scale
#3Remember, don't put large objects in cnt, it makes it a real pain later.
Re: Git at any scale
#4> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push.
Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]
1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
Re: Git at any scale
#5> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push. Again reminded of what an amaizing piece of engineering S3…
Not just durability. But read after write guarantee on a distributed object store.
How many startups and corporations built on top of S3 semantics?
Snowflake and the big lakehouses are one of them. They use S3 as giant disk in the sky.
Re: Git at any scale
#6It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.
Re: Git at any scale
#7Remember, don't put large objects in cnt, it makes it a real pain later.
cnt?
Re: Git at any scale
#8> The "fan-out" is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it.
Doesn't 3PC require all nodes to agree, not just a majority?
Re: Git at any scale
#9It doesn't matter how many technical blog posts they put out, I will still never host my code with Elon Musk.
Re: Git at any scale
#10There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a proprietary product, not an algorithm.