It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.
If he has such a good reputation, why doesn't he write well? He never actually introduces the point of the article, he merely alludes to it in the concluding paragraphs.
There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a propri…
The product is proprietary, but the interface and semantics are documented and freely available. "S3-compatible object store" is a commodity at this point, so it makes sense to treat it as a primitive building block for a distributed system. Of course, each implementation will have its own set of idiosyncrasies and limitations, so you'll still have to make sure that any such divergent behavior doesn't interfere with…
I'd argue that S3-compatible isn't exactly what you'd need though. You'd need S3-compatible with similar latency characteristics for read and write, with four nines of availability and 10+ nines for durability. [0]. Their metrics are, quite honestly, absolutely impossible for most engineering teams and companies to ever achieve, no matter how hard they work at it. In essence, what I mean is Cursor could migrate Origin out of Amazon's S3, but they'd lose a lot of reliability and/or performance in the process.
Great read - the main takeaway I had was “Spokes has an insane replication protocol for a database (Git) with such low write rates”. Leader/follower + LSN Binlog, WAL, whatever - just not their distributed systems nightmare coordination system
> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push. Again reminded of what an amaizing piece of engineering S3…
I really enjoyed the article. What's great about the design outlined in it is that it builds on things that work, like leveraging S3 rather than improvising your own distributed storage system.
Of course S3 goes beyond just Amazon these days. It's a proven design and architecture for doing object storage that you find across different cloud providers that you can also self host if you need to.
This article will no doubt inspire people working on projects like forgejo, gitea, tangled, etc. that are all facing the same scaling challenges.
This is an excellent post that explains why their version of git hosting is more scalable. But it seems like most people won't need it, assuming Github remains free to use and they get their availability problems fixed. Will free Git hosting continue to be sustainable, though, without architectural improvements like this?
Any ideas how these interactive animated diagrams were made?
Looks like a React component tree binding to SVG nodes. Right click and inspect element. Right click on the svg element in the (chrome) inspector. Choose `break on` > `subtree modifications`. That led me to look through the network tab for this: https://cursor.com/marketing-static/_next/static/chunks/0sqf... (look for the text "GitObject" in this file)
I am just too old, you know. I remember Bitbucket, GitLab, and the plethora of other GitHub killers. GitHub is an ecosystem, not a product. Cursor needs at least a decade to beat that! Maybe more!
Maybe what we need is actually fewer ecosystems, and more products.
People are lazy, and when there are no standards (or when they are not followed strictly), it's no small challenge to build a generic interface that works with multiple providers transparently and interchangeably. Many moons ago, we built an LOS (Loan Origination System) for subprime mortgage companies, and I designed a service that standardized input and output for dozens of mortgage vendors - credit, title - you name it. Every vendor was adding their flavor to the standards; everybody read the standards wrongly to some extent, etc. So, when we were pitching the LOS to a bunch of lenders, many of them came back to us and asked, "Can we just buy your servicing subsystem from you?" although it wasn't a separate product at the time. So, naturally, software engineers don't like one-offs unless they support one of the vendors only (pun intended) - the GOAT. It's not so easy to create or extend a standard that works across all vendor implementations. So, it's human nature to try to avoid this process and just support GitHub. The bigger the mass, the stronger the gravity. The biggest mass has the strongest gravity.
This is an excellent post that explains why their version of git hosting is more scalable. But it seems like most people won't need it, assuming Github remains free to use and they get their availability problems fixed. Will free Git hosting continue to be sustainable, though, without architectural improvements like this?
Free git is unsustainable due to AI
I can't disagree with this in the current state.
But I believe it would be easy to fix with limitation for the free accounts (CI time limit, total size of repo limit etc.)