Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making 768 servers look like 1

planetscale.com

Post reply on HN
Print view View original

41–50 of 89 posts

Re: Making 768 servers look like 1

#41

Post by don-code »

p1necone wrote: post #15
groundzeros2015 wrote: post #13

I disagree with the opening premise: > A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding Did you max out the capacity of the best server you can buy? Such a database can serve millions of customers (the numbers given). You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer insp…

Most non FAANG orgs could probably serve all their customers from postgres on a laptop.

Let's hypothesize this is true.

I worked at a startup that had _money_ - a lot of it - flowing in. Stakeholders wanted to see growth, which meant features. They did not want to see us making platform improvements; those didn't show well to clients. Best throw money at the problem, rather than sacrifice the all-important features.

We had a number of architectural quirks that reflected this. One was that we ran our databases in AWS on ephemeral instance store. Yes, the kind that doesn't persist when you stop or lose the instance. The kind that you should never run a database on. We did it so that we could join them eight of them together in RAID 0 on the biggest instance types, and pull 400,000 IOPS to the disk that way. We paired this with hot spares of each shard, as well as hourly backups.

We did this to support a Rails / ActiveRecord data model, for data that should never have been part of a Rails data model. We had exactly one person working part-time on query optimization.

The maintenance burden was immense. The AWS bill, even more so. But again, we are hypothesizing that we could host all of this data on a laptop.

To an extent, maybe up until our thousandth customer or so, I actually think we could have. But we wouldn't have spent most of our time working on _features_ - we'd have instead spent most of our time trying to performance optimize. That would've meant the investors wouldn't have seen growth, which in turn would've highly impacted our success. We had a well defined domain, and optimizing the queries for performance was not it.

Re: Making 768 servers look like 1

#42

Post by exiguus »

groundzeros2015 wrote: post #13

I disagree with the opening premise: > A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding Did you max out the capacity of the best server you can buy? Such a database can serve millions of customers (the numbers given). You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer insp…

As I understand the author: They describe the journey of scale from a single database server over read replicas to sharding. And its not only about a single server, its also about network (backup, restore).

Re: Making 768 servers look like 1

#43

Post by chiph »

citrin_ru wrote: post #22
drdexebtjl wrote: post #5

What about sequences? The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Is the proxy responsible for sequences? What about foreign keys? Do they all have to live on the same shard? How do you do distributed transactions? On cross-shard reads: how do you do sorting? And cross-shard joins? I’d love to be proven wrong, but I suspect the 768 servers look like 1 onl…

> The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Why not divide ID range (63bit?) by the maximum planned number of shards and then set on each shard it's min/max value so ranges will not overlap?

Because if the business grows, then you will need to add servers, and reallocate each existing shard's min/max values. If you left room to grow (divide amongst 2^31 instead of 2^63) you can go a long way before you have to solve that problem. If the business turns out to be wildly successful, you'd probably set up a whole new pool anyway and migrate over (and get a new set of pre-allocated ID ranges).

This might be the perfect case to use an UUIDv7 as a key instead of a number.

Re: Making 768 servers look like 1

#44

Post by exiguus »

themgt wrote: post #38

Even with a large database servers (10s of CPU cores, 100s of gigabytes of RAM) bottlenecks arise pretty quickly. Err, do they? For what percent of real world use cases? The database can scale to handle more traffic by adding replicas. An extreme example of this is OpenAI's use of 50 replicas on a single Primary. So an extreme example is OpenAI needing 50 replicas, but we're doing five blades ... err, we're doing 768…

Personally, I know several databases where single tables have +500GB and the database has +100TB. With this huge databases the restore and backup process over network become indeed a bottleneck. So I can agree with the author. Also, the author does not say that they can't start with a single database server and just read replicas and max hardware out. Real world use cases with +100TB I know about are Stock Market, Traffic Data, Warehouses, Analytics and Monitoring.

Re: Making 768 servers look like 1

#45

Post by groundzeros2015 »

jackb4040 wrote: post #36
groundzeros2015 wrote: post #20

Earlier quoted context omitted.

> you should in theory be able to do dev queries through that black box Because it’s a leaky abstraction which is trying to make guarantees over network connections which are extremely difficult to make within the same kernel. A few questions I would start with: - is the system even ACID compliant? In my reading of this article, no. - is my sql feature set limited? Will it enforce all constraints? Or are their cross-…

Thank you, good points. I'm just learning about this in real time. It looks like it does support transactions, but they basically destroy the performance benefits: https://vitess.io/docs/faq/sharding/advanced/can-i-use-vites... The more I read the more I'm struggling to understand the benefit of a router like this that sits on top of a monolithic SQL, vs a truly distributed DB like cockroach. Like you I'd love to lea…

What problem are you trying to solve? What did you try?

Unfortunately the SQL language is the simplest part of the database. The concurrency and consistency guarantees are the key technology,

Re: Making 768 servers look like 1

#46

Post by bddicken »

groundzeros2015 wrote: post #13

I disagree with the opening premise: > A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding Did you max out the capacity of the best server you can buy? Such a database can serve millions of customers (the numbers given). You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer insp…

Spreading requests out across hundreds, thousands, and in some cases even more is precisely what is done in the industry for big databases! Good examples:

cashapp: https://code.cash.app/planetscale-metal github: https://github.blog/engineering/infrastructure/partitioning-... etsy: https://www.etsy.com/codeascraft/migrating-etsyas-database-s...

These companies could not realistically operate off a single database server.

Re: Making 768 servers look like 1

#47

Post by bddicken »

metalliqaz wrote: post #37

what did they use to make those diagrams/animations?

Hey, author here.

Technically, they're powered by js + gsap + svg.

Process wise (for most of them) I sketched them out in advance in excalidraw for figure out layout, then passed these along to cursor to have it build out an initial draft from the image, then used some styling rules to get all the styles inline, then did a bunch of fine-tuning.

Re: Making 768 servers look like 1

#48

Post by groundzeros2015 »

bddicken wrote: post #46
groundzeros2015 wrote: post #13

I disagree with the opening premise: > A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding Did you max out the capacity of the best server you can buy? Such a database can serve millions of customers (the numbers given). You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer insp…

Spreading requests out across hundreds, thousands, and in some cases even more is precisely what is done in the industry for big databases! Good examples: cashapp: https://code.cash.app/planetscale-metal github: https://github.blog/engineering/infrastructure/partitioning-... etsy: https://www.etsy.com/codeascraft/migrating-etsyas-database-s... These companies could not realistically operate off a single database serv…

> These companies could not realistically operate off a single database server.

I want to see it fail first. I suggest their org chart has more to do with their architecture than database performance.

Even so you’re in the category I said. Hundreds of expensive engineers maintain this stuff.

Re: Making 768 servers look like 1

#49

Post by groundzeros2015 »

don-code wrote: post #41
p1necone wrote: post #15

Earlier quoted context omitted.

Most non FAANG orgs could probably serve all their customers from postgres on a laptop.

Let's hypothesize this is true. I worked at a startup that had _money_ - a lot of it - flowing in. Stakeholders wanted to see growth, which meant features. They did not want to see us making platform improvements; those didn't show well to clients. Best throw money at the problem, rather than sacrifice the all-important features. We had a number of architectural quirks that reflected this. One was that we ran our dat…

What if there was someone involved who had built web applications than run on laptops before and made different architectural decisions than the path your company went down.

Re: Making 768 servers look like 1

#50

Post by bddicken »

themgt wrote: post #38

Even with a large database servers (10s of CPU cores, 100s of gigabytes of RAM) bottlenecks arise pretty quickly. Err, do they? For what percent of real world use cases? The database can scale to handle more traffic by adding replicas. An extreme example of this is OpenAI's use of 50 replicas on a single Primary. So an extreme example is OpenAI needing 50 replicas, but we're doing five blades ... err, we're doing 768…

> So an extreme example is OpenAI needing 50 replicas, but we're doing five blades ... err, we're doing 768 servers because the need arose "pretty quickly"?

If you read the OpenAI article, you'll see that they actually used sharding to offload a bunch of work from their "1 primary 50 replicas setup"

>>> "To mitigate these limitations and reduce write pressure, we’ve migrated, and continue to migrate, shardable (i.e. workloads that can be horizontally partitioned), write-heavy workloads to sharded systems such as Azure Cosmos DB..."

The 768 servers and 1PB example is just one of many configurations. A business with 10TB may choose to go from a monolithic database to a 8-shard setup to improve backup times, eliminate single-point-of-failure, have more breathing room for scaling.

Post reply on HN