Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we built a serverless SQL database

cockroachlabs.com

Post reply on HN
Print view View original

41–50 of 122 posts

Re: How we built a serverless SQL database

#41

Post by phoboslab »

mjb wrote: post #40
phoboslab wrote: post #37

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…

What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability. What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distri…

Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture.

When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.

Re: How we built a serverless SQL database

#42

Post by brokencode »

phoboslab wrote: post #37

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…

The question isn’t whether you have 10k req/s now, but whether you expect to in the future. If you are designing a blog, then yeah, you probably don’t need to worry about it. If you are starting a social network or SAAS business application, then you probably do.

Re: How we built a serverless SQL database

#43

Post by mjibson »

phoboslab wrote: post #41
mjb wrote: post #40

Earlier quoted context omitted.

What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability. What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distri…

Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture. When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.

The setup you describe is very much not simple. I worked at a place with very good DBAs and our replication setup caused us more downtime than anything else. Cockroach and Spanner exist because many programmers observed that what you describe is hard.

Re: How we built a serverless SQL database

#44

Post by psadri »

phoboslab wrote: post #37

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…

Unfortunately, easy to hit that with say GraphQL where each client request can resolve to dozens of db selects vs a single hand written/tuned SQL select.

Re: How we built a serverless SQL database

#45

Post by bitwize »

phoboslab wrote: post #37

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…

(robot voice) Does /dev/null support sharding? Sharding is the secret ingredient in the web scale sauce.

Re: How we built a serverless SQL database

#46

Post by andydb »

babelfish wrote: post #31

Really great post, thanks for sharing. I spent a lot of time a couple months ago researching 'DBaaS' offerings (for fun, not business) and found it difficult to find any posts outlining the architecture of a DBaaS. Really cool to see CRDB putting this out in the open.

It's been something we've done since the start and plan to continue doing. If you read back over our engineering blog, you'll find a surprisingly thorough description of the entire CockroachDB stack, from the lowest storage layer to distributed transactions, Raft consensus, SQL => key-value mapping, online schema changes, cost-based SQL optimizer, Kubernetes usage, and so on. In fact, when we onboard new engineers, we typically point them to a stack of external blog entries to read in order to get up-to-speed on how CockroachDB works.

Being open on how we solve hard problems is the way to build our collective knowledge as a developer community. Certainly CockroachDB itself has benefited enormously from all that has gone before and been published in the open.

Re: How we built a serverless SQL database

#48

Post by Groxx »

mjibson wrote: post #43
phoboslab wrote: post #41

Earlier quoted context omitted.

Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture. When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.

The setup you describe is very much not simple. I worked at a place with very good DBAs and our replication setup caused us more downtime than anything else. Cockroach and Spanner exist because many programmers observed that what you describe is hard.

As a counter-anecdote: multiple startup projects I've worked on with separate MySQL setups where each had just a single master + two slaves (one warm for fast failover in case of hardware failure or upgrades, one cold for slow analytics-style queries) did just fine with millions (to tens of millions) of users. No downtime at all for years on end.

MySQL and Postgres are massively more widely-used than Cockroach and Spanner, broadly very successfully. It's entirely feasible to run it with high uptime.

Post reply on HN