Really great post, thanks for sharing. I spent a lot of time a couple months ago researching 'DBaaS' offerings (for fun, not business) and found it difficult to find any posts outlining the architecture of a DBaaS. Really cool to see CRDB putting this out in the open.
How we built a serverless SQL database
31–40 of 122 posts
Re: How we built a serverless SQL database
#32Earlier quoted context omitted.
Why do you prefer an HTTP API versus a DB connection? Isn't the former going to inherently have the overhead cost of creating the connection + TLS handshakes? My question is similar, which is, is CockroachDB going to have an equivalent RDS proxy so that apps can handle traffic spikes and not have to deal with problems with DB connection pools
I think we won't be getting socket connections in some of the wasm powered JS runtime engines soon. Using http solves that, and a bunch of caching issues.
there are some great projects that present an http api for postgres that you could use with CRDB I think. I'm thinking of something like https://github.com/pramsey/pgsql-http
Re: How we built a serverless SQL database
#33Earlier quoted context omitted.
I think we won't be getting socket connections in some of the wasm powered JS runtime engines soon. Using http solves that, and a bunch of caching issues.
there are some great projects that present an http api for postgres that you could use with CRDB I think. I'm thinking of something like https://github.com/pramsey/pgsql-http
Now it would be awesome if we could use Prisma against an open standard like this.
Re: How we built a serverless SQL database
#34One of the first apps to migrate from CockroachDB Dedicated to Serverless is live now if someone wants to try it out. https://web.flightchop.com/dashboard - posting for a friend.
Re: How we built a serverless SQL database
#35Earlier quoted context omitted.
It's not currently possible, partly because it complicates the deployment model quite a bit. Dynamically bringing SQL pods up and down requires a technology like Kubernetes. It takes some serious operational know-how to keep it running smoothly, which is why we thought it would be perfect for a managed Cloud service. What would be your company's reasons for wanting this available in self-hosted CRDB? What kinds of us…
It's understandly not something a lot of vendors consider, especially early stage products, but classified systems don't have Internet access and can't use managed cloud services (unless offered by AWS via C2S, and possibly in the future by whoever wins JEDI if they resurrect that), leaving no choice but to self-host anything. That's the extremest example, but lots of enterprises airgap part of their network for othe…
In addition to security/legal stuff there are also concerns around slo/business continuity and costs
Re: How we built a serverless SQL database
#36Earlier quoted context omitted.
What's your elevator pitch for why my organization should use CockroachDB Serverless vs. something like AWS Aurora Serverless, particularly if we're already relatively invested in the AWS ecosystem?
Oh boy, I'm an engineer, but I'll do my best to pretend I'm on the sales or marketing team for a minute... First of all, CockroachDB Serverless is available on AWS, and should integrate quite well with that ecosystem, including with Serverless functions offered by AWS Lambda. Here are a few advantages of CockroachDB Serverless that Aurora will struggle to match (note that we're still working on Serverless multi-regio…
You will do great with Sales role as well. Thanks for this write up!
Re: How we built a serverless SQL database
#37> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic.
The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres.
I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even funny anymore.
Re: How we built a serverless SQL database
#38> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…
Alternatively, people really do need to put a lot of thought into scaling, but only because they did something like write some core web service in an interpreted language framework that maxes out at 100 requests per second.
Re: How we built a serverless SQL database
#39Earlier quoted context omitted.
Oh boy, I'm an engineer, but I'll do my best to pretend I'm on the sales or marketing team for a minute... First of all, CockroachDB Serverless is available on AWS, and should integrate quite well with that ecosystem, including with Serverless functions offered by AWS Lambda. Here are a few advantages of CockroachDB Serverless that Aurora will struggle to match (note that we're still working on Serverless multi-regio…
I'm sold. What's your migration story from decades old MySQL Database over to CockroachDB? I'm only half joking there.
not sure if this is working yet for Serverless but... https://www.cockroachlabs.com/docs/v21.1/migrate-from-mysql....
Re: How we built a serverless SQL database
#40> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even…
What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability.
What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distribution isn't only about scale, it's also about durability.
What happens when you need to run an expensive business process ad-hoc? Do you want it to be easy to scale out reads, or to export that data to an analytics system? Or are you OK building something else to handle that case? Distribution isn't only about scale, it's also about flexibility.
What happens when you want to serve customers in one market, and make sure that their data stays local for regulatory compliance reasons or latency? Are you OK with having separate databases? Distribution isn't only about scale, it's also about locality.