How we built a serverless SQL database
cockroachlabs.com
How we built a serverless SQL database
1–10 of 122 posts
Re: How we built a serverless SQL database
#2To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.
Re: How we built a serverless SQL database
#3To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.
What's your elevator pitch for why my organization should use CockroachDB Serverless vs. something like AWS Aurora Serverless, particularly if we're already relatively invested in the AWS ecosystem?
Re: How we built a serverless SQL database
#4To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.
Haven’t yet got time to read the whole thing so sorry if it’s already answered but is it possible to run this sql pod/storage pod separation setup yourself with crdb community/enterprise? We run enterprise crdb but it’s all in one process (with replicas)
Re: How we built a serverless SQL database
#5Does CockroachDB Serverless expose an HTTP api? This sounds like a great fit for use with Cloudflare Workers, but that requires an http api.
Re: How we built a serverless SQL database
#6To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.
I'm quite confused by Request Units, and trying to predict how many would be used by queries/operations.
I launched a test cluster, and the RUs are continuously increasing without me having connected to the cluster yet. At this rate of RU climb, the cluster would use over 8mil of the available 10mil RUs in a month without me touching it.
Coming from AWS, one of the most difficult aspects of using Aurora is guessing how many I/Os will be used for different workloads. It would be a shame to introduce this complexity for CockroachDB Serverless, especially if the RUs are impacted by internal cluster operations that aren't initiated by the user.
Re: How we built a serverless SQL database
#7This sounds great! I’ve wanted to create an open data portal for a while that lets you spin up a (ephemeral, read-only) Postgres database of a dataset and run queries on it, maybe with a notebook. Sounds like this might be perfect!
Re: How we built a serverless SQL database
#8Does CockroachDB Serverless expose an HTTP api? This sounds like a great fit for use with Cloudflare Workers, but that requires an http api.
Great question. We recognize how important this is and are actively working on it.
Re: How we built a serverless SQL database
#9To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.
I'm quite confused by Request Units, and trying to predict how many would be used by queries/operations. I launched a test cluster, and the RUs are continuously increasing without me having connected to the cluster yet. At this rate of RU climb, the cluster would use over 8mil of the available 10mil RUs in a month without me touching it. Coming from AWS, one of the most difficult aspects of using Aurora is guessing h…
I haven't connected to my cluster but my RUs keep going up. Extrapolating I'll be at 20M RUs over 30 days without using it.
Re: How we built a serverless SQL database
#10To any who might see this, I'm the author of the blog post, and led the engineering team that built CockroachDB Serverless. I'll be monitoring this thread in case there are any questions you'd like to ask me about it.
I'm quite confused by Request Units, and trying to predict how many would be used by queries/operations. I launched a test cluster, and the RUs are continuously increasing without me having connected to the cluster yet. At this rate of RU climb, the cluster would use over 8mil of the available 10mil RUs in a month without me touching it. Coming from AWS, one of the most difficult aspects of using Aurora is guessing h…
You've run into a "rough edge" of the beta release that will be fixed soon. When you keep the Cluster Overview page open, it runs queries against your cluster so that it can display information like "# databases in the cluster". Unfortunately, those queries run every 10 seconds in the background, and are consuming RUs, which is why you see RU usage without having connected to the cluster yet. But never fear, we'll get that fixed.
One thing that may not be clear - you get 10M RUs for free, up front, but you also get a constant accumulation of 100 RU/s for free throughout the month. That adds up to >250M free RUs per month. This ensures that your cluster is always accessible, and that you never truly "run out" of RUs - at most you get throttled to 100 RU/s.
I hear you on the difficulty of understanding how your queries map to RUs. SQL queries can be enormously complex and differ by multiple orders of magnitude from one another in terms of their compute cost. That's why we built a near real-time dashboard that shows you how quickly you're consuming RUs. You can run your workload for a few minutes and then check back on the dashboard to see how many RUs that workload consumed.