161–170 of 233 posts
Given that sharding has become a pretty mature practice, is it still worth considering the NewSQL solutions like CRDB, Yugabyte, and TiDB for the sake of auto sharding, given that these NewSQL databases usually trade throughput and latency for auto-sharding and multi-region support? Another added cost is learning how to operate NewSQL databases, assuming one is already familiar with either MySQL or Postgres.
Earlier quoted context omitted.
Their rationale for this choice is covered in the article somewhat extensively near the top. > Additionally, over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch. Given our very aggressive growth rate, we had only months of runway remaining. De-risking an entirely new s…
Ok, what risk? Cockroachdb is already proven technology and costs marginally more (if you use their serverless setup, it's free until you hit real scale). At the startups I've been at that hit scale, scaling sql was always a massive undertaking and affected product development on every single time. If you don't want downtime, don't use databases that require downtime to do a migration? Netflix, roblox, every single o…
> During our evaluation, we explored CockroachDB, TiDB, Spanner, and Vitess. However, switching to any of these alternative databases would have required a complex data migration to ensure consistency and reliability across two different database stores.
Earlier quoted context omitted.
Their rationale for this choice is covered in the article somewhat extensively near the top. > Additionally, over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch. Given our very aggressive growth rate, we had only months of runway remaining. De-risking an entirely new s…
Ok, what risk? Cockroachdb is already proven technology and costs marginally more (if you use their serverless setup, it's free until you hit real scale). At the startups I've been at that hit scale, scaling sql was always a massive undertaking and affected product development on every single time. If you don't want downtime, don't use databases that require downtime to do a migration? Netflix, roblox, every single o…
Earlier quoted context omitted.
Ok, what risk? Cockroachdb is already proven technology and costs marginally more (if you use their serverless setup, it's free until you hit real scale). At the startups I've been at that hit scale, scaling sql was always a massive undertaking and affected product development on every single time. If you don't want downtime, don't use databases that require downtime to do a migration? Netflix, roblox, every single o…
Never used cockroach so pardon my ignorance, but are there no operational challenges with running/using them? Or are they the same challenges? And how compatible is it from an application developer perspective?
Earlier quoted context omitted.
Ok, what risk? Cockroachdb is already proven technology and costs marginally more (if you use their serverless setup, it's free until you hit real scale). At the startups I've been at that hit scale, scaling sql was always a massive undertaking and affected product development on every single time. If you don't want downtime, don't use databases that require downtime to do a migration? Netflix, roblox, every single o…
Never used cockroach so pardon my ignorance, but are there no operational challenges with running/using them? Or are they the same challenges? And how compatible is it from an application developer perspective?
I cannot help but think this all sounds pretty much like a hack (a clever one, though). We do not handle, let's say low-level I/O buffering/caching, by ourselves anymore, right? (at least the folks doing web development/saas). We rely instead of the OS APIs, and that's good. I think we are missing something similar but for db sharding. It seems to me that we are still missing some fundamental piece of technology/infr…
1. They mention that the largest tables ran into several TBs, and they would have soon topped the max IOPS supported by RDS. RDS for PostgreSQL peaks at 256,000 IOPS for a 64 TB volume. For a multi-AZ setup, this costs ~$70K/mo. 2. Let's assume the final outcome was a 5-way shard with each shard supporting ~50,000 IOPS and ~12 TB data. For a multi-AZ setup, this costs ~$100K/mo. 3. It took 9 months to shard their fir…
I imagine scaling out RDS instead mitigated a lot of those costs.
One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?
Yes, we've been doing that at my place basically since the start. Each tenant is a schema in postgres. Works perfectly fine on the one hand, i.e. your tables don't grow to 'infinity' just because you're adding more and more tenants. If there's a particular tenant that has lots of data, only that tenant's indexes and tables grow huge and become slower because of that particular reason etc. If a tenant leaves, you keep…
Certainly needs a bunch of tooling, but worked well. Some apps were stateless and could connect to any physical cluster. Some others were sticky and only connected to a subset.
Similar architecture in my current company as well and we serve nearly a thousand customer instances served across 4 physical clusters.
We do have some basic tools to provision new customers on the emptiest cluster, move customers from one cluster to another etc
Earlier quoted context omitted.
Never used cockroach so pardon my ignorance, but are there no operational challenges with running/using them? Or are they the same challenges? And how compatible is it from an application developer perspective?
CRDB is Postgres compliant so the wire protocol and SQL syntax is all Postgres. It should be a 1 to 1.