31–40 of 233 posts
I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?
I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?
However you will still have shared resource problems - some rogue query destroys IOPS in one tenant now ends up bringing down all tenants etc. There are in theory databases that solve this as well, but my experience has been that at that point what you buy into is a bad version of resource sharing - ie what an operating system does - and you’re better off using OS mechanisms
In other words: yes, but you still have noisy neighbours, and may be better off running lots of small fully separated DBMSes than a big logically separated one
"Sorry, you’ve stumbled upon a temporary technical issue. Please try refreshing this page in a moment." Have they? :) I am excited to read the article when it loads
I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
how "massive" is massive in your case?
Good article none the less! Always appreciate when companies like Figma document technical challenges.
I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
how "massive" is massive in your case?
Not sure what your definition of massive is, I think Spanner would easily beat it.
Also, it's very use-case dependent, you can't "just use" Citus for everything, it's not quite as flexible as a bog-standard pgsql install due to the way it's sharding, you have to be a tad more careful with your data model.
Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works. It seems like they've (hop…
Global consistency is expensive, both latency-wise and cost-wise. In reality most apps don't need global serializability across all objects. For instance, you probably don't need serializability across different tenants, organizations, workspaces, etc. Spanner provides serializability across all objects IIUC - so you pay for it whether you need it or not. The other side of something like Spanner is the quorum-based l…
https://muratbuffalo.blogspot.com/2022/08/strict-serializabi....