I still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?
It's easier to manage 1 database instead of 1000s
Why Has Figma Reinvented the Wheel with PostgreSQL?
61–70 of 98 posts
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#62Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#63I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…
>That said, rolling your own sharding is a MASSIVE undertaking.
Yes. It may not fits your need but take a look at PlanetScale. ( Based on MySQL and Vitess but I have seen quite a few people moving from Postgres )
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#64I still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?
It's easier to manage 1 database instead of 1000s
It's more expensive to screw up one all-important database than one of a thousand.
Same logic allies to compute boxes, see "pets vs cattle" from 15-20 years ago.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#65Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#66Am I the only one here thinking they should have just used MongoDB and be done with it?
I know over simplified approach but majority of problem would be solved.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#67I still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?
> why not a dedicated database for every customer
Well there's trade-offs with this too, whether needing aggregate data across shards for features, reporting, etc. Shared data between customers, users, etc. API access, etc.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#68Earlier quoted context omitted.
We use Citus. Very similar performance properties to DIY sharding but much more polished. Currently at 7 TB, self hosted. Growing roughly at 100 % per year, write-heavy. Works fine for us.
Unfortunately this runs into their "have to move off AWS for managed service" point since the managed service for Citus is now on Azure post acquisition, as Azure Cosmos DB for PostgreSQL Pitching Citus ran into issues where people were hoping it would handle sharding transparently, which isn't the case. But for someone who's evaluating rolling their own sharding, being able to manage sharding keys explicitly is how…
[deleted]
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#69I’m definitely of the opinion that what Figma[0] (and earlier, Notion[1]) did is what I’d call “actual engineering”.
Both of these companies are very specific about their circumstances and requirements
- Time is ticking, and downtime is guaranteed if they don’t do anything
- They are not interested in giving up the massive amount feature AWS supports via RDS, very specially around data recovery (anyone involved with Business Continuity planning would understand the importance of this)
- They need to be able to do it safely, incrementally, and without closing the door on reverting their implementation/rollout
- The impact on Developers should be minimal
“Engineering” at its core is about navigating the solution space given your requirements, and they did it well!
Both Figma and Notion meticulously focused on the minimal feature set they needed, in the most important order, to prevent disastrous downtime (e.g Figma didn’t need to support all of SQL for sharding, just their most used subset).
Both companies call out (rightfully so) that they have extensive experience operating RDS at this point, and that existing solutions either didn’t give them the control they needed (Notion) or required a significant rewrite or their data structures (Figma), which was not acceptable.
I think many people also completely underestimate how important operational experience with your solution is at this scale. Switch to Citus/Vitess? You’ll now find out the hard way all the “gotchas” of running those solutions at scale, and it would guarantedly have resulted in significant downtime as they procured this knowledge.
They’d also have to spend a ton of time catching up to RDS features they were suddenly lacking, which I would wager would take much more time than the time it took implementing their solutions.
Great job to both teams!
[0] https://www.figma.com/blog/how-figmas-databases-team-lived-t...
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#70I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…
What I do not understand is they say "we explored CockroachDB, TiDB, Spanner, and Vitess". Those are not compatible with PostgreSQL beyond the protocol and migration would require massive rewrites and tests to get the same behavior. YugabyteDB is using PostgreSQL for the SQL processing, to provide same features and behavior and distributes with a Spanner-like architecture. I'm not saying that there's no risk and no e…
Maybe it’s just a matter of it being difficult to list all the things they didn’t use. The Figma article itself is a little more clear on their goals…
It’s not really just postgres compatibility they are after, but compatibility with the Amazon RDS version of postgres. They also wanted to have something they could adopt incrementally and back out of when something unanticipated goes wrong.
Also, I think yugabyte uses an older version of the postgres processing engine, which may or may not be a big deal, depending on what they are using.