Why Has Figma Reinvented the Wheel with PostgreSQL? • Hacker News Board

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#51

Post by harisund1990 » Sat, Mar 30, 2024, 2:35 PM UTC

mu53 wrote: post #35

robust-cactus wrote: post #7

Now seems a good time to point out, the wheel has literally been reinvented over and over again. The wheels of yesterday were terrible. Each version gets better. It's fine, reinvent away folks :)

Seriously, a naive database sharding algorithm could be implemented in a week or so by a competent dev. A company like figma (billions in revenue) putting a small team to implementing a database sharding solution for an un-implemented use case (RDS, not just postgres). AND open sourcing it creating a value for the community is a net-good for the industry.

Sharding is the easy part. Eventually you need to implement distributed transactions, taking a consistent backup across shards, PITR, resharding, load balancing, and the list goes on... That takes exponentially more number of people and time and mainly risk.

It works for Figma(for now), but for it to work as a solution for other companies with different hardware, data schema and access patterns will add even more complexity to the mix.

It's a excellent solution but I don't think it be good enough in the long run.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#52

Post by harisund1990 » Sat, Mar 30, 2024, 2:41 PM UTC

The article should be titled "Why Figma HAD TO reinvent the wheel with PostgresSQL". When you have a legacy system and not enough time, or will to move off of it the only option is to get inventive and build with what you have.

There is always a price. In this case the database team did something quick, cheap and easily. But the Application teams now have to deal with handling all the nuaces of the system. Maybe Figma has more people in these Apps teams with time on their hands to handle it.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#53

Post by evanelias » Sat, Mar 30, 2024, 2:46 PM UTC

Ozzie_osman wrote: post #15

I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…

> That said, rolling your own sharding is a MASSIVE undertaking.

It's a large challenge, but it's absolutely doable. A ton of companies did this 10-15 years ago, basically every successful social network, user generated content site, many e-commerce sites, massively multiplayer games, etc. Today's pre-baked solutions didn't exist then, so we all just rolled our own, typically on MySQL back then.

With DIY, the key thing is to sidestep any need for cross-shard joins. This is easier if you only use your relational DB for OLTP, and already have OLAP use-cases elsewhere.

Storing "association" type relation tables on 2 shards helps tremendously too: for example, if user A follows user B, you want to record this on both user A and user B's shards. This way you can do "list all IDs of everyone user A follows" as well as "list all IDs of users following user B" without crossing shard boundaries. Once you have the IDs, you have to do a multi-shard query to get the actual user rows, but that's a simple scatter-gather by ID and easy to parallelize.

Implementing shard splitting is hard, but again definitely doable. Or avoid it entirely by putting many smaller shards on each physical DB server -- then instead of splitting a big server, you can just move an entire shard to another server, which avoids any row-by-row operations.

Many other tricks like this. It's a lot of tribal knowledge scattered across database conference talks from a decade ago :)

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#54

Post by jitl » Sat, Mar 30, 2024, 2:48 PM UTC

iAkashPaul wrote: post #47

jitl wrote: post #36

Earlier quoted context omitted.

(I work at Notion) Our shard key - Workspace ID - is a UUIDv4 so there’s a pretty high number of orgs per shard without conflict.

Hey Jake, I meant capacity per shard in this case, not exhausting the IDs. Any potential solutions for that or is that not an immediate challenge?

Gotcha. With our shard strategy we add more capacity either by scaling up nodes (very easy), or by resharding - adding nodes to the cluster and putting fewer shards on each node.

We recently did a reshard from 32 nodes / 15 shards per node -> 96 nodes / 5 shards per node. That puts us in footing to scale up for a while before we need to reshard again. This is a pretty smooth process, and each time we scale out we get much more scale up capacity.

Our shard logic is very simple static assignment based only on the Workspace ID. If we wanted to add workspace created time routing, we'd need to starting plumbing that information around the system in ways that are slightly annoying. Probably the move would be to re-key the Workspace table to use a date-embedding UUID format.

https://www.notion.so/blog/the-great-re-shard

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#55

Post by jitl » Sat, Mar 30, 2024, 2:54 PM UTC

hobs wrote: post #49

jitl wrote: post #38

Earlier quoted context omitted.

We went through something similar at Notion a few years ago and also chose to stick with RDS Postgres and build sharding logic in our application’s database client. In both our case and Figma’s, sharding Postgres ASAP was of critical importance because of transaction ID wraparound threat or other capacity issues that promise hard days-long downtime. The kind of downtime that costs 10s of millions of dollars of brand…

Or you could just hire some set of people who know how to manage postgres? Seems easier than building an entirely new thing with its own set of bugs that are unknown unknown brand damage awaiting you.

It's not just manage Postgres, it's manage a Citus cluster - (unmanaged Postgres + postgres experts + time for them to implement their stuff) just gets us to parity with RDS but doesn't solve our sharding problem. We asked our Postgres consultants & networks to see if we could find Citus experts we could bring on full-time but didn't have great success. Most of the experts we talked to suggested application level sharding, and it seems like it worked out okay.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#56

Post by lopkeny12ko » Sat, Mar 30, 2024, 3:04 PM UTC

mbesto wrote: post #31

thih9 wrote: post #18

Earlier quoted context omitted.

The article suggests a different reason. What would be your approach if you wanted to stay on RDS? > So, now, let me speculate. The real reason why Figma reinvented the wheel by creating their own custom solution for sharding might be as straightforward as this — Figma wanted to stay on RDS, and since Amazon had decided not to support the CitusData extension in the past, the Figma team had no choice but to develop th…

Fair. But it doesn't really explain why they wanted to stay on RDS. This is their reasoning: > over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch. So they had in house expertise to run performantly on RDS but that same experience couldn't be translated to switching ov…

I was puzzled by this as well. RDS is a managed, cloud product. You don't run it. The whole point is that AWS runs it for you, no?

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#57

Post by __s » Sat, Mar 30, 2024, 3:06 PM UTC

krab wrote: post #43

Ozzie_osman wrote: post #15

I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…

We use Citus. Very similar performance properties to DIY sharding but much more polished. Currently at 7 TB, self hosted. Growing roughly at 100 % per year, write-heavy. Works fine for us.

Unfortunately this runs into their "have to move off AWS for managed service" point since the managed service for Citus is now on Azure post acquisition, as Azure Cosmos DB for PostgreSQL

Pitching Citus ran into issues where people were hoping it would handle sharding transparently, which isn't the case. But for someone who's evaluating rolling their own sharding, being able to manage sharding keys explicitly is how Citus allows efficient joins based on your workload. So yes, if you're looking to roll an unmanaged sharded postgresql cluster, consider starting with Citus

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#58

Post by [deleted] » Sat, Mar 30, 2024, 3:08 PM UTC

[deleted]

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#59

Post by giva » Sat, Mar 30, 2024, 3:12 PM UTC

I still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#60

Post by hfucvyv » Sat, Mar 30, 2024, 3:14 PM UTC

giva wrote: post #59

I still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?

Probably the same reason they used postgres instead of MySQL.