Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Has Figma Reinvented the Wheel with PostgreSQL?

medium.com

Post reply on HN
Print view View original

31–40 of 98 posts

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#31

Post by mbesto »

thih9 wrote: post #18
seanhunter wrote: post #4

The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many…

The article suggests a different reason. What would be your approach if you wanted to stay on RDS? > So, now, let me speculate. The real reason why Figma reinvented the wheel by creating their own custom solution for sharding might be as straightforward as this — Figma wanted to stay on RDS, and since Amazon had decided not to support the CitusData extension in the past, the Figma team had no choice but to develop th…

Fair. But it doesn't really explain why they wanted to stay on RDS. This is their reasoning:

> over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch.

So they had in house expertise to run performantly on RDS but that same experience couldn't be translated to switching over to it running on EC2 + Citus? Rather they used another non-experience concept of building their own sharding? That left me scratching my head.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#33

Post by seanhunter »

thih9 wrote: post #18
seanhunter wrote: post #4

The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many…

The article suggests a different reason. What would be your approach if you wanted to stay on RDS? > So, now, let me speculate. The real reason why Figma reinvented the wheel by creating their own custom solution for sharding might be as straightforward as this — Figma wanted to stay on RDS, and since Amazon had decided not to support the CitusData extension in the past, the Figma team had no choice but to develop th…

Wanting to stay on RDS is a reason doesn't survive the sort of extra scrutiny that I said should be applied in situations where you're doing a lot of work towards an internal goal. It also says in the article that they thought it was too risky to migrate (but somehow building their own sharding solution is going to be less risky for some reason).

I could of course be wrong but it really just feels to me like the reasons given in the article are attempts to justify a decision that was actually made because of "not invented here" syndrome.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#35

Post by mu53 »

robust-cactus wrote: post #7

Now seems a good time to point out, the wheel has literally been reinvented over and over again. The wheels of yesterday were terrible. Each version gets better. It's fine, reinvent away folks :)

Seriously, a naive database sharding algorithm could be implemented in a week or so by a competent dev.

A company like figma (billions in revenue) putting a small team to implementing a database sharding solution for an un-implemented use case (RDS, not just postgres). AND open sourcing it creating a value for the community is a net-good for the industry.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#36

Post by jitl »

iAkashPaul wrote: post #26

Even notion has a similar approach to sharding postgres but both of them could benefit from having shard IDs prefixed with YY/MM/DD(as needed) otherwise it's back to the shard navigator once they max out against org-ids for each shard's capacity

(I work at Notion)

Our shard key - Workspace ID - is a UUIDv4 so there’s a pretty high number of orgs per shard without conflict.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#37

Post by halfcat »

jerrygenser wrote: post #29
mkesper wrote: post #28

Aren't there any good managed postgres solutions supporting citus? The decision here seems to have been to invent a whole new sharding solution instead of building enough in house DBA to self-host postgres (if you want to stay on Amazon, you can use any extension you want on EC2). Speaks for the state of engineering right now.

Citus was bought by Microsoft so now it's only offered as Azure managed service.

Yes, the confusingly named “Azure Cosmos DB for PostgreSQL”

https://learn.microsoft.com/en-us/azure/cosmos-db/postgresql...

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#38

Post by jitl »

seanhunter wrote: post #4

The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many…

We went through something similar at Notion a few years ago and also chose to stick with RDS Postgres and build sharding logic in our application’s database client.

In both our case and Figma’s, sharding Postgres ASAP was of critical importance because of transaction ID wraparound threat or other capacity issues that promise hard days-long downtime. The kind of downtime that costs 10s of millions of dollars of brand damage alone. Possibly even company ending.

In such a situation, failure is not an option, and you must pick the least risky solution. Moving to an unmanaged cluster system and figuring out your own point-in-time backup/restore, access control provisioning, etc etc has a lot more unknown unknowns than sticking with the managed database vendor you know. The potential failure scenarios of Citus have scary worst cases - we get backup and restore wrong but it seems to work fine in test, then we move to Citus, then something breaks and we can’t restore from backup after all. It’s equally bad to mis-estimate the amount of time needed to bring up the new system. Let’s say you estimate 6 months to get parity with RDS built in features needed to survive disaster and start moving data over, but instead it takes 10 months. Is there enough time left to finish before going hard down? The clock is ticking. Staying with RDS keeps a whole class of new risk out of the picture.

At least here at Notion, NO ONE wanted to build something complicated for fun. We really wanted the company we’d spent years working for and on-call for to survive.

Our story: https://www.notion.so/blog/sharding-postgres-at-notion

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#40

Post by eivanov89 »

franckpachot wrote: post #30
Ozzie_osman wrote: post #15

I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…

What I do not understand is they say "we explored CockroachDB, TiDB, Spanner, and Vitess". Those are not compatible with PostgreSQL beyond the protocol and migration would require massive rewrites and tests to get the same behavior. YugabyteDB is using PostgreSQL for the SQL processing, to provide same features and behavior and distributes with a Spanner-like architecture. I'm not saying that there's no risk and no e…

I think Denis addressed this in his post: "Overall, as an engineer, you will never regret taking part in the development of a sharding solution. It’s a complex engineering problem with many non-trivial tasks to solve". In other words, it might be not invented here syndrome (with all due respect to Figma team). Or there might be more nuances we are unaware about.
Post reply on HN