Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

81–90 of 233 posts

Re: How Figma's databases team lived to tell the scale

#81

Post by gregors »

adastral wrote: post #31

I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.

Where's the fun in that? I'm not being snarky either. Maybe it's not the best decision business-wise, but I guarantee it was more challenging and more fun. There's something to be said for that.

Re: How Figma's databases team lived to tell the scale

#82

Post by jerbear4328 »

bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

Actually, Apple does this for iCloud! They use FoundationDB[1] to store billions of databases, one for each user (plus shared or global databases).

See: https://read.engineerscodex.com/p/how-apple-built-icloud-to-...

Discussed on HN at the time: https://news.ycombinator.com/item?id=39028672

[1]: https://github.com/apple/foundationdb https://en.wikipedia.org/wiki/FoundationDB

Re: How Figma's databases team lived to tell the scale

#83

Post by gen220 »

jabart wrote: post #52
adastral wrote: post #31

I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.

Figma uses AWS RDS, RDS doesn't list citus as a supported extension.

This is my guess of why they didn't use Citus. They weren't interested in the options of (1) going multi-cloud [DB in Azure Cosmos / Backend(s) in AWS] (2) going all-in on Azure [DB in Azure Cosmos / Backend(s) in Azure] (3) self-managing Postgres+Citus in EC2.

It'd be interesting to compare the expected capex of developing this in-house solution + the opex of maintaining it vs the same categories of expected costs for option (3) – because I imagine that's probably the most palatable option.

They also may have pre-paid for dedicated RDS instances for the next X years (before this horizontal scaling initiative began, to boot), as AWS allows companies to do this at a pretty steep discount rate, which would probably tilt them away from (3).

Re: How Figma's databases team lived to tell the scale

#84

Post by dwaltrip »

cynicalsecurity wrote: post #77

They came up with a really over-engineered, over-complicated attempt at splitting data into multiple databases. I'm not sure this was the best idea or even a good idea, but then I also don't understand how it came to the situation when they only had a few months before their current database gets overflown and the whole system collapses.

Can you offer insight into what a better approach might have been?

Re: How Figma's databases team lived to tell the scale

#85

Post by gen220 »

gshulegaard wrote: post #64
adastral wrote: post #31

I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.

I have worked on teams that have both sharded and partitioned PostgreSQL ourselves (somewhat like Figma) (Postgres 9.4-ish time frame) as well as those that have utilized Citus. I am a strong proponent of Citus and point colleagues in that direction frequently, but depending on how long ago Figma was considering this path I will say that there were some very interesting limitations to Citus not that long ago. For exa…

You also benefit from the tailwind of the CitusData team making continued improvement to the extension, whereas an in-house system depends on your company's ability to hire and retain people to maintain + improve the in-house system.

It's hard to account for the value of benefits that have yet to accrue, but this kind of analysis, even if you pretty heavily-discount that future value, tilts the ROI in favor of solutions like Citus, IMO. Especially if your time horizon is 5+ or 10+ years out.

Like you said, if they made this decision 3ish years ago, you would have had to be pretty trusting on that future value. A choice, made today, hinges less on that variable.

Re: How Figma's databases team lived to tell the scale

#86

Post by emseetech »

esafak wrote: post #6
Closi wrote: post #4

Earlier quoted context omitted.

This assumes that using a distributed database from the start doesn't offer it's own penalties/drawbacks (particularly in 2016). Particularly considering as per the figma note, they consider their data highly relational (i.e. presumably this means lots of joins in queries) What database would you have chosen?

That's a fair response for then, but not today where you're spoiled for choice. Spanner was released as a service in 2017, which isn't far off.

Spanner is quite expensive though.

Re: How Figma's databases team lived to tell the scale

#87

Post by hintymad »

Given that sharding has become a pretty mature practice, is it still worth considering the NewSQL solutions like CRDB, Yugabyte, and TiDB for the sake of auto sharding, given that these NewSQL databases usually trade throughput and latency for auto-sharding and multi-region support? Another added cost is learning how to operate NewSQL databases, assuming one is already familiar with either MySQL or Postgres.

Re: How Figma's databases team lived to tell the scale

#89

Post by TheP1000 »

dwaltrip wrote: post #84
cynicalsecurity wrote: post #77

They came up with a really over-engineered, over-complicated attempt at splitting data into multiple databases. I'm not sure this was the best idea or even a good idea, but then I also don't understand how it came to the situation when they only had a few months before their current database gets overflown and the whole system collapses.

Can you offer insight into what a better approach might have been?

As others have mentioned, moving to per tenant databases can really simplify things at scale and doesn't leave a massive amount of engineering complexity and debt in its wake.

I feel sorry for the team managing this 5 years from now.

Re: How Figma's databases team lived to tell the scale

#90

Post by suhastech »

ijustlovemath wrote: post #57

> NoSQL databases are another common scalable-by-default solution that companies adopt as they grow. However, we have a very complex relational data model built on top of our current Postgres architecture and NoSQL APIs don’t offer this kind of versatility. As I understand it, NoSQL is for people who need a backend that ingests just about any unstructured data, for teams that may not have a complex relational model w…

Using NoSQL might not be the best idea in this case. I've seen it backfire for many companies. They start with NoSQL, but then end up needing relational features as their business grows. This leads to performance issues, redundancy, and data sync problems early on, which shouldn't be happening.

Especially in the early days, NoSQL companies used to market their databases as general-purpose database that scale easily, but that hasn't always been the case obviously.

I usually recommend starting with a relational database like PostgreSQL. If scaling becomes necessary later on, you can invest in sharding the database. Figma's approach seems reasonable given the tools available at the time.

I've helped small companies switch from NoSQL to SQL because the benefits of NoSQL weren't worth the trade-offs at their stage of growth. In case, anyone is in a similar boat: https://mongotosqlmigration.com/

Post reply on HN