Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Has Figma Reinvented the Wheel with PostgreSQL?

medium.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#81

Post by jitl »

vbezhenar wrote: post #77
hobs wrote: post #49

Earlier quoted context omitted.

Or you could just hire some set of people who know how to manage postgres? Seems easier than building an entirely new thing with its own set of bugs that are unknown unknown brand damage awaiting you.

It seems that this day the art of configuring a database has been long lost. I also completely don't understand the issue. Just buy two huge behemoth servers, put your postgres there in a replicated mode and move on. It'll sustain huge load. Surely those companies can afford to hire one sysadmin.

You can’t necessarily play Cookie Clicker with database hardware scaling and have a good time. Query performance and upkeep processes often begin to degrade well before a table reaches the maximum hardware-bound size. We were using an instance with 96 cores and 350gb of RAM which seems over provisioned on paper and still hitting a variety of issues like stalled out Postgres auto-vacuum.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#82

Post by pas »

Tehnix wrote: post #69

I’m definitely of the opinion that what Figma[0] (and earlier, Notion[1]) did is what I’d call “actual engineering”. Both of these companies are very specific about their circumstances and requirements - Time is ticking, and downtime is guaranteed if they don’t do anything - They are not interested in giving up the massive amount feature AWS supports via RDS, very specially around data recovery (anyone involved with…

the right way to look at it - IMHO - is to interpret "lots of RDS experience" as complete lack of run-your-own postgreSQL experience. and given that it's not surprising that their cost-benefit math give them the answer of "invest into a custom middleware, instead of moving to running our postgreSQL plus some sharding thing on top"

of course it's not cheap, but probably they are deep into the AWS money pit anyway (so running Citus/whatever would be similar TCO)

and it's okay, AWS is expensive for a lot of bootstrapped startups with huge infra requirements for each additional user, but Figma and Notion are virtually on the exact opposite of that spectrum

also it shows that there's no trivial solution in this space, sharding SQL DBs is very hard in general, and the extant solutions have sharp edges

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#83

Post by giva »

jitl wrote: post #72
giva wrote: post #59

I still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?

Multi-tenant design is a huge win in terms of reducing developer toil and expense. Many customers will have a tiny amount of data. For those customers a dedicated database is huge amount of overhead. There may not be any single customer who it makes sense to allocate dedicated "hardware" for. Sure you have to deal with a one-time pain to shard your thingy, but you don't need to pay for tens-of-thousands of individual…

I don't mean one database server for each customer. I say one database for customer. Hundreds or thousands of customer can be on the same database server. When you need more resources, you add another server. If customer grows too much, you move it on another server.

There is a bit of overhead, but not huge by any means.

Being able to update schema one customer at a time is a huge plus in my view, as it gives you a lot of flexibility in rollout. You deploy a new version of the application on a new application server and move the customers on the new servers updating their schema one by one (automatically, obviously)

Backups are routinely automated anyway.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#84

Post by nine_k »

jitl wrote: post #73
nine_k wrote: post #64

Earlier quoted context omitted.

It's more expensive to screw up one all-important database than one of a thousand. Same logic allies to compute boxes, see "pets vs cattle" from 15-20 years ago.

The difference between "pets" and "cattle" are that pets have state and need to be taken care of, you can't recreate them from scratch trivially. Cattle are stateless and can be created and destroyed easily. The whole point of a database is to contain the state - as a pet - so the rest of your application can be stateless - as cattle. To really get cattle database systems, you need a self-managing cluster architectur…

This is fair.

But restoring a small DB from a fresh backup, if things go really wrong, is faster, and does not affect other customers.

I completely agree wrt having a hot spare / cluster with transparent failover and management.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#86

Post by giva »

gedy wrote: post #67
giva wrote: post #59

I still can't understand why they decided to use a single database for all their customers. If each customer needs access to its own data, why not a dedicated database for every customer?

> why not a dedicated database for every customer Well there's trade-offs with this too, whether needing aggregate data across shards for features, reporting, etc. Shared data between customers, users, etc. API access, etc.

Sure there are trade offs. If a significant part of the value of the app rely on data sharing or transactions between customers this is clearly unfeasible. But if the app mostly deals with a significant amount of private data of the customers that occasionally needs to be shared, I think using separate databases (not servers) is the safest option.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#87

Post by hobs »

jitl wrote: post #55
hobs wrote: post #49

Earlier quoted context omitted.

Or you could just hire some set of people who know how to manage postgres? Seems easier than building an entirely new thing with its own set of bugs that are unknown unknown brand damage awaiting you.

It's not just manage Postgres, it's manage a Citus cluster - (unmanaged Postgres + postgres experts + time for them to implement their stuff) just gets us to parity with RDS but doesn't solve our sharding problem. We asked our Postgres consultants & networks to see if we could find Citus experts we could bring on full-time but didn't have great success. Most of the experts we talked to suggested application level sha…

Absolutely, I am just saying that you are talking on now all the inconsistencies of a third party management system and building your solution on top of that; you don't get the infra savings and benefits of managing your own, you gain some velocity for now and as big name clients probably will be stable for a few years.

I had a problem just recently where I worked at a place that's using blue/green aws rds deployments with mysql replication, and binlogs cant be moved in that service.

This is something that is bog simple in a non-managed service, and as a result we can either manage app replication, re-sync data on each b/g upgrade, or do physical replication (slow). My point isn't that rds is bad, it's just that if you are already deciding to implement your own significant infrastructure on top of database it seems weird to me to not just have the knobs on the thing itself.

Though you could say the same is true of the storage, and tbqh most of the cloud storage is dogshit these days but we just deal with it.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#88

Post by Ozzie_osman »

evanelias wrote: post #53
Ozzie_osman wrote: post #15

I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…

> That said, rolling your own sharding is a MASSIVE undertaking. It's a large challenge, but it's absolutely doable. A ton of companies did this 10-15 years ago, basically every successful social network, user generated content site, many e-commerce sites, massively multiplayer games, etc. Today's pre-baked solutions didn't exist then, so we all just rolled our own, typically on MySQL back then. With DIY, the key thi…

It's definitely doable. I was at Google circa 2006, pre Spanner, with sharded MySQL. Ads ran on top of it. It was a pain.

And yes, there are many tricks like having more logical shards than physical ones, collocating tables by the same shard_id, etc. It's still difficult. You need tooling for everything from shard splitting (even if that is just loving a logical shard), to schema migrations, not to mention if you end up needing cross-shard transactions or cross-shard joins.

Generally, you'd need a team of very strong infrastructure engineers. Most companies don't have the resources for that. There are definitely some engineers out there that could whip this all together.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#89

Post by Ozzie_osman »

krab wrote: post #43
Ozzie_osman wrote: post #15

I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…

We use Citus. Very similar performance properties to DIY sharding but much more polished. Currently at 7 TB, self hosted. Growing roughly at 100 % per year, write-heavy. Works fine for us.

Curious why you needed to shard at 7TB? I can imagine for some workloads, especially if it's write-heavy, you might start hitting constraints around vacuuming and things like that? But 7TB should be manageable on a (somewhat large and beefy) single machine.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#90

Post by davitocan »

giva wrote: post #83
jitl wrote: post #72

Earlier quoted context omitted.

Multi-tenant design is a huge win in terms of reducing developer toil and expense. Many customers will have a tiny amount of data. For those customers a dedicated database is huge amount of overhead. There may not be any single customer who it makes sense to allocate dedicated "hardware" for. Sure you have to deal with a one-time pain to shard your thingy, but you don't need to pay for tens-of-thousands of individual…

I don't mean one database server for each customer. I say one database for customer. Hundreds or thousands of customer can be on the same database server. When you need more resources, you add another server. If customer grows too much, you move it on another server. There is a bit of overhead, but not huge by any means. Being able to update schema one customer at a time is a huge plus in my view, as it gives you a l…

I've done this before where we ran a schema per customer and it was fabulous. Once the customer was large enough we could justify allocating a separate DB for them. The application was written in such a way that it knew which data store to query based on the user.
Post reply on HN