Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

111–120 of 233 posts

Re: How Figma's databases team lived to tell the scale

#111

Post by hacker_newz »

jakewins wrote: post #34
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

The problem - conceptually - is made much simpler this way; we make use of this at work. However you will still have shared resource problems - some rogue query destroys IOPS in one tenant now ends up bringing down all tenants etc. There are in theory databases that solve this as well, but my experience has been that at that point what you buy into is a bad version of resource sharing - ie what an operating system do…

If tenants are on separate databases how would that be an issue?

Re: How Figma's databases team lived to tell the scale

#112

Post by simonw »

Thaxll wrote: post #88

How transactions work when you end up querying different shards?

I imagine that's one of the reasons they have "colos" - you can aim to collocate tables that are likely to be part of the same transaction in the same shard by putting them in the same colo group.

Re: How Figma's databases team lived to tell the scale

#113

Post by endisneigh »

jerrygenser wrote: post #105
rockostrich wrote: post #54

Earlier quoted context omitted.

GCP products have a much better track record than Google consumer products when it comes to support since there are usually enterprise customers with multi-year contracts worth tens, if not hundreds, of millions of dollars using them.

IoT is one example of a big backbone service that was sunset.

It had barely any usage though from what I can tell from searching about it.

Not that it’s any solace to those affected.

Re: How Figma's databases team lived to tell the scale

#114

Post by dexwiz »

danpalmer wrote: post #95
jerbear4328 wrote: post #82

Earlier quoted context omitted.

Actually, Apple does this for iCloud! They use FoundationDB[1] to store billions of databases, one for each user (plus shared or global databases). See: https://read.engineerscodex.com/p/how-apple-built-icloud-to-... Discussed on HN at the time: https://news.ycombinator.com/item?id=39028672 [1]: https://github.com/apple/foundationdb https://en.wikipedia.org/wiki/FoundationDB

> store billions of databases This is sort of true and sort of false. When you think of a "database", if you're thinking of a Postgres database, you're way off the reality of what "database" means here. FoundationDB has a concept called "layers", and essentially they have created a layer that looks like a separate database on top of a layer that is separately encrypted groups of keys. They don't have billions of Foun…

This sounds like a pretty standard multitenant datastore. Everything has a user/group Id on it, and a logical layer that locks a connection to a specific group.

Re: How Figma's databases team lived to tell the scale

#115

Post by dhash »

bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

I believe physalia [0] explores this concept at production scale quite well.

[0] https://blog.acolyer.org/2020/03/04/millions-of-tiny-databas...

Re: How Figma's databases team lived to tell the scale

#116

Post by paxys »

bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

This works great until (1) your largest customer outgrows the largest available DB (happens sooner than you'd think for large companies) or (2) you do need transactions across different customers, say to facilitate some kind of sharing. Going all-in on the isolated tenant strategy means when you hit one of these cases it's a nightmare to unwind and rearchitect your entire DB layer.

Re: How Figma's databases team lived to tell the scale

#117

Post by sgarland »

gen220 wrote: post #83
jabart wrote: post #52

Earlier quoted context omitted.

Figma uses AWS RDS, RDS doesn't list citus as a supported extension.

This is my guess of why they didn't use Citus. They weren't interested in the options of (1) going multi-cloud [DB in Azure Cosmos / Backend(s) in AWS] (2) going all-in on Azure [DB in Azure Cosmos / Backend(s) in Azure] (3) self-managing Postgres+Citus in EC2. It'd be interesting to compare the expected capex of developing this in-house solution + the opex of maintaining it vs the same categories of expected costs f…

Especially because Option 3 lets you go waaaay farther on vertical scaling, since you can get native NVMe drives (they mentioned hitting IOPS limits for RDS), more exotic instance classes with far more RAM, and do stuff like ZFS for native compression and snapshots.

Re: How Figma's databases team lived to tell the scale

#118

Post by infra_dev »

simonw wrote: post #110
TheP1000 wrote: post #89

Earlier quoted context omitted.

As others have mentioned, moving to per tenant databases can really simplify things at scale and doesn't leave a massive amount of engineering complexity and debt in its wake. I feel sorry for the team managing this 5 years from now.

Moving to a per-tenant database sounds like even more work to me than moving to shards. Moving to per-tenant means rewriting _everything_ - moving to shards has you rewriting a lot less.

This is possible to do, but lots of engineering. You can provide the experience of a single DB while each tenant can be placed in their own dedicated Postgres compute. This would help the application to stay the same while tenants are moved to independent computes (you can even move only a few tenants and leave the rest on a shared Postgres compute).

Re: How Figma's databases team lived to tell the scale

#119

Post by jakewins »

hacker_newz wrote: post #111
jakewins wrote: post #34

Earlier quoted context omitted.

The problem - conceptually - is made much simpler this way; we make use of this at work. However you will still have shared resource problems - some rogue query destroys IOPS in one tenant now ends up bringing down all tenants etc. There are in theory databases that solve this as well, but my experience has been that at that point what you buy into is a bad version of resource sharing - ie what an operating system do…

If tenants are on separate databases how would that be an issue?

Separate logical databases, within the same RDBMs, so sharing CPU, disks, RAM etc

Re: How Figma's databases team lived to tell the scale

#120

Post by sp1nozick »

jeffbee wrote: post #67

Given the list of authors and acknowledgees, what I'd really like to read is the differences between this solution and Dropbox's.

horizontal sharding is such a foundational skillset in working with databases - it's essential to database architecture
Post reply on HN