Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

121–130 of 233 posts

Re: How Figma's databases team lived to tell the scale

#121

Post by ummonk »

bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.

Re: How Figma's databases team lived to tell the scale

#123

Post by emmanueloga_ »

hintymad wrote: post #87

Given that sharding has become a pretty mature practice, is it still worth considering the NewSQL solutions like CRDB, Yugabyte, and TiDB for the sake of auto sharding, given that these NewSQL databases usually trade throughput and latency for auto-sharding and multi-region support? Another added cost is learning how to operate NewSQL databases, assuming one is already familiar with either MySQL or Postgres.

In the OP they said they built their own sharding solution because it was risky to move to a different (newsql) solution and they already had sharding expertise with PG.

I think if starting from scratch it makes sense to look at these newsql DBs that are built to horizontally scale from the very beginning.

Re: How Figma's databases team lived to tell the scale

#124

Post by sgarland »

hacker_newz wrote: post #111
jakewins wrote: post #34

Earlier quoted context omitted.

The problem - conceptually - is made much simpler this way; we make use of this at work. However you will still have shared resource problems - some rogue query destroys IOPS in one tenant now ends up bringing down all tenants etc. There are in theory databases that solve this as well, but my experience has been that at that point what you buy into is a bad version of resource sharing - ie what an operating system do…

If tenants are on separate databases how would that be an issue?

Because database, in Postgres terms, doesn’t mean physical node. It’s more akin to a VM than anything else. The term for an installation of Postgres is database cluster, which can contain N databases, each of which can contain M schemas.

Re: How Figma's databases team lived to tell the scale

#125

Post by cavisne »

bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

Spanners interleaved tables seem like a similar solution, ie you interleave customer data so it all ends up on the same set of hosts for performance, while still having the ability to create transactions across customers.

Re: How Figma's databases team lived to tell the scale

#126

Post by winrid »

paxys wrote: post #116
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

This works great until (1) your largest customer outgrows the largest available DB (happens sooner than you'd think for large companies) or (2) you do need transactions across different customers, say to facilitate some kind of sharing. Going all-in on the isolated tenant strategy means when you hit one of these cases it's a nightmare to unwind and rearchitect your entire DB layer.

A figma customer won't exceed the requirements of an i3.metal...

Re: How Figma's databases team lived to tell the scale

#127

Post by yashap »

mannyv wrote: post #24

Hmm, wonder why they didn't try FoundationDB. Interesting that they had problems with vacuuming. I always thought that part of Postgres was the worst part. I vaguely remember that you needed twice the space to vacuum successfully, which hopefully has changed in later versions. An article about why vacuum is needed in pg (as compared to mysql/innodb). http://rhaas.blogspot.com/2011/02/mysql-vs-postgresql-part-2...

They want pretty full SQL support, so FoundationDB would be out. I’d think “buy” options for them would be more like Citus, Yugabyte, Cockroach or Spanner.

Re: How Figma's databases team lived to tell the scale

#128

Post by jasonwatkinspdx »

ummonk wrote: post #121
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.

I think there's quite a few people chasing similar ideas, like Azure's Durable Entities.

I've been calling it the Lots of Little Databases model vs the Globe Spanning Gorilla.

Like the Spanner paper points out, even if your distributed database semantically appears like a single giant instance, in practice performance means developers avoid using distributed joins, etc, because these can lead to shuffling very large amounts of intermediate results across the network. So the illusion of being on a single giant machine ends up leaking through the reality, and people end up writing workarounds for distributed joins like async materialization.

If we give up the single machine illusion we get a lot of simplification, at the cost of features devs were unlikely to use anyhow. I see having consistent distributed commit but without cross shard joins as a really interesting alternative.

And besides scalability I like the extra security rope of fine grained partitioning from the start.

I'll write a blog post along these lines if I get anything worthwhile done.

Re: How Figma's databases team lived to tell the scale

#129

Post by jamesfinlayson »

aeyes wrote: post #46
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

I have pondered about this for quite some time and came to the conclusion that it would make schema migrations more difficult to handle. I think Shopify is using an approach which is similar to what you are describing. The advantage is that you don't end up with hot shards because you can move around large customers independently. In practice there isn't a big difference, they just colocate several customers accordin…

I remember Postgres table spaces being used to separate customers at a previous job - I can't remember how migrations were handled (pretty sure they were applied per table space) but I don't think it was a problem (at our scale anyway).
Post reply on HN