Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

211–220 of 233 posts

Re: How Figma's databases team lived to tell the scale

#211

Post by fauigerzigerk »

ummonk wrote: post #121
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.

That would be fantastic. Unfortunately it's not true. D1 doesn't support the one database per customer approach unless you have just a handful of customers that you can set up manually.

You have to create each database manually using wrangler or the website. Then you have to create a binding for each database in wrangler.toml so that the database becomes accessible as a variable in your Workers code. Then you have to change your Worker source code to do something with that variable. Then you redeploy.

The issue is that Workers cannot create or list databases. There's no API for it.

Re: How Figma's databases team lived to tell the scale

#212

Post by joegibbs »

zarzavat wrote: post #143
paxys wrote: post #116

Earlier quoted context omitted.

This works great until (1) your largest customer outgrows the largest available DB (happens sooner than you'd think for large companies) or (2) you do need transactions across different customers, say to facilitate some kind of sharing. Going all-in on the isolated tenant strategy means when you hit one of these cases it's a nightmare to unwind and rearchitect your entire DB layer.

Figma is more or less a desktop application that happens to run in a web browser. If I use Photoshop to edit a .psd file I don’t think “man that psd file should really be stored in a single planet-sized database of all psd files in existence”. It’s just a file on my computer. Figma requires a little bit more intermingling of data than Photoshop, it has multiuser support for one, so a pure local storage based approach…

Agreed - it's not like you'd ever have to do SELECT * FROM Shapes WHERE 'type' = 'circle'. Could they have stored each document as a file, stored references to the files in the database, opened the file in the backend when someone opened it in the frontend, and written it back when they've stopped editing it?

Re: How Figma's databases team lived to tell the scale

#213

Post by adbachman »

jimbokun wrote: post #170
jamra wrote: post #165

Earlier quoted context omitted.

CRDB is Postgres compliant so the wire protocol and SQL syntax is all Postgres. It should be a 1 to 1.

Are all the corresponding latencies for every query one to one too?

This is the important question.

We evaluated several horizontally scalable DBs and Cockroach was by far the slowest for our access patterns.

Re: How Figma's databases team lived to tell the scale

#214

Post by VHRanger »

dijit wrote: post #203
VHRanger wrote: post #190

Earlier quoted context omitted.

Is there a reason there's comparatively little storage in your machines in relation to RAM or even CPUs? Do your machines do compute heavy loads or something? For a DB I'd expect a lot more storage per node

NVMe SSDs aren't so large unfortunately. a 1U server has capacity for 8 drives, we used 2 slots for the OS (RAID1), 2 slots for the WAL volume (2 slots) leaving only 4 slots in RAID10. So I'm already cheating a little and claiming WAL storage was part of total storage.

Shouldn't you try to get something with PCIe bifurcation in this case?

I doubt you're saturating the PCIe bus bandwidth on any of them?

I imagine your DB is extremely high performance, though!

Re: How Figma's databases team lived to tell the scale

#215

Post by nightpool »

menthe wrote: post #204
nightpool wrote: post #184

Earlier quoted context omitted.

Figma has millions of customers. The idea of having a Postgres pod for each one would be nearly impossible without completely overhauling their DB choice.

You are making a major conflation here. While they do millions of users, they were last reported to only have ~60k tenants. Decently sized EKS nodes can easily hold nearly 800 pods each (as documented), that'd make it 75 nodes. Each EKS cluster supports up to 13,500 nodes. Spread in a couple of regions to improve your customer experience, you're looking at 20 EKS nodes per cluster. This is a nothingburger. Besides, i…

Reported where? Does that include a monolithic "free tenant" that would be larger than thousands of their other tenants put together? Every Figma user has their own personal workspace along with the workspace of any organizations they may be invited to

Re: How Figma's databases team lived to tell the scale

#216

Post by lenkite »

"Schema changes must be coordinated across all shards to ensure the databases stay in sync. Foreign keys and globally unique indexes can no longer be enforced by Postgres."

One of my best practices for RDBMS data after executing a few sharding projects is to mandate no use of foreign keys even for v1 of a project.

Re: How Figma's databases team lived to tell the scale

#217

Post by tharkun__ »

nightpool wrote: post #185
tharkun__ wrote: post #172

Earlier quoted context omitted.

Let's address these one by one based on our experience (the part of the journey that I've been there at least as the implementation of the solution predates me but I live with it). Migrations Things slow down past 100 [...] No one wants friction in their deployment process, especially as we’re attempting to deploy daily or more frequently. We have more than a thousand schemas per database shard. That is why I said yo…

That's still 2 orders of magnitude smaller than the scale of Figma—they would need to somehow manage millions of Postgres schemas. I don't think it's a realistic possibility

https://news.ycombinator.com/item?id=39711815 Shopify seems to have millions of customers. Granted, they have MySQL and it might not use database schemas and something more MySQL specific. In general though ...

Re: How Figma's databases team lived to tell the scale

#218

Post by ummonk »

fauigerzigerk wrote: post #211
ummonk wrote: post #121

Earlier quoted context omitted.

This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.

That would be fantastic. Unfortunately it's not true. D1 doesn't support the one database per customer approach unless you have just a handful of customers that you can set up manually. You have to create each database manually using wrangler or the website. Then you have to create a binding for each database in wrangler.toml so that the database becomes accessible as a variable in your Workers code. Then you have to…

They have said they're working on some sort of solution (which they intend to have available by the time D1 exists beta) that will allow some sort of dynamic bindings specifically to address this use case.

In the meantime, though it's not a great solution, one can create and query databases through the REST API instead https://developers.cloudflare.com/api/operations/cloudflare-...

Re: How Figma's databases team lived to tell the scale

#219

Post by nightpool »

tharkun__ wrote: post #217
nightpool wrote: post #185

Earlier quoted context omitted.

That's still 2 orders of magnitude smaller than the scale of Figma—they would need to somehow manage millions of Postgres schemas. I don't think it's a realistic possibility

https://news.ycombinator.com/item?id=39711815 Shopify seems to have millions of customers. Granted, they have MySQL and it might not use database schemas and something more MySQL specific. In general though ...

"We can attach a shop_id to all shop-owned tables and have it serve as a sharding key. Moving a shop from one shard to another involves selecting all records from all tables that have the desired shop_id and copying them to another MySQL shard. For this post, it’s helpful to think of each pod as a MySQL shard."

Sounds like this is just normal sharding. Completely different from having a totally separate schema like GP proposes (with that approach you don't use sharding keys at all).

Re: How Figma's databases team lived to tell the scale

#220

Post by fauigerzigerk »

ummonk wrote: post #218
fauigerzigerk wrote: post #211

Earlier quoted context omitted.

That would be fantastic. Unfortunately it's not true. D1 doesn't support the one database per customer approach unless you have just a handful of customers that you can set up manually. You have to create each database manually using wrangler or the website. Then you have to create a binding for each database in wrangler.toml so that the database becomes accessible as a variable in your Workers code. Then you have to…

They have said they're working on some sort of solution (which they intend to have available by the time D1 exists beta) that will allow some sort of dynamic bindings specifically to address this use case. In the meantime, though it's not a great solution, one can create and query databases through the REST API instead https://developers.cloudflare.com/api/operations/cloudflare-...

>They have said they're working on some sort of solution

That's great to hear, thanks!

Post reply on HN