Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

201–210 of 233 posts

Re: How Figma's databases team lived to tell the scale

#201

Post by aledalgrande »

themoonisachees wrote: post #155
jasonwatkinspdx wrote: post #128

Earlier quoted context omitted.

I think there's quite a few people chasing similar ideas, like Azure's Durable Entities. I've been calling it the Lots of Little Databases model vs the Globe Spanning Gorilla. Like the Spanner paper points out, even if your distributed database semantically appears like a single giant instance, in practice performance means developers avoid using distributed joins, etc, because these can lead to shuffling very large…

An advantage worth noting is that having actually separated databases means you physically can't make these expensive operations, so a junior dev can't write incredibly inefficient code that would bring down your entire infra.

Bring down the infra or foot you a 6 figures bill at the end of the month

Re: How Figma's databases team lived to tell the scale

#202

Post by asguy »

nightpool wrote: post #184
menthe wrote: post #160

Earlier quoted context omitted.

This, seriously. The long-term maintenance, tribal knowledge & risks associated with this giant hack will be greater than anything they'd ever have expected. Inb4 global outage post-mortem & key-man dependency salaries. There's no virtually no excuse not spinning up a pg pod (or two) for each tenant - heck even a namespace with the whole stack. Embed your 4-phases migrations directly in your releases / deployments, s…

Figma has millions of customers. The idea of having a Postgres pod for each one would be nearly impossible without completely overhauling their DB choice.

Thousands of directories over thousands of installations? It’s not that far fetched.

Re: How Figma's databases team lived to tell the scale

#203

Post by dijit »

VHRanger wrote: post #190
dijit wrote: post #39

Earlier quoted context omitted.

I've had CitusDB running across 68 bare metal machines (40 vCPU, 768GiB ram, 20TiB of storage each + 40GiB network links) and it ran decently well. Not sure what your definition of massive is, I think Spanner would easily beat it. Also, it's very use-case dependent, you can't "just use" Citus for everything, it's not quite as flexible as a bog-standard pgsql install due to the way it's sharding, you have to be a tad…

Is there a reason there's comparatively little storage in your machines in relation to RAM or even CPUs? Do your machines do compute heavy loads or something? For a DB I'd expect a lot more storage per node

NVMe SSDs aren't so large unfortunately.

a 1U server has capacity for 8 drives, we used 2 slots for the OS (RAID1), 2 slots for the WAL volume (2 slots) leaving only 4 slots in RAID10.

So I'm already cheating a little and claiming WAL storage was part of total storage.

Re: How Figma's databases team lived to tell the scale

#204

Post by menthe »

nightpool wrote: post #184
menthe wrote: post #160

Earlier quoted context omitted.

This, seriously. The long-term maintenance, tribal knowledge & risks associated with this giant hack will be greater than anything they'd ever have expected. Inb4 global outage post-mortem & key-man dependency salaries. There's no virtually no excuse not spinning up a pg pod (or two) for each tenant - heck even a namespace with the whole stack. Embed your 4-phases migrations directly in your releases / deployments, s…

Figma has millions of customers. The idea of having a Postgres pod for each one would be nearly impossible without completely overhauling their DB choice.

You are making a major conflation here. While they do millions of users, they were last reported to only have ~60k tenants.

Decently sized EKS nodes can easily hold nearly 800 pods each (as documented), that'd make it 75 nodes. Each EKS cluster supports up to 13,500 nodes. Spread in a couple of regions to improve your customer experience, you're looking at 20 EKS nodes per cluster. This is a nothingburger.

Besides, it's far from being rocket science to co-locate tenant schemas on medium-sized pg instances, monitor tenant growth, and re-balance schemas as necessary. Tenants' contracts does not evolve overnight, and certainly does not grow orders of magnitude on week over week basis - a company using Figma either has 10 seats, 100 seats, 1000, or 10,000 seats. It's easy to plan ahead for. And I would MUCH rather having to think of re-balancing a heavy hitter customer's schema to another instance every now and then (can be 100% automated too), compared to facing a business-wide SPOF, and having to hire L07+ DBAs to maintain a proprietary query parser / planner / router.

Hell, OVH does tenant-based deployments of Ceph clusters, with collocated/coscheduled SSD/HDD hardware and does hot-spot resolution. And running Ceph is significantly more demanding and admin+monitoring heavy.

Re: How Figma's databases team lived to tell the scale

#205

Post by ndriscoll »

bastawhiz wrote: post #175
jasonwatkinspdx wrote: post #138

Earlier quoted context omitted.

I'd respond by saying (1) is more rare than you're asserting. There is a huge long tail of companies with datasets that won't fit on a single machine but can be handled by a dozen or so, and where no single customer or service is near the limits of an individual node. These customers are poorly served at the moment. Most of the easy to implement SaaS options for what they need would be hugely costly vs a small fleet…

> that won't fit on a single machine It's rarely an issue with the number of bytes and more an issue with hot shards. Whatever shard Google is on (that is, Google the Figma customer) is surely going to be one _hell_ of a hot shard. They have more designers/engineers/PMs actively using Figma than most startups have _users total_. You don't need more than one really really hot customer for this to become more than a hy…

Google still only has ~180k employees, and obviously not all of them use figma, and obviously not all of their figma users are performing actions simultaneously. I'd be surprised if it broke 10k QPS (would an org like Google even have 10k peak active user sessions? Seems doubtful). Human generated traffic tends to just not reach that large of scales unless you're trying to fit the entire planet on one instance.

RDS can be absurdly limited with IOPS, granted, but a modern laptop for example ought to be up to the task. Realistically you could probably be fine even on RDS but you might need to pay through the nose for extra IOPS.

Re: How Figma's databases team lived to tell the scale

#206

Post by metadat »

It looks like figma mostly just implemented the Instagram sharding paper, and then did the obvious next level of semi-intelligent yet delicate and fragile query routing.

Definitely a technically challenging and overall fun / satisfying engineering exercise!

https://instagram-engineering.com/sharding-ids-at-instagram-...

Re: How Figma's databases team lived to tell the scale

#208

Post by kgeist »

simonw wrote: post #110
TheP1000 wrote: post #89

Earlier quoted context omitted.

As others have mentioned, moving to per tenant databases can really simplify things at scale and doesn't leave a massive amount of engineering complexity and debt in its wake. I feel sorry for the team managing this 5 years from now.

Moving to a per-tenant database sounds like even more work to me than moving to shards. Moving to per-tenant means rewriting _everything_ - moving to shards has you rewriting a lot less.

What do you mean by "rewriting everything"? Or maybe your definition of "per-tenant database" is different from mine. In our product, it's just a small layer which routes requests to the target organization's DB. When an organization is created, we create a new DB. Most of application code has no idea there are different DBs under the hood.

There are logical DBs (folders/files on a single physical server), and there's a few physical servers. We're currently at the stage where the first physical server hosts most of smaller organizations, and all other physical servers are usually dedicated servers for larger clients with high loads.

Re: How Figma's databases team lived to tell the scale

#210

Post by thesz »

martin-adams wrote: post #162
brigadier132 wrote: post #159

Earlier quoted context omitted.

Ok, what risk? Cockroachdb is already proven technology and costs marginally more (if you use their serverless setup, it's free until you hit real scale). At the startups I've been at that hit scale, scaling sql was always a massive undertaking and affected product development on every single time. If you don't want downtime, don't use databases that require downtime to do a migration? Netflix, roblox, every single o…

Sounds like their discomfort was in the migration path to 'any other database' alongside not having the experience with another database to mitigate any unknown unknowns. > During our evaluation, we explored CockroachDB, TiDB, Spanner, and Vitess. However, switching to any of these alternative databases would have required a complex data migration to ensure consistency and reliability across two different database st…

> ensure consistency and reliability across two different database stores.

This is main known known. And this is hard thing to attain.

My favorite story on that is testing of tendermint consensus implementation [1]. The testing process found a way to break the consensus and the reason was that protocol implementation and KV store controlled by protocol used different databases.

[1] https://jepsen.io/analyses/tendermint-0-10-2

Post reply on HN