Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

181–190 of 233 posts

Re: How Figma's databases team lived to tell the scale

#181

Post by thanksgiving »

bastawhiz wrote: post #174
thekhatribharat wrote: post #146

1. They mention that the largest tables ran into several TBs, and they would have soon topped the max IOPS supported by RDS. RDS for PostgreSQL peaks at 256,000 IOPS for a 64 TB volume. For a multi-AZ setup, this costs ~$70K/mo. 2. Let's assume the final outcome was a 5-way shard with each shard supporting ~50,000 IOPS and ~12 TB data. For a multi-AZ setup, this costs ~$100K/mo. 3. It took 9 months to shard their fir…

> A PostgreSQL-compatible NewSQL like YugabyteDB should cost ~$15K/mo to match top-of-the-line RDS performance. "to match" is doing a lot of work here. It's extremely unwise that a "compatible" database will have the same performance characteristics and no performance cliffs. > So Figma spent ~25x ($400K/$15K) They nearly got acquired for $20B, I don't think they give a hoot about 400K if it means keeping their stack…

That’s the problem with case studies, isn’t it? What works for one company in one industry might be a death sentence for another organization with slimmer margins…

Re: How Figma's databases team lived to tell the scale

#182

Post by __float »

bastawhiz wrote: post #174
thekhatribharat wrote: post #146

1. They mention that the largest tables ran into several TBs, and they would have soon topped the max IOPS supported by RDS. RDS for PostgreSQL peaks at 256,000 IOPS for a 64 TB volume. For a multi-AZ setup, this costs ~$70K/mo. 2. Let's assume the final outcome was a 5-way shard with each shard supporting ~50,000 IOPS and ~12 TB data. For a multi-AZ setup, this costs ~$100K/mo. 3. It took 9 months to shard their fir…

> A PostgreSQL-compatible NewSQL like YugabyteDB should cost ~$15K/mo to match top-of-the-line RDS performance. "to match" is doing a lot of work here. It's extremely unwise that a "compatible" database will have the same performance characteristics and no performance cliffs. > So Figma spent ~25x ($400K/$15K) They nearly got acquired for $20B, I don't think they give a hoot about 400K if it means keeping their stack…

They're also using sticker pricing, which as we know is likely far from what they actually paid.

(And the rest of the argument sounds a lot like our former employer with their...choice of document DB :) which I'll defend too!)

Re: How Figma's databases team lived to tell the scale

#183

Post by MarkMarine »

kamikaz1k wrote: post #163
brigadier132 wrote: post #159

Earlier quoted context omitted.

Ok, what risk? Cockroachdb is already proven technology and costs marginally more (if you use their serverless setup, it's free until you hit real scale). At the startups I've been at that hit scale, scaling sql was always a massive undertaking and affected product development on every single time. If you don't want downtime, don't use databases that require downtime to do a migration? Netflix, roblox, every single o…

Never used cockroach so pardon my ignorance, but are there no operational challenges with running/using them? Or are they the same challenges? And how compatible is it from an application developer perspective?

There are TONS of operational issues running cockroach. At the last company I was at cockroach was probably over used as a magical way to run multiple DCs and keep things consistent without high developer overhead, but it was #1 source of large outages. So much so that we’d run a cockroach segmented out for a single microservice to limit the blast radius when it eventually failed.

That and its comically more expensive than Postgres, if you think IOPs are expensive wait till you see the service contract.

Re: How Figma's databases team lived to tell the scale

#184

Post by nightpool »

menthe wrote: post #160
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

This, seriously. The long-term maintenance, tribal knowledge & risks associated with this giant hack will be greater than anything they'd ever have expected. Inb4 global outage post-mortem & key-man dependency salaries. There's no virtually no excuse not spinning up a pg pod (or two) for each tenant - heck even a namespace with the whole stack. Embed your 4-phases migrations directly in your releases / deployments, s…

Figma has millions of customers. The idea of having a Postgres pod for each one would be nearly impossible without completely overhauling their DB choice.

Re: How Figma's databases team lived to tell the scale

#185

Post by nightpool »

tharkun__ wrote: post #172
Syntaf wrote: post #153

Earlier quoted context omitted.

I recall a popular rails gem[1] once upon a time that provided multi-tenancy via postgres schemas. As it turns out, even the company the initially developed the gem ended up ditching the approach due to some of the issues you outlined above. Managing separate schemas feels like one of those nefarious decisions that make things simple _initially_ but get you into a world of hurt when you need to scale. The company is…

Let's address these one by one based on our experience (the part of the journey that I've been there at least as the implementation of the solution predates me but I live with it). Migrations Things slow down past 100 [...] No one wants friction in their deployment process, especially as we’re attempting to deploy daily or more frequently. We have more than a thousand schemas per database shard. That is why I said yo…

That's still 2 orders of magnitude smaller than the scale of Figma—they would need to somehow manage millions of Postgres schemas. I don't think it's a realistic possibility

Re: How Figma's databases team lived to tell the scale

#186

Post by hinkley »

jimbokun wrote: post #170
jamra wrote: post #165

Earlier quoted context omitted.

CRDB is Postgres compliant so the wire protocol and SQL syntax is all Postgres. It should be a 1 to 1.

Are all the corresponding latencies for every query one to one too?

In ye olden times I used to stop bosses from throwing away the slowest machine we had, and try to get at least one faster machine.

It’s still somewhat the case, but at the time the world was rotten with concurrent code that only worked because an implicit invariant (almost) always held. One that was enforced by the relative time or latency involved with two competing tasks. Get new motherboards or storage or memory and that invariant goes from failing only when the exact right packet loss happens, to failing every day, or hour, or minute.

Yes, it’s a bug, but it wasn’t on your radar and the system was trucking along yesterday and now everything is on fire.

The people who know this think the parent is a very interesting question. The people who don’t, tend to think it’s a non sequitur.

Re: How Figma's databases team lived to tell the scale

#187

Post by [deleted] »

thekhatribharat wrote: post #146

1. They mention that the largest tables ran into several TBs, and they would have soon topped the max IOPS supported by RDS. RDS for PostgreSQL peaks at 256,000 IOPS for a 64 TB volume. For a multi-AZ setup, this costs ~$70K/mo. 2. Let's assume the final outcome was a 5-way shard with each shard supporting ~50,000 IOPS and ~12 TB data. For a multi-AZ setup, this costs ~$100K/mo. 3. It took 9 months to shard their fir…

[deleted]

Re: How Figma's databases team lived to tell the scale

#188

Post by [deleted] »

dexwiz wrote: post #114
danpalmer wrote: post #95

Earlier quoted context omitted.

> store billions of databases This is sort of true and sort of false. When you think of a "database", if you're thinking of a Postgres database, you're way off the reality of what "database" means here. FoundationDB has a concept called "layers", and essentially they have created a layer that looks like a separate database on top of a layer that is separately encrypted groups of keys. They don't have billions of Foun…

This sounds like a pretty standard multitenant datastore. Everything has a user/group Id on it, and a logical layer that locks a connection to a specific group.

[deleted]

Re: How Figma's databases team lived to tell the scale

#189

Post by pests »

dexwiz wrote: post #114
danpalmer wrote: post #95

Earlier quoted context omitted.

> store billions of databases This is sort of true and sort of false. When you think of a "database", if you're thinking of a Postgres database, you're way off the reality of what "database" means here. FoundationDB has a concept called "layers", and essentially they have created a layer that looks like a separate database on top of a layer that is separately encrypted groups of keys. They don't have billions of Foun…

This sounds like a pretty standard multitenant datastore. Everything has a user/group Id on it, and a logical layer that locks a connection to a specific group.

FoundationDB is really just a set of structures and tools to build any type of database you want on top of a solid foundation.

"FoundationDB decouples its data storage technology from its data model. FoundationDB’s core ordered key-value storage technology can be efficiently adapted and remapped to a broad array of rich data models. Using indexing as an example, FoundationDB’s core provides no indexing and never will. Instead, a layer provides indexing by storing two kinds of key-values, one for the data and one for the index."

https://apple.github.io/foundationdb/layer-concept.html

Then existing standard layers like the Record layer, providing "(very) roughly equivalent to a simple relational database" providing structured types, index, complex types, queries, etc.

https://github.com/FoundationDB/fdb-record-layer

Or one for documents, which speaks the MongoDB wire protocol

https://github.com/FoundationDB/fdb-document-layer

Re: How Figma's databases team lived to tell the scale

#190

Post by VHRanger »

dijit wrote: post #39
_boffin_ wrote: post #33

Earlier quoted context omitted.

how "massive" is massive in your case?

I've had CitusDB running across 68 bare metal machines (40 vCPU, 768GiB ram, 20TiB of storage each + 40GiB network links) and it ran decently well. Not sure what your definition of massive is, I think Spanner would easily beat it. Also, it's very use-case dependent, you can't "just use" Citus for everything, it's not quite as flexible as a bog-standard pgsql install due to the way it's sharding, you have to be a tad…

Is there a reason there's comparatively little storage in your machines in relation to RAM or even CPUs?

Do your machines do compute heavy loads or something?

For a DB I'd expect a lot more storage per node

Post reply on HN