Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

221–230 of 233 posts

Re: How Figma's databases team lived to tell the scale

#221

Post by simonw »

kgeist wrote: post #208
simonw wrote: post #110

Earlier quoted context omitted.

Moving to a per-tenant database sounds like even more work to me than moving to shards. Moving to per-tenant means rewriting _everything_ - moving to shards has you rewriting a lot less.

What do you mean by "rewriting everything"? Or maybe your definition of "per-tenant database" is different from mine. In our product, it's just a small layer which routes requests to the target organization's DB. When an organization is created, we create a new DB. Most of application code has no idea there are different DBs under the hood. There are logical DBs (folders/files on a single physical server), and there'…

If you didn't write your application with multi-tenant in mind from the start I would expect you would need to review almost every line of code that touches a database to make a transition like this.

Re: How Figma's databases team lived to tell the scale

#222

Post by gen220 »

wayne-li2 wrote: post #102
gen220 wrote: post #85

Earlier quoted context omitted.

You also benefit from the tailwind of the CitusData team making continued improvement to the extension, whereas an in-house system depends on your company's ability to hire and retain people to maintain + improve the in-house system. It's hard to account for the value of benefits that have yet to accrue, but this kind of analysis, even if you pretty heavily-discount that future value, tilts the ROI in favor of soluti…

Huh, I would have thought the opposite. Companies at Figma size are easily able to hire talent to maintain a core part of their engineering stack. On the other hand, they retain no control of Citus decision making. Those tailwinds could easily have been headwinds if they went in a direction that did not suit Figma.

I think this is true for things higher up the "stack", but doesn't necessarily apply to tech like Postgres [and Citus, IMO].

The line separating "build in-house" vs "use OSS" exists, and it's at a different layer of the stack in every company. IMO, for most companies in 2024, the line puts Citus on the same side as Postgres.

FWIW, I would have assumed that Citus would be on the other end of the line, until I had to look into Citus for work for a similar reason that Figma did. You can pick and choose among the orthogonal ideas they implement that most cleanly apply to the present stage of your business, and I would've chosen to build things the same way they did (TBH, Figma's choices superficially appear to be 1:1 to Citus's choices).

Re: How Figma's databases team lived to tell the scale

#223

Post by skunkworker »

simonw wrote: post #108
skunkworker wrote: post #92

Earlier quoted context omitted.

What is your definition of "decently well", and is your primary cluster (without replicas) above 1PB?

They said 20TiB * 68, which I think is 1.5PB.

That could be all of the nodes, or just the primaries without replicas.

Re: How Figma's databases team lived to tell the scale

#224

Post by kgeist »

simonw wrote: post #221
kgeist wrote: post #208

Earlier quoted context omitted.

What do you mean by "rewriting everything"? Or maybe your definition of "per-tenant database" is different from mine. In our product, it's just a small layer which routes requests to the target organization's DB. When an organization is created, we create a new DB. Most of application code has no idea there are different DBs under the hood. There are logical DBs (folders/files on a single physical server), and there'…

If you didn't write your application with multi-tenant in mind from the start I would expect you would need to review almost every line of code that touches a database to make a transition like this.

In our code, the only DB-related piece of code which is aware of multi-tenant databases is the getDBConnection(accountId) function. Once you have the connection, you execute exact same SQL queries as before. The function is hidden deep inside the framework/infrastructure layer, so application code is completely unaware of it.

Re: How Figma's databases team lived to tell the scale

#225

Post by tharkun__ »

nightpool wrote: post #219
tharkun__ wrote: post #217

Earlier quoted context omitted.

https://news.ycombinator.com/item?id=39711815 Shopify seems to have millions of customers. Granted, they have MySQL and it might not use database schemas and something more MySQL specific. In general though ...

"We can attach a shop_id to all shop-owned tables and have it serve as a sharding key. Moving a shop from one shard to another involves selecting all records from all tables that have the desired shop_id and copying them to another MySQL shard. For this post, it’s helpful to think of each pod as a MySQL shard." Sounds like this is just normal sharding. Completely different from having a totally separate schema like G…

I don't see how it follows from that. "Attach" seems very generic.

But I do see now, searching for that part of the info specifically, they mention

    Peppy Peppers’ orders can be migrated by running SELECT FROM orders WHERE shop_id = 1 FOR UPDATE on Pod 1’s database
Yeah that's just having the tenant id in the table directly. Oh well!

Re: How Figma's databases team lived to tell the scale

#226

Post by skysignal »

How does the economics work with this?

* 21 engineers worked on it

* average salary 200K per year

* 18 months i.e 1.5 years

* 50% of the bandwidth

* 1x extra cost for all the pending features for next 3 years

so 21 * 200000 * 1.5 * 0.5 * 2 = 6.3 Million

* since they mentioned they were running largest instance in 2020, which is r5.24xlarge i.e 15.84 per hour

* they grew 100x from 2020

* assuming they went reserved instances, they would have gotten about 50% discount

* at this scale, they would have got atleast 20-30% enterprise commitment discount from aws, lets consider 25% discount

so 15.84 * 730 * 12 * 100 * 0.5 * 0.75 = 5.2 Million

if they were to go with planetscale and even if planetscale charges them 30% extra, i.e 6.76 Million, which means they are only saving about 1.56 Million per year, so it will take them atleast 4 years more to realise their cap-ex investment into building this in-house, not to mention all the extra maintainence outside the new features they are yet to implement, like reliability, observability, version upgrades etc,...

what am I missing?

Re: How Figma's databases team lived to tell the scale

#227

Post by derekperkins »

aledalgrande wrote: post #198
tharkun__ wrote: post #132

Earlier quoted context omitted.

Yes, we've been doing that at my place basically since the start. Each tenant is a schema in postgres. Works perfectly fine on the one hand, i.e. your tables don't grow to 'infinity' just because you're adding more and more tenants. If there's a particular tenant that has lots of data, only that tenant's indexes and tables grow huge and become slower because of that particular reason etc. If a tenant leaves, you keep…

> a single physical piece of hardware that will show its limitations if too many large or active tenants happen to be on the same shard Shopify has pretty much mastered this https://shopify.engineering/mysql-database-shard-balancing-t...

They have since migrated to Vitess

Re: How Figma's databases team lived to tell the scale

#229

Post by tianzhou »

tharkun__ wrote: post #132
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

Yes, we've been doing that at my place basically since the start. Each tenant is a schema in postgres. Works perfectly fine on the one hand, i.e. your tables don't grow to 'infinity' just because you're adding more and more tenants. If there's a particular tenant that has lots of data, only that tenant's indexes and tables grow huge and become slower because of that particular reason etc. If a tenant leaves, you keep…

FWIW, we at bytebase building the batch change feature to make this less painful https://www.bytebase.com/docs/change-database/batch-change/

Re: How Figma's databases team lived to tell the scale

#230

Post by Closi »

esafak wrote: post #6
Closi wrote: post #4

Earlier quoted context omitted.

This assumes that using a distributed database from the start doesn't offer it's own penalties/drawbacks (particularly in 2016). Particularly considering as per the figma note, they consider their data highly relational (i.e. presumably this means lots of joins in queries) What database would you have chosen?

That's a fair response for then, but not today where you're spoiled for choice. Spanner was released as a service in 2017, which isn't far off.

Well Spanner has it's own disadvantages, like having your entire stack tightly coupled to a google cloud product (i.e. you have to sacrifice being cloud agnostic).
Post reply on HN