Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

151–160 of 233 posts

Re: How Figma's databases team lived to tell the scale

#151

Post by Scubabear68 »

yard2010 wrote: post #142
Scubabear68 wrote: post #55

Earlier quoted context omitted.

We found this out the hard way in a small startup. The per query and I/O expense was through the roof.

Did it work though? Did you achieve unlimited scaling? Because if so you should compare the price to the price of a team of great minds such as in this article, working for 2 years to get a solution. I bet it still would be cheaper to pay people over Amazon, but I'm curious about the numbers

It worked fine, the problem was our workloads were minuscule and it still cost $3,000 a month to support 50ish users on a lightly used platform.

The same load in a non-Serverless instance was around $100/month.

Re: How Figma's databases team lived to tell the scale

#152

Post by vmfunction »

jamesfinlayson wrote: post #129
aeyes wrote: post #46

Earlier quoted context omitted.

I have pondered about this for quite some time and came to the conclusion that it would make schema migrations more difficult to handle. I think Shopify is using an approach which is similar to what you are describing. The advantage is that you don't end up with hot shards because you can move around large customers independently. In practice there isn't a big difference, they just colocate several customers accordin…

I remember Postgres table spaces being used to separate customers at a previous job - I can't remember how migrations were handled (pretty sure they were applied per table space) but I don't think it was a problem (at our scale anyway).

>There is usually not much point in making more than one tablespace per logical file system, since you cannot control the location of individual files within a logical file system. However, PostgreSQL does not enforce any such limitation, and indeed it is not directly aware of the file system boundaries on your system. It just stores files in the directories you tell it to use.

seems like the limitation is the logical file system. Which probably will work for most users.

Re: How Figma's databases team lived to tell the scale

#153

Post by Syntaf »

tharkun__ wrote: post #132
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

Yes, we've been doing that at my place basically since the start. Each tenant is a schema in postgres. Works perfectly fine on the one hand, i.e. your tables don't grow to 'infinity' just because you're adding more and more tenants. If there's a particular tenant that has lots of data, only that tenant's indexes and tables grow huge and become slower because of that particular reason etc. If a tenant leaves, you keep…

I recall a popular rails gem[1] once upon a time that provided multi-tenancy via postgres schemas.

As it turns out, even the company the initially developed the gem ended up ditching the approach due to some of the issues you outlined above.

Managing separate schemas feels like one of those nefarious decisions that make things simple _initially_ but get you into a world of hurt when you need to scale. The company is since defunct but they have an article where they discuss why they ditched the approach [2], TL;DR it's too difficult to maintain and scale

[1] https://github.com/influitive/apartment#tenants-on-different... [2] https://web.archive.org/web/20201108191323/https://influitiv...

Re: How Figma's databases team lived to tell the scale

#154

Post by avianlyric »

thekhatribharat wrote: post #146

1. They mention that the largest tables ran into several TBs, and they would have soon topped the max IOPS supported by RDS. RDS for PostgreSQL peaks at 256,000 IOPS for a 64 TB volume. For a multi-AZ setup, this costs ~$70K/mo. 2. Let's assume the final outcome was a 5-way shard with each shard supporting ~50,000 IOPS and ~12 TB data. For a multi-AZ setup, this costs ~$100K/mo. 3. It took 9 months to shard their fir…

Their rationale for this choice is covered in the article somewhat extensively near the top.

> Additionally, over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch. Given our very aggressive growth rate, we had only months of runway remaining. De-risking an entirely new storage layer and completing an end-to-end-migration of our most business-critical use cases would have been extremely risky on the necessary timeline. We favored known low-risk solutions over potentially easier options with much higher uncertainty, where we had less control over the outcome.

TL;DR they were working in a short timeline, with a limited team size, and wanted to minimise any risks to the business.

Clearly cost is an issue for Figma, but downtime, or worse data loss, would have a ginormous impact on their business and potential future growth. If your product is already profitable, your user base growing fast, and with your ARR. Why would risk that growth and future ARR just to save a few $10Ks a month? A very low risk DB migration that lets you keep scaling and raking in more money, is way better than a high risk migration that might save some cash in the long term, but also risks killing your primary business if it goes wrong.

Re: How Figma's databases team lived to tell the scale

#155

Post by themoonisachees »

jasonwatkinspdx wrote: post #128
ummonk wrote: post #121

Earlier quoted context omitted.

This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.

I think there's quite a few people chasing similar ideas, like Azure's Durable Entities. I've been calling it the Lots of Little Databases model vs the Globe Spanning Gorilla. Like the Spanner paper points out, even if your distributed database semantically appears like a single giant instance, in practice performance means developers avoid using distributed joins, etc, because these can lead to shuffling very large…

An advantage worth noting is that having actually separated databases means you physically can't make these expensive operations, so a junior dev can't write incredibly inefficient code that would bring down your entire infra.

Re: How Figma's databases team lived to tell the scale

#156

Post by cqqxo4zV46cp »

Syntaf wrote: post #153
tharkun__ wrote: post #132

Earlier quoted context omitted.

Yes, we've been doing that at my place basically since the start. Each tenant is a schema in postgres. Works perfectly fine on the one hand, i.e. your tables don't grow to 'infinity' just because you're adding more and more tenants. If there's a particular tenant that has lots of data, only that tenant's indexes and tables grow huge and become slower because of that particular reason etc. If a tenant leaves, you keep…

I recall a popular rails gem[1] once upon a time that provided multi-tenancy via postgres schemas. As it turns out, even the company the initially developed the gem ended up ditching the approach due to some of the issues you outlined above. Managing separate schemas feels like one of those nefarious decisions that make things simple _initially_ but get you into a world of hurt when you need to scale. The company is…

This is the conclusion I came to when faced with the same quandary.

Re: How Figma's databases team lived to tell the scale

#157

Post by albert_e »

sgarland wrote: post #124
hacker_newz wrote: post #111

Earlier quoted context omitted.

If tenants are on separate databases how would that be an issue?

Because database, in Postgres terms, doesn’t mean physical node. It’s more akin to a VM than anything else. The term for an installation of Postgres is database cluster, which can contain N databases, each of which can contain M schemas.

Thanks! Is there a good primer to this terminology that clarifies these terms on various popular database and cloud platforms?

It seems there is good potential for confusion unless we use the same terms consistently when discussing architecture and design across teams.

Even the term RDS (Relational Database Service) is sometimes said to be inaccurate since it is a "Relational Database SERVER as a Service"

A few related terms that cause confusion:

"Schema" could refer to a Database Schema or in some contexts, a single table's logical data structure. (Or a single data set's data structure -- or a standard for one, like JSON Schema)

Data Catalog products like "AWS Glue Data Catalog" which only store the metadata or schemas of the table they crawl ... refer to the entities they store as "Databases" and "Tables" (but no "Schemas") and documentation includes guides talk about "creating a database"[1] and "creating a table"[2] in AWS Glue. There has to be a better way to refer to all these entities without using the word schema with so many meanings -- or calling both physical tables and their metadata as "tables". Otherwise this is needlessly confusing and hard for beginners.

---

EDIT: even more madness. This StackOverflow discussion [3] has more examples of confusing usage:

> On Oracle .... A Schema is effectively a user. More specifically it's a set of tables/procs/indexes etc owned by a user. Another user has a different schema (tables he/she owns) however user can also see any schemas they have select priviliedges on.

> MySQL: database/schema :: table

> SQL Server: database :: (schema/namespace ::) table

> Oracle: database/schema/user :: (tablespace ::) table

[1]: https://docs.aws.amazon.com/glue/latest/dg/define-database.h...

[2]: https://docs.aws.amazon.com/glue/latest/dg/tables-described....

[3]: https://stackoverflow.com/questions/298739/what-is-the-diffe...

Re: How Figma's databases team lived to tell the scale

#158

Post by vmfunction »

ijustlovemath wrote: post #57

> NoSQL databases are another common scalable-by-default solution that companies adopt as they grow. However, we have a very complex relational data model built on top of our current Postgres architecture and NoSQL APIs don’t offer this kind of versatility. As I understand it, NoSQL is for people who need a backend that ingests just about any unstructured data, for teams that may not have a complex relational model w…

if you have postgres, just use https://github.com/FerretDB/FerretDB

Re: How Figma's databases team lived to tell the scale

#159

Post by brigadier132 »

avianlyric wrote: post #154
thekhatribharat wrote: post #146

1. They mention that the largest tables ran into several TBs, and they would have soon topped the max IOPS supported by RDS. RDS for PostgreSQL peaks at 256,000 IOPS for a 64 TB volume. For a multi-AZ setup, this costs ~$70K/mo. 2. Let's assume the final outcome was a 5-way shard with each shard supporting ~50,000 IOPS and ~12 TB data. For a multi-AZ setup, this costs ~$100K/mo. 3. It took 9 months to shard their fir…

Their rationale for this choice is covered in the article somewhat extensively near the top. > Additionally, over the past few years, we’ve developed a lot of expertise on how to reliably and performantly run RDS Postgres in-house. While migrating, we would have had to rebuild our domain expertise from scratch. Given our very aggressive growth rate, we had only months of runway remaining. De-risking an entirely new s…

Ok, what risk? Cockroachdb is already proven technology and costs marginally more (if you use their serverless setup, it's free until you hit real scale). At the startups I've been at that hit scale, scaling sql was always a massive undertaking and affected product development on every single time.

If you don't want downtime, don't use databases that require downtime to do a migration?

Netflix, roblox, every single online gambling website all use cockroachdb.

Re: How Figma's databases team lived to tell the scale

#160

Post by menthe »

bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

This, seriously. The long-term maintenance, tribal knowledge & risks associated with this giant hack will be greater than anything they'd ever have expected. Inb4 global outage post-mortem & key-man dependency salaries.

There's no virtually no excuse not spinning up a pg pod (or two) for each tenant - heck even a namespace with the whole stack.

Embed your 4-phases migrations directly in your releases / deployments, slap a py script to manage progressive rollouts, and you're done.

Discovery is automated, blast / loss radius is reduced to the smallest denominator, you can now monitor / pin / adjust the stack for each customer individually as necessary, sort the release ordering / schedule based on client criticality / sensitivity, you can now easily geolocate the deployment to the tenant's location, charge by resource usage, and much more.

And you can still query & roll-up all of your databases at once for analytics with Trino/DBT with nothing more but a yaml inventory.

No magic, no proprietary garbage.

Post reply on HN