Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Figma's databases team lived to tell the scale

figma.com

Post reply on HN
Print view View original

191–200 of 233 posts

Re: How Figma's databases team lived to tell the scale

#191

Post by weitendorf »

nojvek wrote: post #178
weitendorf wrote: post #75

Earlier quoted context omitted.

I'm biased having worked on GCP, but I think GCP actually has a very good track record of not sunsetting entire products or removing core functionality. When I worked on AppEngine, I would often see apps written 10+ years ago still chugging along. It is true though that GCP sometimes sunsets specific product functionality, requiring changes on the customers' part. Some of these are unavoidable (eg committing to apply…

My previous employer was a GCP customer. Google did pull occasional shenanigans totally breaking us with random upgrades without notifying us. Their support wouldn’t acknowledge it was their fault until we were mad at them. My newer employer is AWS. Their offerings are a lot more stable and support is helpful. If you want to build a serious business I would avoid GCP. Google doesn’t really give a shit at winning Clou…

Ugh I'm so sorry, you should definitely have been notified in advance about breaking changes, unless they were accidental bugs and regressions. That was something we took pretty seriously for the products I worked on.

I have definitely enjoyed using AWS support. But I've also encountered some broken/janky functionality (eg using custom domain + api gateway + websockets + Lambda for the apparently niche-task of having a real website use the Lambda-websockets integration) on AWS that I don't think would have made it to production at Google. I also really dislike how they handle project-level logging compared to how it's done on GCP.

Ultimately I do think some GCP products like Bigquery, Cloud Run (I'm biased here), and Spanner as well as general DevEx/reliability factors are compelling enough for GCP to be worth serious consideration, even if AWS offers better support.

Re: How Figma's databases team lived to tell the scale

#192

Post by paulryanrogers »

brigadier132 wrote: post #161
hintymad wrote: post #87

Given that sharding has become a pretty mature practice, is it still worth considering the NewSQL solutions like CRDB, Yugabyte, and TiDB for the sake of auto sharding, given that these NewSQL databases usually trade throughput and latency for auto-sharding and multi-region support? Another added cost is learning how to operate NewSQL databases, assuming one is already familiar with either MySQL or Postgres.

Sharding mysql and postgres has been a shitshow at every company I've worked at.

Why? How did they shard?

Re: How Figma's databases team lived to tell the scale

#193

Post by weitendorf »

thekhatribharat wrote: post #146

1. They mention that the largest tables ran into several TBs, and they would have soon topped the max IOPS supported by RDS. RDS for PostgreSQL peaks at 256,000 IOPS for a 64 TB volume. For a multi-AZ setup, this costs ~$70K/mo. 2. Let's assume the final outcome was a 5-way shard with each shard supporting ~50,000 IOPS and ~12 TB data. For a multi-AZ setup, this costs ~$100K/mo. 3. It took 9 months to shard their fir…

Figma, worth $10billion, was migrating what seems like their core production data. They probably didn't want to bet the company on a comparatively small software vendor like Yugabyte.

Most likely the engineering cost was much much higher than your quotes, but still insignificant compared to the potential risks. And migrating from RDS to not-RDS could easily not have been cheap in engineering time either, depending on how much software they've built around it.

Re: How Figma's databases team lived to tell the scale

#194

Post by shalabhc »

jasonwatkinspdx wrote: post #128
ummonk wrote: post #121

Earlier quoted context omitted.

This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.

I think there's quite a few people chasing similar ideas, like Azure's Durable Entities. I've been calling it the Lots of Little Databases model vs the Globe Spanning Gorilla. Like the Spanner paper points out, even if your distributed database semantically appears like a single giant instance, in practice performance means developers avoid using distributed joins, etc, because these can lead to shuffling very large…

"Lots of Little Databases" reminded me of https://www.actordb.com/ which does lots of server-side sqlite instances, but the project now looks defunct.

Re: How Figma's databases team lived to tell the scale

#195

Post by weitendorf »

nojvek wrote: post #180
opportune wrote: post #60

Earlier quoted context omitted.

My perspective from working both inside and outside of Google: The external spanner documentation doesn’t seem as good as the internal documentation, in my opinion. Because it’s not generally well known outside of G, they ought to do a better job explaining it and its benefits. It truly is magical technology but you have to be a database nerd to see why. It’s also pretty expensive and because you generally need to re…

If you really have that much data and traffic, the $ costs start to add up to multiple engineer comp costs. At that point it’s cheaper to move to something you have good control over. I.e sharding at application layer and connecting to the DB instance replica where the customer data is hosted.

Depends. The cost may pay for itself but the engineers you have already may have higher ROI things to do. It's also nice to have operational stuff managed for you. Personally I'd be happy to pay extra for the kinds of problems Spanner solves to free myself up to do other things (to a point, ofc).

> sharding at application layer and connecting to the DB instance replica where the customer data is hosted.

Spanner does global consistency/replication. If having good performance per-tenant globally is a concern, this helps a lot, and is hard to implement on your own. It can also ultimately save you money by limiting cross-region traffic.

Re: How Figma's databases team lived to tell the scale

#196

Post by DonnyV »

Is it me or did they just recreate Mongodb Shard feature? But now they have to maintain it. They would of been better off going Mongodb or another document database. Then you wouldn't have to worry about scheme changes. Since your scheme lives there n your data model. You could just map changed fields. They should of just used a tenant based design. If a customer was hindering performance on neighboring tenants. They could just migrate them over to their own large server.

Re: How Figma's databases team lived to tell the scale

#197

Post by hintymad »

brigadier132 wrote: post #161
hintymad wrote: post #87

Given that sharding has become a pretty mature practice, is it still worth considering the NewSQL solutions like CRDB, Yugabyte, and TiDB for the sake of auto sharding, given that these NewSQL databases usually trade throughput and latency for auto-sharding and multi-region support? Another added cost is learning how to operate NewSQL databases, assuming one is already familiar with either MySQL or Postgres.

Sharding mysql and postgres has been a shitshow at every company I've worked at.

Thanks. This is kinda of information I look for. Could you give more specifics? Why were they shit shows? If the tables are naturally shardable, say by user ID as described in the Figma's case, would the situation be different?

Re: How Figma's databases team lived to tell the scale

#198

Post by aledalgrande »

tharkun__ wrote: post #132
bjornsing wrote: post #29

One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?

Yes, we've been doing that at my place basically since the start. Each tenant is a schema in postgres. Works perfectly fine on the one hand, i.e. your tables don't grow to 'infinity' just because you're adding more and more tenants. If there's a particular tenant that has lots of data, only that tenant's indexes and tables grow huge and become slower because of that particular reason etc. If a tenant leaves, you keep…

> a single physical piece of hardware that will show its limitations if too many large or active tenants happen to be on the same shard

Shopify has pretty much mastered this https://shopify.engineering/mysql-database-shard-balancing-t...

Re: How Figma's databases team lived to tell the scale

#199

Post by ummonk »

themoonisachees wrote: post #155
jasonwatkinspdx wrote: post #128

Earlier quoted context omitted.

I think there's quite a few people chasing similar ideas, like Azure's Durable Entities. I've been calling it the Lots of Little Databases model vs the Globe Spanning Gorilla. Like the Spanner paper points out, even if your distributed database semantically appears like a single giant instance, in practice performance means developers avoid using distributed joins, etc, because these can lead to shuffling very large…

An advantage worth noting is that having actually separated databases means you physically can't make these expensive operations, so a junior dev can't write incredibly inefficient code that would bring down your entire infra.

Also makes it a lot harder for devs to do some footgun and leak data across domains.

Re: How Figma's databases team lived to tell the scale

#200

Post by ummonk »

jasonwatkinspdx wrote: post #128
ummonk wrote: post #121

Earlier quoted context omitted.

This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.

I think there's quite a few people chasing similar ideas, like Azure's Durable Entities. I've been calling it the Lots of Little Databases model vs the Globe Spanning Gorilla. Like the Spanner paper points out, even if your distributed database semantically appears like a single giant instance, in practice performance means developers avoid using distributed joins, etc, because these can lead to shuffling very large…

Interesting. Durable Entities strikes me as closer to Cloudflare's Durable Objects (both in name and in design as actors backed by persistent storage).
Post reply on HN