It would have benefitted the article to use less of annoying corporate talk.
How Figma's databases team lived to tell the scale
51–60 of 233 posts
Re: How Figma's databases team lived to tell the scale
#52I see they don't mention Citus ( https://github.com/citusdata/citus ), which is already a fairly mature native Postgres extension. From the details given in the article, it sounds like they just reimplemented it. I wonder if they were unaware of it or disregarded it for a reason —I currently am in a similar situation as the one described in the blog, trying to shard a massive Postgres DB.
Figma uses AWS RDS, RDS doesn't list citus as a supported extension.
Re: How Figma's databases team lived to tell the scale
#53Could you use Aurora Limitless for this instead? https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-au...
I doubt even VC money can afford this service. Serverless Aurora is incredibly expensive for most workloads. I have yet to find a use case for any SaaS product that is used >4 hours a day. Since all my products span at least 3 time zones there is at least 12 hours of activity a day.
What products
Re: How Figma's databases team lived to tell the scale
#54Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works. It seems like they've (hop…
Never a good idea to rely on Google proprietary tech (unless you are Google)... it could be sunset at any time without warning. I use GCP but I try my best to stay Google agnostic (avoid GCP-only offerings, etc) so that I can move to AWS if Google pulls the rug out from under me.
GCP products have a much better track record than Google consumer products when it comes to support since there are usually enterprise customers with multi-year contracts worth tens, if not hundreds, of millions of dollars using them.
Re: How Figma's databases team lived to tell the scale
#55Could you use Aurora Limitless for this instead? https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-au...
I doubt even VC money can afford this service. Serverless Aurora is incredibly expensive for most workloads. I have yet to find a use case for any SaaS product that is used >4 hours a day. Since all my products span at least 3 time zones there is at least 12 hours of activity a day.
We found this out the hard way in a small startup. The per query and I/O expense was through the roof.
Re: How Figma's databases team lived to tell the scale
#56One thought that comes up: Wouldn’t it be easier to have each customer in their own (logical) database? I mean, you don’t need transactions across different customers, right? So you’re essentially solving a harder problem than the one you’ve got. Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?
It sounds like they actually did something like this. Their shard key selection could be customer, project, folder or something in their data model at a reasonably high logical level in their hierarchy.
Re: How Figma's databases team lived to tell the scale
#57> NoSQL databases are another common scalable-by-default solution that companies adopt as they grow. However, we have a very complex relational data model built on top of our current Postgres architecture and NoSQL APIs don’t offer this kind of versatility.
As I understand it, NoSQL is for people who need a backend that ingests just about any unstructured data, for teams that may not have a complex relational model worked out/stabilized. Postgres has this in its native jsonb datatype, but they wouldn't need to use that much since it sounds like they already have a good data model. What am I missing here?
Re: How Figma's databases team lived to tell the scale
#58Could you use Aurora Limitless for this instead? https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-au...
Yes, but that wasn't available when they did this migration
Re: How Figma's databases team lived to tell the scale
#59Could you use Aurora Limitless for this instead? https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-au...
At my company, we were given an early talk on Limitless. Never once did the reps mention that it ran on Serverless. Dear lord, that's going to be a hard no from me. Unless they've dramatically changed pricing for Limitless as opposed to normal Serverless, that'll be through the roof.
Re: How Figma's databases team lived to tell the scale
#60Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works. It seems like they've (hop…
My perspective from working both inside and outside of Google:
The external spanner documentation doesn’t seem as good as the internal documentation, in my opinion. Because it’s not generally well known outside of G, they ought to do a better job explaining it and its benefits. It truly is magical technology but you have to be a database nerd to see why.
It’s also pretty expensive and because you generally need to rewrite your applications to work with it, there is a degree of lockin. So taking on Spanner is a risky proposition - if your prices get hiked or it starts costing more than you want, you’ll have to spend even more time and money migrating off it. Spanner’s advantages over other DBs (trying to “solve” the CAP theorem) then become a curse, because it’s hard to find any other DB that gives you horizontal scaling, ACID, and high availability out of the box, and you might have to solve those problems yourself/redesign the rest of your system.
Personally I would consider using Cloud Spanner, but I wouldn’t bet my business on it.