I find the wheel is most typically reinvented in pursuit of venture capital.
[deleted]
Why Has Figma Reinvented the Wheel with PostgreSQL?
41–50 of 98 posts
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#42The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many…
I can buy your comment as an interesting and even credible hypothesis, but the absolutes which you deal in (“doesn’t pass even the most basic sniff test”) are damning. You are clearly lacking huge amounts of information and context and are passing your own assumptions as hard facts. Also, I’m assuming Amazon or Google will sometimes roll their own solutions on problems of a scale in the same ballpark as Figma’s. But…
I like the approach you took for questioning an unqualified claim.
Seems like a useful argument design pattern.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#43I'm at a company that is weighing a very similar decision (we are on RDS Postgres with a rapidly growing database that will require some horizontal partitioning). There really isn't an easy solution. We spoke to people who have done sharding in-house (Figma, Robinhood) as well as others who migrated to natively distributed systems like Cockroach (Doordash). If you decide to move off of RDS but stay on Postgres, you c…
We use Citus. Very similar performance properties to DIY sharding but much more polished. Currently at 7 TB, self hosted. Growing roughly at 100 % per year, write-heavy. Works fine for us.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#44Aren't there any good managed postgres solutions supporting citus? The decision here seems to have been to invent a whole new sharding solution instead of building enough in house DBA to self-host postgres (if you want to stay on Amazon, you can use any extension you want on EC2). Speaks for the state of engineering right now.
Citus was bought by Microsoft so now it's only offered as Azure managed service.
It's AGPL so you can self-host. You can even pay someone to host it for you if you don't like Microsoft's offer. I find it easier than rolling your own AND self-hosting.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#45The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many…
I can buy your comment as an interesting and even credible hypothesis, but the absolutes which you deal in (“doesn’t pass even the most basic sniff test”) are damning. You are clearly lacking huge amounts of information and context and are passing your own assumptions as hard facts. Also, I’m assuming Amazon or Google will sometimes roll their own solutions on problems of a scale in the same ballpark as Figma’s. But…
In this particular case of sharding a postgresql solution, in my opinion, the parent is right. Any major cloud provider would give companies of their scale assistance. This is their bread and butter. The posts likely hide the requirement of stay on aws, but we don’t know they did not talk about that. Likewise cockroach or yugabyte were also available options.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#46Nice article, I also wondered why they omitted citus. Is there any plan from rds to offer it? Obviously it could interfere with demand for aurora limitless
> Is there any plan from rds to offer it? I don't think so because Microsoft bought Citus.
That, combined with AGPL licensing means the other big clouds won't touch it, as their lawyers have (rightly or wrongly) deemed AGPL too risky.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#47Even notion has a similar approach to sharding postgres but both of them could benefit from having shard IDs prefixed with YY/MM/DD(as needed) otherwise it's back to the shard navigator once they max out against org-ids for each shard's capacity
(I work at Notion) Our shard key - Workspace ID - is a UUIDv4 so there’s a pretty high number of orgs per shard without conflict.
Hey Jake, I meant capacity per shard in this case, not exhausting the IDs. Any potential solutions for that or is that not an immediate challenge?
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#48Now seems a good time to point out, the wheel has literally been reinvented over and over again. The wheels of yesterday were terrible. Each version gets better. It's fine, reinvent away folks :)
Seriously, a naive database sharding algorithm could be implemented in a week or so by a competent dev. A company like figma (billions in revenue) putting a small team to implementing a database sharding solution for an un-implemented use case (RDS, not just postgres). AND open sourcing it creating a value for the community is a net-good for the industry.
And that would be the solution you'd absolutely abhor.
Database sharding has a bunch of gotchas and things to think about because you must consider query access patterns along with your sharding (unless you want the devs to get owned or have very weird behavior.)
Building something super simple can be ok for the base use case but if you are a multi-billion dollar company you can probably afford a few dbas to actually make your platform good.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#49The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many…
We went through something similar at Notion a few years ago and also chose to stick with RDS Postgres and build sharding logic in our application’s database client. In both our case and Figma’s, sharding Postgres ASAP was of critical importance because of transaction ID wraparound threat or other capacity issues that promise hard days-long downtime. The kind of downtime that costs 10s of millions of dollars of brand…
Or you could just hire some set of people who know how to manage postgres? Seems easier than building an entirely new thing with its own set of bugs that are unknown unknown brand damage awaiting you.
Re: Why Has Figma Reinvented the Wheel with PostgreSQL?
#50Aren't there any good managed postgres solutions supporting citus? The decision here seems to have been to invent a whole new sharding solution instead of building enough in house DBA to self-host postgres (if you want to stay on Amazon, you can use any extension you want on EC2). Speaks for the state of engineering right now.
Crunchy data (which can run on AWS/Azure/GCP) supports Citus: https://www.crunchydata.com/blog/announcing-citus-support-fo...