Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Has Figma Reinvented the Wheel with PostgreSQL?

medium.com

Post reply on HN
Print view View original

91–98 of 98 posts

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#91

Post by [deleted] »

jitl wrote: post #38
seanhunter wrote: post #4

The answer is obvious: they invented their own sharding solution because it's a really really cool problem to work on and they have more engineers than they really need to develop their actual product. A more resource-constrained team would have found a solution that sharded their backend using one of the existing solutions out there. I have seen this several times before and it's always a symptom of having too many…

We went through something similar at Notion a few years ago and also chose to stick with RDS Postgres and build sharding logic in our application’s database client. In both our case and Figma’s, sharding Postgres ASAP was of critical importance because of transaction ID wraparound threat or other capacity issues that promise hard days-long downtime. The kind of downtime that costs 10s of millions of dollars of brand…

[deleted]

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#92

Post by mixmastamyk »

Ozzie_osman wrote: post #89
krab wrote: post #43

Earlier quoted context omitted.

We use Citus. Very similar performance properties to DIY sharding but much more polished. Currently at 7 TB, self hosted. Growing roughly at 100 % per year, write-heavy. Works fine for us.

Curious why you needed to shard at 7TB? I can imagine for some workloads, especially if it's write-heavy, you might start hitting constraints around vacuuming and things like that? But 7TB should be manageable on a (somewhat large and beefy) single machine.

Don’t wait until the last possible second to make a big strategic move—do it early on your own schedule. Especially when growing at a high rate.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#93

Post by seanhunter »

thih9 wrote: post #74
seanhunter wrote: post #33

Earlier quoted context omitted.

Wanting to stay on RDS is a reason doesn't survive the sort of extra scrutiny that I said should be applied in situations where you're doing a lot of work towards an internal goal. It also says in the article that they thought it was too risky to migrate (but somehow building their own sharding solution is going to be less risky for some reason). I could of course be wrong but it really just feels to me like the reas…

Looks like you can’t think of a good reason to stay on RDS in this case, is that correct?

I can totally see why they want to stay on RDS, but think the other considerations should almost certainly outweigh that.

My main point is this decision makes no sense on its face[1]. Obviously I'm lacking the real context, so there may be overwhelming circumstances which mean that it was the right decision anyway, but these weren't explained in TFA for me. In TFA the reasoning was superficial, and this is the sort of decision that really should be held to a very high standard because as I say these types of internal goals have the potential to burn a ton of valuable engineering time on things which don't affect the customer-facing offering.

Now we have in a sibling thread someone from notion saying they did the same thing and for me exactly the same reasoning applies. It could be that all these different Saas companies are so special that them each building their own individual postgres sharding solutions to work around the fact that they can't get a sharded, managed postgres instance makes sense. Or not.

[1] That's what I mean by saying it doesn't pass the sniff test. It might actually be the right decision but your instincts should rebel against it because it feels very wrong. So there needs to be a serious examination before going down that path.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#94

Post by krab »

Ozzie_osman wrote: post #89
krab wrote: post #43

Earlier quoted context omitted.

We use Citus. Very similar performance properties to DIY sharding but much more polished. Currently at 7 TB, self hosted. Growing roughly at 100 % per year, write-heavy. Works fine for us.

Curious why you needed to shard at 7TB? I can imagine for some workloads, especially if it's write-heavy, you might start hitting constraints around vacuuming and things like that? But 7TB should be manageable on a (somewhat large and beefy) single machine.

You're right we could. In fact, it was a single server until about 2 TB. We considered a larger server and in fact at that point we could have just added a few more disks. But we still decided to shard.

First, the data size is growing and we didn't really know the growth rate in advance. Sharding gives you some flexibility in the infrastructure sizing. And yes, you don't want to wait until the last minute.

Second, it helps us to spread the disk I/O. Possible on a single machine if you're a little bit careful with disk types and sizes. But again, the overall load still grows.

Third, all the bulk operations take a long time on a single server. Each of the distributed servers takes about an hour to back up and 2-3 hours to restore. I'd feel uneasy if it was much longer.

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#95

Post by jiggawatts »

djtango wrote: post #10
jiggawatts wrote: post #6

Earlier quoted context omitted.

I just read through several thousand lines of code re-implementing the concept of a distributed queue from the ground up... for an application that has maybe a few hundred users. And doesn't need queues, at all. This issue is so pervasive that we've all just assumed that it must be necessary.

I couldn't get the context of this response - is this application you read unrelated to the featured article? I just read the article and from what I can tell the Figma team made a somewhat reasonable sounding decision

Yes, unrelated. My point was that wheel-reinvention is a curse of the software industry because it's just so easy to reinvent every wheel on a whim. DevOps and is no different. How many large orgs have their own build tooling, or some special sauce around large repos?

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#97

Post by pas »

eivanov89 wrote: post #40
franckpachot wrote: post #30

Earlier quoted context omitted.

What I do not understand is they say "we explored CockroachDB, TiDB, Spanner, and Vitess". Those are not compatible with PostgreSQL beyond the protocol and migration would require massive rewrites and tests to get the same behavior. YugabyteDB is using PostgreSQL for the SQL processing, to provide same features and behavior and distributes with a Spanner-like architecture. I'm not saying that there's no risk and no e…

I think Denis addressed this in his post: "Overall, as an engineer, you will never regret taking part in the development of a sharding solution. It’s a complex engineering problem with many non-trivial tasks to solve". In other words, it might be not invented here syndrome (with all due respect to Figma team). Or there might be more nuances we are unaware about.

they wanted to stay on RDS, maybe not "them", maybe it was the decision of some manager

also, it's... strange that they had 18 months and "extremely tight timeline pressure" we simply don't know enough about the situation

Re: Why Has Figma Reinvented the Wheel with PostgreSQL?

#98

Post by jerrygenser »

krab wrote: post #44
jerrygenser wrote: post #29

Earlier quoted context omitted.

Citus was bought by Microsoft so now it's only offered as Azure managed service.

It's AGPL so you can self-host. You can even pay someone to host it for you if you don't like Microsoft's offer. I find it easier than rolling your own AND self-hosting.

Is there another managed service of Citus that is NOT offered by MSFT? I understand it's legally possible, but is anyone else actually hosting a managed Citus?
Post reply on HN