21–30 of 233 posts
Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works. It seems like they've (hop…
Earlier quoted context omitted.
If every startup used every scale-proof method available from the beginning they'd never have the time or resources to build the products to get the customers that would require them to use the scale-proof methods
I agree with your general point but there are relational distributed databases, open source and "serverless", that are compatible with MySQL (planetscale, tidb, vitess...) and postgres (citus, cockroachdb...) edited for examples.
This is such a familiar story. Company starts with a centralized database, runs into scaling problems, then spends man-years sharding it. Just use a distributed database.
This assumes that using a distributed database from the start doesn't offer it's own penalties/drawbacks (particularly in 2016). Particularly considering as per the figma note, they consider their data highly relational (i.e. presumably this means lots of joins in queries) What database would you have chosen?
Interesting that they had problems with vacuuming. I always thought that part of Postgres was the worst part. I vaguely remember that you needed twice the space to vacuum successfully, which hopefully has changed in later versions.
An article about why vacuum is needed in pg (as compared to mysql/innodb).
http://rhaas.blogspot.com/2011/02/mysql-vs-postgresql-part-2...
Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works. It seems like they've (hop…
Hmm, wonder why they didn't try FoundationDB. Interesting that they had problems with vacuuming. I always thought that part of Postgres was the worst part. I vaguely remember that you needed twice the space to vacuum successfully, which hopefully has changed in later versions. An article about why vacuum is needed in pg (as compared to mysql/innodb). http://rhaas.blogspot.com/2011/02/mysql-vs-postgresql-part-2...
Hmm, wonder why they didn't try FoundationDB. Interesting that they had problems with vacuuming. I always thought that part of Postgres was the worst part. I vaguely remember that you needed twice the space to vacuum successfully, which hopefully has changed in later versions. An article about why vacuum is needed in pg (as compared to mysql/innodb). http://rhaas.blogspot.com/2011/02/mysql-vs-postgresql-part-2...
We do not handle, let's say low-level I/O buffering/caching, by ourselves anymore, right? (at least the folks doing web development/saas). We rely instead of the OS APIs, and that's good. I think we are missing something similar but for db sharding. It seems to me that we are still missing some fundamental piece of technology/infrastructure to handle horizontal data sharding.
Not sure postgres (logical) databases would scale that well, but don’t see a principal reason why it couldn’t. Has anyone explored this further?
Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works. It seems like they've (hop…
The other side of something like Spanner is the quorum-based latency is often optimized by adding another cache on top, which instantly defeats the original consistency guarantees. The consistency of (spanner+my_cache) is not the same as the consistency of spanner. So if we're back to app level consistency guarantees anyway, turns out the "managed" solution is only partial.
Ideally the managed db systems would have flexible consistency, allowing me to configure not just which object sets need consistency but also letting me configure caches with lag tolerance. This would let me choose trade-offs without having to implement consistent caching and other optimization tricks on top of globally consistent/serializable databases.