101–110 of 135 posts
> The site is experiencing higher than normal traffic and we have temporarily halted database creation. Ironic coming from the infinitely scalable database, isn't it?
Earlier quoted context omitted.
Sure you can! and we will make it worth it.
I can't decide whether this joke is marketing folly or strategic genius. On the one hand, making light of how big an undertaking a _database engine migration_ would be makes you come off sounding like the "mongodb is web scale" guy. That's a pretty terrible look for a company proposing to take on critical production infrastructure. On the other hand, adding postgresql support to vitess is probably a _massive_ underta…
Earlier quoted context omitted.
As I understand it, Vitess is basically a really powerful sharding system, which goes a step further than typical sharding solutions by basically making the shards one or more unique databases. In the case of someone like slack, because your tenant (e.g your company slack), is completely isolated from other tenants, you can treat that basically as its own database, and have a master for just that DB, allowing much be…
I was Chief Architect at Slack from 2016 to 2020, and was privileged to work with the engineers who were doing the work of migrating to Vitess in that timeframe. The assumption that tenants are perfectly isolated is actually the original sin of early Slack infrastructure that we adopted Vitess to migrate away from. From some earlier features in the Enterprise product (which joins lots of "little Slacks" into a corpor…
> Today, we serve 2.3 million QPS at peak. 2M of those queries are reads and 300K are writes.
I think the "today" QPS numbers are still doable with a properly tuned single-writer galera cluster running on machines with TBs of memory. Of course, with Slack workload, there would be too much historical data to fit into a single host, so I can see the reasons to shard into multiple clusters/hosts.
Still, the numbers seem a little off. Let's say back in fall 2016 there were already 200K write QPS at peak, with 200 sharded hosts accepting write. That's just 1K write QPS at peak per host on average, and let's say 20K write QPS at peak for a particularly hot shard. What could be the bottleneck? Replication lag? Data size? I don't think any of the articles from Slack has talked about this.
What Vitess provides is invaluable, especially the very solid implementation of secondary index. But sometimes I feel like it is being used/advocated as a sledgehammer ("just keep sharding") without looking at what could be done better at the lower MySQL/InnODB level, in exchange for a much more costly cloud bill.
Before GitHub launched, I built some large eCommerce sites, and for a vcs we used CVS, and then Subversion. We had a person on our team with the title of release manager, because branching in Subversion, and merging back to make releases, was a specialist effort that took time, patience, and managing a tremendous amount of fighting between teams of what features and fixes could even be merged together in order to shi…
I think you just described why developers liked MongoDB. It does not get a lot of love on HN, but having the DB schema map to your object model is very convenient.
From the Vercel point of view, this promises to answer one of the most frequent, interesting, and technically challenging questions since we first launched our "immutable deploys". That is: how can I pair a brand new frontend preview deploy, with a serverless database with the specific schema my new feature needs? This technology makes the whole serverless stack feel complete.
It doesn’t solve the N+1 queries problem in a generic way. That is a major hurdle in scaling complexity-wise , which in turn is often deeply coupled with business realities. More to the point, it probably cannot solve it efficiently at all, since it is not a graph database and thus cannot be paired with a generic GraphQL resolver (would generate join queries instead of lookups across edges) and a stack of generated s…
Before GitHub launched, I built some large eCommerce sites, and for a vcs we used CVS, and then Subversion. We had a person on our team with the title of release manager, because branching in Subversion, and merging back to make releases, was a specialist effort that took time, patience, and managing a tremendous amount of fighting between teams of what features and fixes could even be merged together in order to shi…
Earlier quoted context omitted.
he is probably referring to the docdb document store in yugabyte: https://docs.yugabyte.com/latest/architecture/layered-archit...
Indeed I was referring to yugabyte, apologies for the clumsy phrasing, I havent used crdb but I guess it is a postgres frontend layered on a KV store instead of a document store?
CockroachDB (aka CRDB) is completely custom and compatible with Postgres wire/datatype protocols, which operates directly on its own key/value store called Pebble (but originally was also RocksDB): https://www.cockroachlabs.com/blog/distributed-sql-key-value...
Both systems are foundationally the same SQL-on-KV but implement it very differently.
It looks great, congrats
Can someone please explain how Vitess works, in plain English? How does it magically make MySQL scale? And then what does PlanetScale add on top of Vitess hosted anywhere else? Sorry, the linked blog post is both very abstract and assumes a high level of preexisting knowledge about database scaling.
Implementations vary but there are the 2 major architectures: systems like Vitess/Proxy SQL/Citus/Timescale that act as a proxy layer on top of existing RDBMS running on multiple servers to make them look like a single database, and entirely custom projects like CockroachDB/TiDB/Yugabyte/Cloud Spanner which have their own native processing and data layers.
OLAP relational data warehouses like Vertica/Greenplum/MemSQL/Redshift/Bigquery are also natively distributed but focus on large-scale analytics with features like column-oriented storage and vectorized processing.