The DynamoDB Paper

Lately, the problem I've seen is people who haven't even considered whether their problem is truly unique enough that a traditional RDBMS couldn't handle it without some tuning. (Here I don't count "set up the obvious index" as "tuning", because if you're using a non-RDBMS the same work is encompassed in figuring out what to use as keys. No escaping that one regardless of technology.)

I'm losing track of the number of teams in my company I've seen switching databases after they rolled to production because it turns out they picked a database that doesn't support the primary access pattern for their data in some cases, or in other cases, a very common secondary access pattern. In all the cases I've seen so far, it's been for quantities of data that an RDMBS would have chewed up and spat out without even noticing. It's amazing how much trouble you can get yourself into with non-relational databases with just a few hundred megabytes of data, or even a few tens of megabytes of data if you fall particularly hard for the "it's fast and easy!" hype too hard and end up accidentally writing a pessimal schema because you thought using a non-relational database meant you got to think less about your schema than a relational DB.

That is precisely backwards; NoSQL-type DBs get their power from you spending a lot more time and care in thinking about exactly how you plan on accessing data. Many NoSQL databases loosen the constraints on what you can store in a given record, but in return they are a great deal more fussy about how you access records. If you want to skip careful design of how you access records, you want the relational DB. And nowadays, tossing a JSON field into a relational row is quite cheap and effective for those "catch alls" in the schema.

There's some interesting hybrids out there now if you want a bit of both worlds. For instance, Clickhouse is not an SQL database, but it more gracefully handles a lot of SQL-esque workloads than many other NoSQL-esque databases. You can get much farther with "I need a NoSQL-style database, but every once in a while I need an SQL-like bit of functionality", than you can in something like Cassandra.

Like Cassandra, dynamodb requires the data model to be designed very carefully to be able to get the max out of them.

More often than not, that simply adds more complexity; people often underestimate how much a sharded mysql/Postgres can scale.

My default choice for the longest time: Postgres for the data I care about, ES as secondary index and S3 as blob storage.

The key concept in Dynamo is that you use a partition key on all your bits of data (my mental model is that you get one server per partition) and you then can arrange data using a sort key in that partition. You can then range/inequality query over the sort keys. That’s the gist of it.

The power and scalability comes from the fact that each partition can be individually allocated and scaled, so as long as you spread over partitions you have practically no limits.

And you can do quite a bit with that sort key range/inequality thing. I was pleasantly surprised by how much of Redis I could implement: https://github.com/dbProjectRED/redimo.go