Good job! But I'm wondering when Amazon can start to contribute to open source world...
The DynamoDB Paper
41–50 of 95 posts
Re: The DynamoDB Paper
#42Earlier quoted context omitted.
DynamoDB is amazing, but not very flexible once you have designed your database. No abstraction layer will allow you to run queries ad-hoc in a performant way.
> No abstraction layer will allow you to run queries ad-hoc in a performant way. Depends on the size of the data. Run analytics queries (i.e. things that return summary data not all rows) on 10GB of data through clickhouse or duckdb or datafusion and they'll generally return in milliseconds.
What does this have to do with DynamoDB? The point is that once you've gotten your data into DynamoDB, you're strongly limited in how you can use it until you load it into something else.
Re: The DynamoDB Paper
#43Earlier quoted context omitted.
> No abstraction layer will allow you to run queries ad-hoc in a performant way. Depends on the size of the data. Run analytics queries (i.e. things that return summary data not all rows) on 10GB of data through clickhouse or duckdb or datafusion and they'll generally return in milliseconds.
What does this have to do with DynamoDB? The point is that once you've gotten your data into DynamoDB, you're strongly limited in how you can use it until you load it into something else.
I didn't see an obvious connection between the two sentences.
Re: The DynamoDB Paper
#44These days I’d probable take a closer look at spanner. It is a consistent and scalable db. It makes life much easier for developers. Like Cassandra, dynamodb requires the data model to be designed very carefully to be able to get the max out of them. More often than not, that simply adds more complexity; people often underestimate how much a sharded mysql/Postgres can scale. My default choice for the longest time: Po…
Global strict serializability is coming to Cassandra very soon [1]
[1] https://cwiki.apache.org/confluence/download/attachments/188...
Re: The DynamoDB Paper
#45Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…
I also recommend Alex DeBrie's "The DynamoDB Book" ( https://www.dynamodbbook.com/ ). It is a great resource that talks about these design patterns in depth. It has served me and my team well over the past few years.
Seconded! Alex DeBrie is a great teacher.
Re: The DynamoDB Paper
#46Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…
It's worth noting that a lot of the early database designs, including this 2018 video pre-date some dramatic improvements to dynamodb usability.
I think the biggest ones were:
- an increase in the number of GSIs you can create (Dec 2018) [1]
- making on-demand possible [2]
- an increase in the default limit for number of tables you can create (Mar 2022) [3]
I don't think these new features necessarily make the single-table, overloaded GSI strategy that's discussed in the video obsolete, but they enable applications which are growing to adopt an incremental GSI approach and use multiple tables as their data access patterns mature.
Some other posters have recommended Alex DeBrie's dynamodb book and I also think that's an excellent resource, but I'd caution people who are getting into dynamodb not to be scared by the claims that dynamodb is inflexible to data access changes, since AWS has been adding a lot of functionality to support multi-table, unknown access patterns, emerging secondary indexes, etc.
- [1] https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-dy...
- [2] https://aws.amazon.com/blogs/aws/amazon-dynamodb-on-demand-n...
- [3] https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-dy...
Re: The DynamoDB Paper
#47One big benefit of DynamoDB over RDS on AWS is that the access layer is API based so you don’t have issues with held open connections when accessing via AWS Lambda.
Re: The DynamoDB Paper
#48Earlier quoted context omitted.
I think it also depends on the system you’re using it on. I think one of the biggest advantages of DDB is that is scales so well (with good design to avoid hot partitions). Afaik, RDBMS simply cannot scale in the same way due to their design. Yes, they can scale somewhat, but as you said it requires lots of tuning, and you’ll still reach a hardish limit.
All databases scale the same way - by partitioning and sharding the dataspace. RDBMS have harder restrictions due to the features they provide and the performance expectations, but you can just as easily use a bunch of relational servers to partition a table (or several) across them by range or hashes of the primary key. That's basically what key/value stores like DynamoDB do, and why DynamoDB was even built on MySQL…
"just as easily" would be the contested part, I'd guess
Re: The DynamoDB Paper
#49How well does DyanmoDB scale when paired with AppSync and GraphQL? The selling point here being you can use GQL as your schema for the DB too and get automatic APIs for free
Just fine?
I should have made it clear: I was hoping to get some folks to talk to their experience using it this way. I haven't find alot in terms of real world evaluation of it.
It can also use Aurora Serverless V2, and I am curious about that as well, FWIW
Re: The DynamoDB Paper
#50> From the paper [0]: DynamoDB consists of tens of microservices.
Ha! For folks who think two-pizza teams mean 100s of microservices... this is probably the second most scaled-out storage service at AWS (behind S3?), and it runs tens of microservices (pretty sure these aren't micro the way most folks would presume 'em to be).
> What's exciting for me about this paper is that it covers DynamoDB's journey...
Assuming these comments are true [1][2], in a classic Amazon fashion [3], the paper fails to acknowledge a FOSS database (once?) underneath it: MySQL/InnoDB (and references it as B-Tree instead).
[0] https://web.archive.org/web/20220712155558/https://www.useni...
[1] https://news.ycombinator.com/item?id=13173927