Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The DynamoDB Paper

brooker.co.za

Post reply on HN
Print view View original

61–70 of 95 posts

Re: The DynamoDB Paper

#61

Post by jd_mongodb »

deepGem wrote: post #39
Patrol8394 wrote: post #36

These days I’d probable take a closer look at spanner. It is a consistent and scalable db. It makes life much easier for developers. Like Cassandra, dynamodb requires the data model to be designed very carefully to be able to get the max out of them. More often than not, that simply adds more complexity; people often underestimate how much a sharded mysql/Postgres can scale. My default choice for the longest time: Po…

True. Spanner and the likes of Spanner, CockroachDB, YugaByte all are strongly consistent and scalable dbs. The greatest advantage IMO is the ability to just use SQL without having to worry about carefully designing a data model. What bothers me however is that these data stores are not truly relational data stores. They spin a relational layer on top of a scalable key-value data store. Is it necessary to use a stron…

You seem to think MongoDB is eventually consistent. MongoDB is designed as strongly consistent database. You can choose to query a secondary and that will be eventually consistent but that is not the default behaviour.

Re: The DynamoDB Paper

#62

Post by dboreham »

deepGem wrote: post #39
Patrol8394 wrote: post #36

These days I’d probable take a closer look at spanner. It is a consistent and scalable db. It makes life much easier for developers. Like Cassandra, dynamodb requires the data model to be designed very carefully to be able to get the max out of them. More often than not, that simply adds more complexity; people often underestimate how much a sharded mysql/Postgres can scale. My default choice for the longest time: Po…

True. Spanner and the likes of Spanner, CockroachDB, YugaByte all are strongly consistent and scalable dbs. The greatest advantage IMO is the ability to just use SQL without having to worry about carefully designing a data model. What bothers me however is that these data stores are not truly relational data stores. They spin a relational layer on top of a scalable key-value data store. Is it necessary to use a stron…

> What bothers me however is that these data stores are not truly relational data stores

Suggests there may be an impossibility theory lurking somewhere.

Re: The DynamoDB Paper

#63

Post by dalyons »

manigandham wrote: post #14
fubbyy wrote: post #5

Earlier quoted context omitted.

I think it also depends on the system you’re using it on. I think one of the biggest advantages of DDB is that is scales so well (with good design to avoid hot partitions). Afaik, RDBMS simply cannot scale in the same way due to their design. Yes, they can scale somewhat, but as you said it requires lots of tuning, and you’ll still reach a hardish limit.

All databases scale the same way - by partitioning and sharding the dataspace. RDBMS have harder restrictions due to the features they provide and the performance expectations, but you can just as easily use a bunch of relational servers to partition a table (or several) across them by range or hashes of the primary key. That's basically what key/value stores like DynamoDB do, and why DynamoDB was even built on MySQL…

"can just as easily use a bunch of relational servers to partition a table" is not true at all. Managing, maintaining and tuning a sharded relational cluster is an astonishing amount of operations work. partition management, re-partioning, partition failover / promotions / demotions, query routing, shard discovery, upgrades... it goes on an on. All this work is gone if you pick dynamo. Not saying that dynamo is always better, but IMHO people very much underestimate the ops cost of running a sharded relational cluster at scale.

Re: The DynamoDB Paper

#64

Post by david38 »

solatic wrote: post #59
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

> give so much stuff for free Interesting choice of words. Performance wise, sure. Money wise? I'm still waiting for a SQL database with pay-per-request pricing. The cost difference is enormous, particularly when you remember that you don't need to spend manpower managing the underlying hardware. Engineering tradeoffs are more complicated than only considering raw scalability performance and "I can run it myself on a…

Who manages hardware these days? Aurora works quite well.

Re: The DynamoDB Paper

#65

Post by ngc248 »

mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

Indeed with the GSI's etc you can implement a priority queue or store data in the order you want etc. Once you are clear on the access patterns of your app DynamoDB is amazing to model for and will scale with your app. But if you are not clear about your app's access patterns or need adhoc queries, then dynamoDB is not a good fit.

Re: The DynamoDB Paper

#66

Post by deanCommie »

ignoramous wrote: post #50

> From the paper [0]: DynamoDB consists of tens of microservices. Ha! For folks who think two-pizza teams mean 100s of microservices... this is probably the second most scaled-out storage service at AWS (behind S3?), and it runs tens of microservices (pretty sure these aren't micro the way most folks would presume 'em to be). > What's exciting for me about this paper is that it covers DynamoDB's journey... Assuming t…

Lots can change over the years. Your links are from 2016 - it's not conceivable that in the last 6 years, Amazon has changed some of the implementation?

Re: The DynamoDB Paper

#68

Post by shepherdjerred »

kumarvvr wrote: post #58
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

> Designing application specifically for DynamoDB will take _a lot_ of time and effort Disagree with this. Your team could think of it as a document database, and you can have utility libraries that filter and sort based on PK / SK combinations to provide a seamless experience.

If you want your DynamoDB table to scale well you'll have to put in a lot of upfront effort.

Re: The DynamoDB Paper

#69

Post by shepherdjerred »

whalesalad wrote: post #11
ciberado wrote: post #9

Earlier quoted context omitted.

DynamoDB is amazing, but not very flexible once you have designed your database. No abstraction layer will allow you to run queries ad-hoc in a performant way.

It’s true. 400kb max item size, too. 1mb max query size I believe. Good luck grabbing a shit load of data at once without a parallel scan. Dynamo is a precision tool and it’s great at those specific workloads but it’s not a one size fits all by any means.

400kb is the max item size, the pattern to get around that is to store objects in S3 and URLs/keys to those objects in DDB

Re: The DynamoDB Paper

#70

Post by 0xthrowaway »

solatic wrote: post #59
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

> give so much stuff for free Interesting choice of words. Performance wise, sure. Money wise? I'm still waiting for a SQL database with pay-per-request pricing. The cost difference is enormous, particularly when you remember that you don't need to spend manpower managing the underlying hardware. Engineering tradeoffs are more complicated than only considering raw scalability performance and "I can run it myself on a…

>Interesting choice of words. Performance wise, sure. Money wise? I'm still waiting for a SQL database with pay-per-request pricing. The cost difference is enormous, particularly when you remember that you don't need to spend manpower managing the underlying hardware.

I assume you're saying DynamoDB is less expensive than SQL because of pay-per-request.

Working on applications with a modest amount of data (a few TB over a few years) pay per request has been incredibly expensive even with scaled provisioning. I would much rather have an SQL database and pay for the server/s. Then I could afford a few more developers!

Post reply on HN