Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The DynamoDB Paper

brooker.co.za

Post reply on HN
Print view View original

81–90 of 95 posts

Re: The DynamoDB Paper

#81

Post by coredog64 »

0xbadcafebee wrote: post #77
mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

Can be performant, nowadays anyway. Worked with a team who built their own implementation because Amazon's was too slow and expensive. It's a weird model. Too small of a dataset and it doesn't quite make sense to use Dynamo. Too big of a dataset and it's full of footguns. Medium-sized may be too expensive.

Too-small seems to be the perfect use case for DDB. I need someplace to stash stuff and look it up by key. A full RDS is overkill, as is anything else that requires nodes that charge by the hour.

Re: The DynamoDB Paper

#82

Post by coredog64 »

revicon wrote: post #47

One big benefit of DynamoDB over RDS on AWS is that the access layer is API based so you don’t have issues with held open connections when accessing via AWS Lambda.

RDS proxy should fix this, but the proxy team is out of sync with the RDS team. I’ve seen RDS ahead of proxy by two major versions.

Re: The DynamoDB Paper

#83

Post by hintymad »

ignoramous wrote: post #50

> From the paper [0]: DynamoDB consists of tens of microservices. Ha! For folks who think two-pizza teams mean 100s of microservices... this is probably the second most scaled-out storage service at AWS (behind S3?), and it runs tens of microservices (pretty sure these aren't micro the way most folks would presume 'em to be). > What's exciting for me about this paper is that it covers DynamoDB's journey... Assuming t…

I'm not sure about DDB, but I know in AWS in general building a new service does not give you credit by default. It's not like the shit Uber promoted: Yeh! We have 8000 services. Look how great we are! In fact, people usually question if someone proposes to create a new service. Working Backwards (i.e., solving real user problems) and Invent and Simplify are indeed two powerful leadership principles. And of course, the sheer amount of work involved in setting up a new service is so much that people have to think twice between starting a new service.

Re: The DynamoDB Paper

#84

Post by alexnewman »

bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

totally, or s3

Re: The DynamoDB Paper

#85

Post by rmbyrro »

deanCommie wrote: post #66
ignoramous wrote: post #50

> From the paper [0]: DynamoDB consists of tens of microservices. Ha! For folks who think two-pizza teams mean 100s of microservices... this is probably the second most scaled-out storage service at AWS (behind S3?), and it runs tens of microservices (pretty sure these aren't micro the way most folks would presume 'em to be). > What's exciting for me about this paper is that it covers DynamoDB's journey... Assuming t…

Lots can change over the years. Your links are from 2016 - it's not conceivable that in the last 6 years, Amazon has changed some of the implementation?

DynamoDB was already large scale at that time.

The point is: the number of services don't need to scale with the level of demand.

Re: The DynamoDB Paper

#86

Post by manigandham »

dalyons wrote: post #63
manigandham wrote: post #14

Earlier quoted context omitted.

All databases scale the same way - by partitioning and sharding the dataspace. RDBMS have harder restrictions due to the features they provide and the performance expectations, but you can just as easily use a bunch of relational servers to partition a table (or several) across them by range or hashes of the primary key. That's basically what key/value stores like DynamoDB do, and why DynamoDB was even built on MySQL…

"can just as easily use a bunch of relational servers to partition a table" is not true at all. Managing, maintaining and tuning a sharded relational cluster is an astonishing amount of operations work. partition management, re-partioning, partition failover / promotions / demotions, query routing, shard discovery, upgrades... it goes on an on. All this work is gone if you pick dynamo. Not saying that dynamo is alway…

The point is the scaling fundamentals are the same across databases.

Whether that work is managed or not is a different topic, and you can find plenty of managed offerings of scale-out relational databases.

Re: The DynamoDB Paper

#87

Post by deanCommie »

rmbyrro wrote: post #85
deanCommie wrote: post #66

Earlier quoted context omitted.

Lots can change over the years. Your links are from 2016 - it's not conceivable that in the last 6 years, Amazon has changed some of the implementation?

DynamoDB was already large scale at that time. The point is: the number of services don't need to scale with the level of demand.

My comment was about " paper fails to acknowledge a FOSS database (once?) underneath it: MySQL/InnoDB (and references it as B-Tree instead)." should have been more clear

Re: The DynamoDB Paper

#88

Post by jerryjerryjerry »

manigandham wrote: post #56
jerryjerryjerry wrote: post #41

Good job! But I'm wondering when Amazon can start to contribute to open source world...

They already do: https://aws.amazon.com/opensource/

ohh you are right, sorry Amazon, I didn't notice this. But I'm still hoping to see your contribution to system kernels of databases, bigdata, etc.

Re: The DynamoDB Paper

#89

Post by bluehatbrit »

0xthrowaway wrote: post #67

DynamoDB is (edit: can be) extremely expensive compared to alternatives (e.g. self hosted SQL). Make sure the benifits (performance, managed, scale) outweigh the costs!

I'd put emphasis on the "can be", it very much depends on your read/write patterns and configuration. Assuming you know those up front and they fit dynamo well it can be several times cheaper than any sort of SQL.

Every time I've made use of it it's ended up costing pennies compared to what SQL would cost, sometimes literal pennies. If you turn on all the fancy features from day 1 and fill it with tons of data you don't need and make too many reads/writes per-request though you can get into very pricey territory very quickly.

We tried to aim for 1 read and/or 1 write per request to our service and that worked really well for our use cases. It kept costs low and performance high but we had a really well understood problem. If I was a startup and didn't know quite how my product would turn out, I don't think I'd consider dynamo for a while.

Re: The DynamoDB Paper

#90

Post by deepGem »

manigandham wrote: post #55
deepGem wrote: post #39

Earlier quoted context omitted.

True. Spanner and the likes of Spanner, CockroachDB, YugaByte all are strongly consistent and scalable dbs. The greatest advantage IMO is the ability to just use SQL without having to worry about carefully designing a data model. What bothers me however is that these data stores are not truly relational data stores. They spin a relational layer on top of a scalable key-value data store. Is it necessary to use a stron…

What part of SQL requires not having to design a data model? What exactly do you mean by that? And technically all relational databases are relational layers on top of a key/value subsystem. Splitting that apart and scaling the storage is how most of the NewSQL databases scale , from CRDB to Yugabyte to Neon.

What I mean is that the data model in the NoSQL world is tightly coupled with the query pattern. So you define the query pattern and then tailor the data model to that query pattern. In the relational world, you typically choose the index based on your query pattern. Not tailor the data model to your query pattern. You follow normalization principles.

Of course, every data store needs a data model. No debate there :).

Not sure what you mean by relational databases are relational layers on top of key-value stores. InnoDB has a 16KB page data as it's fundamental data structure.

https://dev.mysql.com/doc/internals/en/innodb-page-structure...

Post reply on HN