Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DynamoDB 10 years later

amazon.science

Post reply on HN
Print view View original

61–70 of 225 posts

Re: DynamoDB 10 years later

#61

Post by 300bps »

sebastialonso wrote: post #55
redwood wrote: post #18

Earlier quoted context omitted.

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

> they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable querying, and then they bolt on Redis to make the disjointed backend feel fast. This made my head explode. Why would you explicitly join two systems made to solve different issues together? This sounds rather like a lack of architectural vision. Postgre…

In my experience, leave DynamoDB alone and it will work great.

My experience agrees with yours and I'm likewise puzzled by the grandparent comment. But just a shout out to DAX (DyanmoDB Accelerator) which makes it scale through the roof:

https://aws.amazon.com/dynamodb/dax/

Re: DynamoDB 10 years later

#62

Post by _pdp_ »

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

DynamoDB is like S3 but with query features. It is not a relational db. It is a document storage. So you need to use it for what it is.

Our entire solution is basically based on top of lambda and dynamodb tables and it works really as long as you don't threat the tables like SQL.

Re: DynamoDB 10 years later

#63

Post by Keyframe »

urthor wrote: post #49
nine_k wrote: post #22

Earlier quoted context omitted.

If this is not the only option, what would you suggest instead? How to simplify it?

The alternative is to go to GCP and use the big GCP selling point, which is Big Table/Big Query. Those databases build most of that in, and it's all one fairly excellent distributed monolith.

It's still a marriage.

Re: DynamoDB 10 years later

#64

Post by andy_ppp »

jerf wrote: post #59
redwood wrote: post #18

Earlier quoted context omitted.

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

It seems to me that what this is saying is that storage has become so cheap that if another database provides even slight advantages over another for some workload it is likely to be deployed and have all the data copied over to it. HN entrepreneurs take note, this also suggests to me that there may be a market for a database (or a "metadatabase") that takes care of this for you. I'd love to be able to have a "relati…

Postgres with Cassandra built in and scaled separately would be really great.

Re: DynamoDB 10 years later

#65

Post by salil999 »

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there.

Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :)

Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are)

I worked at DynamoDB for over 2 years.

Re: DynamoDB 10 years later

#66

Post by mjb »

tybit wrote: post #11

For anyone else expecting this to be a paper given the domain name, it’s not. It’s a non technical interview with a couple of the original papers authors. Not bad, just not as exciting as I imagine a paper detailing what they’ve learnt from a distributed systems perspective etc operating Dynamo then DynamoDB for so long now.

We don't have a paper on DynamoDB's internals (yet?), but here's a talk you might find interesting from one of the folks who built and ran DDB for a long time: https://www.youtube.com/watch?v=yvBR71D0nAQ

And Doug Terry talking through the details of how DynamoDB's transaction protocol works: https://www.usenix.org/conference/fast19/presentation/terry

If we did publish more about the internals of DDB, what would you be looking to learn? Architecture? Operational experience? Developer experience? There's a lot of material we could share, and it's useful to hear where people would like us to focus.

Re: DynamoDB 10 years later

#67

Post by mythrwy »

whalesalad wrote: post #58
afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

Dynamo is incredibly hard to use correctly I’d urge you to start writing a prototype, a lot of your assumptions might get thrown out the window. Dynamo is not necessarily good for reading high volume. You’ll end up needing to use a parallel scan approach which is not fast.

Also can be _very_ expensive if you do not use it correctly.

Re: DynamoDB 10 years later

#68

Post by vasili111 »

salil999 wrote: post #65

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintain…

>Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :)

>Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are)

How those two can coexist?

Re: DynamoDB 10 years later

#69

Post by awsthro00945 »

redwood wrote: post #18
afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

>No one uses DynamoDB alone

Almost every single team at Amazon that I can think of off the top of my head uses DynamoDB (or DDB + S3) as its sole data store. I know that there are teams out there using relational DBs as well (especially in analytics), but in my day-to-day working with a constantly changing variety of teams that run customer-facing apps, I haven't seen RDS/Redis/etc being used in months.

Re: DynamoDB 10 years later

#70

Post by xtracto »

manishsharan wrote: post #53

Quite a few of the teams that were early adopters of AWS DynamoDB were not prepared for the pricing nuances that had to be taken into consideration when building their solutions.

I remember trying Dynamodb around 2015/2016: You had to specify your expected read and write throughout and you would be billed for that. At that time we had a pretty spikey traffic use case which made using dynamodb efficiently impossible
Post reply on HN