Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DynamoDB 10 years later

amazon.science

Post reply on HN
Print view View original

81–90 of 225 posts

Re: DynamoDB 10 years later

#81

Post by vp8989 »

salil999 wrote: post #65

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintain…

"And way too many operational issues."

I've seen this kind of thing mentioned many times, pretty baffling TBH based on Dynamo's pretty good reputation in industry. Are these mostly to the stateless components of the product, or do they see data loss?

Re: DynamoDB 10 years later

#82

Post by indogooner »

vasili111 wrote: post #68
salil999 wrote: post #65

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintain…

>Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There ar…

It sounds incredulous but I have heard similar things about Oracle. May be a large dev team can duct tape enough so that the product is solid.

Re: DynamoDB 10 years later

#83

Post by gigatexal »

awsthro00945 wrote: post #77
gigatexal wrote: post #28

Earlier quoted context omitted.

At his rank at AWS I don’t know if money was such an issue. He strikes me as a person who cares deeply about the underlying tech. But I have no idea one way or the other.

I think I've seen you post something similar on r/aws about how Rick was "top DynamoDb person at AWS" (apologies if that wasn't you). I think you are overestimating Rick's "rank". I just looked him up (I had not heard of him before seeing his name mentioned on r/aws a few days ago) and he was an L7 TPM/Practice Manager in AWS's sales organization. That's not really a notably high position, and in the grand scheme of…

was not me at r/aws

unless he posts here about it we can't really know -- we can only speculate but I think he had a higher amount of influence than his title/rank might suggest. I think Rick's influence with respect to DynamoDB is akin to that of Kelsey Hightower's influence over k8s at Google.

Re: DynamoDB 10 years later

#84

Post by jvanvleet »

manishsharan wrote: post #53

Quite a few of the teams that were early adopters of AWS DynamoDB were not prepared for the pricing nuances that had to be taken into consideration when building their solutions.

I was one of these. However I now understand that the pricing nuances reflected a reality that I appreciate. We used DDB in a way that was not the best fit and the cost was a reflection of this.

Re: DynamoDB 10 years later

#85

Post by jgraettinger1 »

jerf wrote: post #59
redwood wrote: post #18

Earlier quoted context omitted.

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

It seems to me that what this is saying is that storage has become so cheap that if another database provides even slight advantages over another for some workload it is likely to be deployed and have all the data copied over to it. HN entrepreneurs take note, this also suggests to me that there may be a market for a database (or a "metadatabase") that takes care of this for you. I'd love to be able to have a "relati…

> I'd love to be able to have a "relational database" that is also some "NoSQL" databases (since there's a few major useful paradigms there) that just takes care of this for me. I imagine I'd have to declare my schemas, but I'd love it if that's all I had to do and then the DB handled keeping sync and such.

You might be interested in what we're building [0]

It synchronizes your data systems so that, for example, you can CDC tables from your Postgres DB, transform them in interesting ways, and then materialize the result in a view within Elastic or DynamoDB that updates continuously and with millisecond latency.

It will even propagate your sourced SQL schemas into JSON schemas, and from there to, say, equivalent Elastic Search schema.

[0]: https://github.com/estuary/flow

Re: DynamoDB 10 years later

#86

Post by ignoramous »

manigandham wrote: post #31
victor106 wrote: post #8

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

DynamoDB (and other dynamo-like systems like Cassandra, Bigtable) are just advanced key/value stores. They support multiple levels of keys->values but fundamentally you need the key to find the associated value. If you want to search by parameters that aren't keys then you need to store your data that way. Most of these systems have secondary indexes now, and that's basically what they do for you automatically in the…

> If you want to search by parameters that aren't keys then you need to store your data that way.

Not that I recommend it, but by using space-filling curves, one could to index multiple dimensions onto DynamoDB's bi-dimensional (hash-key, range-key) primary-index: https://aws.amazon.com/blogs/database/z-order-indexing-for-m... and https://web.archive.org/web/20220120151929/https://citeseerx...

Re: DynamoDB 10 years later

#88

Post by artembugara »

We realized how great Dynamo was only after we migrated off AWS.

Dynamo was a key factor to us when we were releasing the MVP of our News API [0]. We used Dynamo, ElasticSearch, Lambda and could make it running in 60 days while being full-time employed.

Also, the best tech talk I saw was given by Rick Houlihan on re:Invent [1]

I highly recommend every engineer to watch it: it's a great overview of SQL vs NoSQL

[0] https://newscatcherapi.com/blog/how-we-built-a-news-api-beta...

[1] https://www.youtube.com/watch?v=HaEPXoXVf2k

Re: DynamoDB 10 years later

#89

Post by qvrjuec »

whalesalad wrote: post #58
afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

Dynamo is incredibly hard to use correctly I’d urge you to start writing a prototype, a lot of your assumptions might get thrown out the window. Dynamo is not necessarily good for reading high volume. You’ll end up needing to use a parallel scan approach which is not fast.

I'd say Dynamo is extremely good at reading high volume, with the appropriate access pattern. It's very efficient at retrieving huge amounts of well partitioned data using the data's keys, but scanning isn't so efficient.

Re: DynamoDB 10 years later

#90

Post by neuronexmachina »

grncdr wrote: post #80
jerf wrote: post #59

Earlier quoted context omitted.

It seems to me that what this is saying is that storage has become so cheap that if another database provides even slight advantages over another for some workload it is likely to be deployed and have all the data copied over to it. HN entrepreneurs take note, this also suggests to me that there may be a market for a database (or a "metadatabase") that takes care of this for you. I'd love to be able to have a "relati…

I think there was a project like this a few years ago (wrapping a relational DB + ElasticSearch into one box) and I thought it was CrateDB, but from looking at their current website I think I'm misremembering. The concept didn't appeal to me very much then, so I never looked into it further. --- To address your larger point, I think Postgres has a better chance of absorbing other datastores (via FDW and/or custom ind…

Were you thinking of ZomboDB? https://github.com/zombodb/zombodb
Post reply on HN