We landed on DynamoDB when we migrated a monolith to microservice architecture. I have to say that DynamoDB fits fairly well in the microservices world where the service scope is small, query patterns are pretty narrow and don't really change much. Building new things using DynamoDB when query patterns aren't necessarily known is very painful and require tedious migration strategies unless you don't mind paying for GSIs.
DynamoDB 10 years later
131–140 of 225 posts
Re: DynamoDB 10 years later
#132Earlier quoted context omitted.
We don't have a paper on DynamoDB's internals (yet?), but here's a talk you might find interesting from one of the folks who built and ran DDB for a long time: https://www.youtube.com/watch?v=yvBR71D0nAQ And Doug Terry talking through the details of how DynamoDB's transaction protocol works: https://www.usenix.org/conference/fast19/presentation/terry If we did publish more about the internals of DDB, what would you b…
All of it - architecture, operational experience, best practices etc.
Just want to second this. All of the above sounds really interesting to me!
Re: DynamoDB 10 years later
#133We realized how great Dynamo was only after we migrated off AWS. Dynamo was a key factor to us when we were releasing the MVP of our News API [0]. We used Dynamo, ElasticSearch, Lambda and could make it running in 60 days while being full-time employed. Also, the best tech talk I saw was given by Rick Houlihan on re:Invent [1] I highly recommend every engineer to watch it: it's a great overview of SQL vs NoSQL [0] ht…
BTW Rick Houlihan left AWS recently to work for Mongo.
https://twitter.com/houlihan_rick/status/1472969503575265283
On that thread he criticizes AWS regarding DynamoDB openly.
> I will always love DynamoDB, but the fact is it is losing ground fast because AWS focuses most of their resources on the half baked #builtfornopurpose database strategy. I always hated that idea, I just bit my tongue instead of saying it.
> The problem is the other half-baked database services that all compete for the same business. DocumentDB, Keyspaces, Timestream, Neptune, etc. Databases take decades to optimize, the idea that you can pump them out like web apps is silly.
> I was very tired of explaining over and over again that DynamoDB is actually not the dumbed down Key-Value store that the marketing message implied. When AWS created 6 different NoSQL databases they had to make up reasons for each one and the messaging makes no sense.
Re: DynamoDB 10 years later
#134I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintain…
So they just rolled out global replication, and I can't for the life of me figure out how they resolve write conflicts without cell timestamps or any other obvious CRDT measures.
Questions were handwaved away, and the usual Amazon black box non-answers which always smells like they are hiding problems.
Any ideas how this is working? It seems bolt-on and not well thought out, and I doubt they'll ever pay for Aphyr to put it through his torture tests.
Re: DynamoDB 10 years later
#135I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintain…
I worked at a company who re-implemented the entire Dynamo paper and API, and it was exactly the same story. Completely eliminated all my illusions about the supposed superiority of distributed systems. It was a mound of tires held together with duct tape, with a tiki torch in each tire.
Did they have a spare 100 million hanging around to burn? That seems pretty ridiculous. Why did they not just run cassandra?
Re: DynamoDB 10 years later
#136We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
The salespeople always promise magic and handwave CAP away.
But data at scale is about:
1) knowing your queries ahead of time (since you've presumably reached the limit of PG/maybesql/o-rackle.
2) dealing with CAP at the application level: distributed transactions, eventual consistency, network partitions.
3) dealing with a lot more operational complexity, not less.
So if the snake oil salesmen say it will be seamless, they are very very very much lying. Either that, or you are paying a LOT of money for other people to do the hard work.
Which is what happens with managing your own NoSQL vs DynamoDB. You'll pay through the roof for DynamoDB at true big data scales.
Re: DynamoDB 10 years later
#137To be honest, as a customer, it is hard for me to justify using DynamoDB. Some of this criticism can be out of date: 1. DynamoDB is not as convenient. There are a bit too many dials to turn. 2. DynamoDB does not have a SQL facade on top. 3. DynamoDB is proprietary, I believe there's no OSS API equivalent if you want to migrate out. 4. DynamoDB was kind of expensive. But it has been a while since I last check the pric…
I strongly agree that most early stage businesses should be on Postgres. There's simply too much churn in early stage data models. Also, unforeseen esoteric needs jumping out of the wood work that you can knock out a SQL query for instead of having to build a solution come up constantly. However, this does assume that your development team has a competent understanding of SQL.
I've been in a couple startups that went Dynamo first and development velocity was a pale shadow of velocity with Postgres. When one of those startups dumped dynamo for Postgres velocity multiplied immediately. I'd estimate we were moving at around 1000% and the complete transition took less time than even I expected (about a month). Once the business matures, moving tables onto dynamo and wrapping them in a microservice makes a lot of sense. Dynamo does solve a lot of problems that become increasingly material as the business evolves.
Eventually, SQL's presence declines and transitions into an analytics system as narrower, but easier for ops, options proliferate.
Re: DynamoDB 10 years later
#138Can anyone recommend me a good paper (or other resource) describing the designs of DynamoDB and S3? Ideally something in the spirt of the original Dynamo paper (i.e. https://www.allthingsdistributed.com/files/amazon-dynamo-sos... )
Re: DynamoDB 10 years later
#139Can anyone recommend me a good paper (or other resource) describing the designs of DynamoDB and S3? Ideally something in the spirt of the original Dynamo paper (i.e. https://www.allthingsdistributed.com/files/amazon-dynamo-sos... )
This one is pretty good for DynamoDB: https://youtu.be/yvBR71D0nAQ
Re: DynamoDB 10 years later
#140We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
If it’s possible in your situation, instead of vendor lock-in, invest in cacheability of your service and leverage HTTP cache as much as possible.