DynamoDB 10 years later • Hacker News Board

Re: DynamoDB 10 years later

#111

Post by rmbyrro » Thu, Jan 20, 2022, 4:39 PM UTC

picardo wrote: post #46

ndm000 wrote: post #34

Earlier quoted context omitted.

> they bolt it onto Postgres I am working with a company that is redesigning an enterprise transactional system, currently backed by an Oracle database with 3000 tables. It’s B2B so loads are predictable and are expected to grow no more than 10% per year. They want to use DynamoDB as their primary data store, with Postgres for edge cases it seems to me the opposite would be more beneficial. At what point does DynamoD…

You can make Postgres scale, but there is an operational cost to it. DynamoDB does that for you out of the box. (So does Aurora, to be honest, but there is also an overhead to setting up an Aurora cluster to the needs of your business.) I've found also that in Postgres the query performance does not keep up with bursts of traffic -- you need to overprovision your db servers to cope with the highest traffic days. Dyna…

Using +1 DynamoDB table is a bad idea in the first place.

Re: DynamoDB 10 years later

#112

Post by rmbyrro » Thu, Jan 20, 2022, 4:41 PM UTC

vosper wrote: post #38

ndm000 wrote: post #34

Earlier quoted context omitted.

> they bolt it onto Postgres I am working with a company that is redesigning an enterprise transactional system, currently backed by an Oracle database with 3000 tables. It’s B2B so loads are predictable and are expected to grow no more than 10% per year. They want to use DynamoDB as their primary data store, with Postgres for edge cases it seems to me the opposite would be more beneficial. At what point does DynamoD…

They should be looking at Aurora, not Dynamo. Using Dynamo as the primary store for relational data (3000 tables!) sounds like an awful idea to me. I’d rather stay on Oracle. https://aws.amazon.com/rds/aurora/?aurora-whats-new.sort-by=...

It really depends much more on the access patterns than data shape.

Certain access patterns can do pretty well with 3,000 relational tables denormalized to a single DynamoDB table.

Re: DynamoDB 10 years later

#113

Post by goostavos » Thu, Jan 20, 2022, 4:43 PM UTC

awsthro00945 wrote: post #69

redwood wrote: post #18

Earlier quoted context omitted.

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

>No one uses DynamoDB alone Almost every single team at Amazon that I can think of off the top of my head uses DynamoDB (or DDB + S3) as its sole data store. I know that there are teams out there using relational DBs as well (especially in analytics), but in my day-to-day working with a constantly changing variety of teams that run customer-facing apps, I haven't seen RDS/Redis/etc being used in months.

The thing about Amazon is that it is massive. In my neck of the woods, I've got the complete opposite experience. So many teams have the exact DDB induced infrastructure sprawl as described by the GP (e.g. supplemental RDBMS, Elastic, caching layers, etc..).

Which says nothing of DDB. It's an god-tier tool if what you need matches what it's selling. However, I see too many teams reach for it by default without doing any actual analysis (including young me!), thus leading to the "oh shit, how will we...?" soup of ad-hoc supporting infra. Big machines look great on the promo-doc tho. So, I don't expect it to stop.

Re: DynamoDB 10 years later

#114

Post by rmbyrro » Thu, Jan 20, 2022, 4:43 PM UTC

jerf wrote: post #59

redwood wrote: post #18

Earlier quoted context omitted.

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

It seems to me that what this is saying is that storage has become so cheap that if another database provides even slight advantages over another for some workload it is likely to be deployed and have all the data copied over to it. HN entrepreneurs take note, this also suggests to me that there may be a market for a database (or a "metadatabase") that takes care of this for you. I'd love to be able to have a "relati…

I'm afraid it's not feasible to develop a single general purpose implementation for that.

The amount of complexity to guarantee data integrity while covering all possible use cases will be just unmanageable.

I'd be extremely happy to be proven wrong, though...

Re: DynamoDB 10 years later

#115

Post by notyourwork » Thu, Jan 20, 2022, 4:45 PM UTC

geodel wrote: post #110

salil999 wrote: post #65

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintain…

Very interesting! To me the takeaway is large/interesting/challenging engineering projects are pretty close to disasters generally. Some time they do become disaster actually. On the other hand if a project looks like straight up designed, neatly put into JIRA stories, and developers deliver code consistently week after week then it may be a successfully planned and delivered project. But it would mostly be doing stu…

Challenging the cutting edge of your product domain is what I get from this. Easy things are easy and predictable. Hard things and unpredictable evolving requirements are a tension against the initial system design which is the foundation of your code base. Over time the larger projects get the perhaps further they deviate from the original design. If you could predict it up front in many cases its not all that interesting or challenging of a problem. Duct tape is fine to use as long as you understand when you've gone too far and might want to re-design from scratch based on prior learnings.

Re: DynamoDB 10 years later

#116

Post by afandian » Thu, Jan 20, 2022, 4:47 PM UTC

phamilton wrote: post #108

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

I don't know your scaling needs, but I would highly recommend just using Aurora postgresql for read-only workloads. We have some workloads that are essentially K/V store lookups that were previously slated for dynamodb. On an Aurora cluster of 3*r6g.xlarge we easily handle 25k qps with p99 in the single-digit ms range. Aurora can scale up to 15 instances and up to 24xlarge, so it would not be unreasonable to see 100x…

Thanks, that's what I hope will work. I might drop you a mail at some point.

Re: DynamoDB 10 years later

#117

Post by ddoolin » Thu, Jan 20, 2022, 4:50 PM UTC

As a developer, I really have a love-hate relationship with Dynamo. I love how fast and easy it is to setup and get rolling.

The partitioning scheme came off as confusing and opaque but I think that says more about Amazon's documentation than the scheme itself.

I do not like that there's no really third party tooling integration to be able to query. Their UI in the console is _so freaking terrible_ yet you have no other way than code to query it. This problem is so bad that I will avoid using it where I can despite it being a good option, performance-wise.

Re: DynamoDB 10 years later

#118

Post by rmbyrro » Thu, Jan 20, 2022, 4:50 PM UTC

uberdru wrote: post #102

Marazan wrote: post #39

Earlier quoted context omitted.

DyanmoDB is pretty much the opposite of magic. It is a resource that can often be the right tool for the job but you really have to understand what the job is and carefully measure Dynamo up for what you are doing. It is _easy_ to misunderstand or miss something that would make Dynamo hideously expensive for your use case.

What use cases would likely make it hideously expensive, in your view? Like, what are the red flags?

Hot keys are much lesser of an issue nowadays. It'd been a big one in old DDB architectures.

I'd say requiring scans or filters as opposed to queries is one of the biggest issues that can bite your pocket.

Think carefully about how you'll access your data later. You won't be able to change it drastically and cheaply later.

Re: DynamoDB 10 years later

#119

Post by giaour » Thu, Jan 20, 2022, 4:50 PM UTC

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

If you use the "pay per request" billing model instead of provisioned throughput, DynamoDB scaling is self-managing, and you can treat your DB as a fire-and-forget key/value store. You need to plan how you'll query your data and structure the keys accordingly, but honestly, that applies even more to S3 than it does to Dynamo.

Re: DynamoDB 10 years later

#120

Post by rmbyrro » Thu, Jan 20, 2022, 4:51 PM UTC

jugg1es wrote: post #48

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

I do not recommend starting off with a decision to use DynamoDB before you have worked with it directly for some time to understand it. You could spend months trying to shoehorn your use case into it before realizing you made a mistake. That said, DynamoDB can be incredibly powerful and inexpensive tool if used right.

I think this can be said about any technology, really...