Live data from Hacker News

Distributed Transactions at Scale in Amazon DynamoDB (2023)

muratbuffalo.blogspot.com

31–40 of 62 posts

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#31
No, thank you. DynamoDB promotes a terrible dev workflow, and there’s no way around it. It’s like embedding S3 directly into your application code.

The client libraries are gigantic, and the documentation is misleading at times. Plus, Dynamo expects your access patterns to be static, which isn’t true most of the time. Hyperscaling is great, but many aren’t willing to give up everything else just for that.

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#32
post #7

Using DynamoDB in 2025 is such a weird proposition. Horrible dev experience, no decent clients/libs, complex pricing, weird scaling in/out mechanism, slow, it only works well for well defined use-cases.

2 times I have used DynamoDB and been extremely happy; - In a SAAS API service we used dynamodb to look up API keys and track their daily usage data. It is fast enough to look up k/v pairs (api key => key info). And also aggregate small sets (We'd sum up call counts for current month and check if the API key had enough credits). This meant that the API itself did not need our RDBMS to function. We also had a postgres…

> We also had a postgresql instance for all relational data, subscriptions, user info etc. Had a trigger that would push any api key / subscription change to DynamoDB.

Wouldn't doing it right there in postgres limit your footprint?

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#33

I came here for the bad takes, and I have not been disappointed. Dynamo slays when you know your access patterns and need consistent performance and no operations requirements. Turns out, that's the case most of the time. Think about it as application state instead of a db. It's not key-value like Redis. GSIs with compound keys allow access to data across multiple dimensions on virtually unlimited data with consisten…

> Turns out, that's the case most of the time.

Here's most of the time out in the real world:

- Low-cardinality partition key leading to hot keys, trashing capacity utilization.

- Bad key design means access patterns are off the table forever, as nobody wants to take on data migration with BatchWriteItem.

- Read/write spikes causing throttling errors. The capacity concept is difficult - people don't understand how capacity relates to partitions and object sizes, or wrongly assume "On-Demand Capacity" means throttling is impossible, or that Provisioned Capacity Autoscaling is instant.

- Multiple GSIs to cover multiple access patterns = "why is our bill so high?".

I've seen these issues over and over again while working with real organizations.

Of course it's impressive technology, it's just so littered with traps that I've stopped recommending it except in very specific cases.

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#34

I came here for the bad takes, and I have not been disappointed. Dynamo slays when you know your access patterns and need consistent performance and no operations requirements. Turns out, that's the case most of the time. Think about it as application state instead of a db. It's not key-value like Redis. GSIs with compound keys allow access to data across multiple dimensions on virtually unlimited data with consisten…

Agreed. It's wild to me how many people think they need arbitrary queries on their transactional database and then go write a CRUD app with no transactional consistency between resources and everything is a projection from a user or org resource -- you can easily model that with Dynamo. You can offload arbitrary analytical queries or searches to a different database and stop conflating that need with your app's core data source.

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#36
post #5

Earlier quoted context omitted.

Very often I find myself wanting to store item(s) using a key. My items are not relations, and I don't see the point in transforming them to and from relational form. And if I did, each row would have like 5 columns set to NULL, in addition to a catch-all string 'data' column where I put the actual stuff I really need. Which is how you slow down an SQL database. So RDBMS is no good for me, and I'm no good for RDBMS.…

> I still lean into events & eventual consistency to manage state across the various nodes. You can get really far with a RDMS before event sourcing etc is needed, the benefit being both your dev and user experience are going to be much simpler and easier. If you already know your problem domain and scaling concerns up front sure. But starting with a scalable pattern like this is a premature optimization otherwise an…

> You can get really far with a RDMS before event sourcing etc is needed

You can manage up to 0 partners easily. Once you go above that threshold, you're into "2-Generals" territory. At that point you're either inconsistent, eventually-consistent, or you're just bypassing your own database and using theirs directly.

> dev and user experience are going to be much simpler and easier.

I have objects, not relations. I'm not going to do the work of un-nesting a fat json transaction to store it in a single relation (or worse, normalise it into rows across multiple tables).

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#37

Earlier quoted context omitted.

Relational databases (sans JSON columns) are limited in the same way that seatbelts limit your ability to be transfenestrated during a crash. Having a rigid and well-designed schema is a mechanism to keep you or your team from doing stupid things.

nosql does not imply no schema

I think most would infer that, but in any case, it most definitely implies a non-rigid schema.

Cassandra et al. IMO only fall under the NoSQL banner by retconning the meaning to be “Not Only SQL.” Columnar DBs are a fine idea for certain uses.

Document DBs and/or chucking everything into a JSON column, though… those can die in a fire.

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#38

Earlier quoted context omitted.

nosql does not imply no schema

I think most would infer that, but in any case, it most definitely implies a non-rigid schema. Cassandra et al. IMO only fall under the NoSQL banner by retconning the meaning to be “Not Only SQL.” Columnar DBs are a fine idea for certain uses. Document DBs and/or chucking everything into a JSON column, though… those can die in a fire.

the only thing nosql means is that there are no relations. mongodb 8 and newer for example support schemas and validations, cascading checks, etc. dynamodb, more relevantly does also support a schema, and in fact you can't even create a table without defining one.

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#39
DynamoDB is great if you want to fork over a lot of money to AWS on a monthly basis. It's inability to allow you to query on arbitrary attributes/fields without creating a separate index (i.e. copy of the entire collection) is mind-boggling. There are so many better, less expensive options out there for 99.99999% of use-cases. Friends don't let friends use DynamoDB.

Re: Distributed Transactions at Scale in Amazon DynamoDB (2023)

#40
So one shot transactions can check if every timestamp in every write and item inside the transaction packet depends on data that is before the timestamp of that particular monotonic transaction timestamp?

And the pattern of including "check" transaction item is how we manually maintain data integrity (characteristic of Atomic in DBMS)

And we know which transactions are writing because they told us they wanted to write in the prepare phase (the part that the transaction manager handles separate from the one shot transaction information perspective from the client with its own communication between the transaction manager and storage nodes)

I implemented a toy dynamodb that is a trie in front of a hash map, it handles the "begins with" query style.

Post reply on HN