[deleted]
Amazon Aurora Limitless Database
61–70 of 71 posts
Re: Amazon Aurora Limitless Database
#62It mentions it can scale write transactions to millions per second, but I see no mention of how it can scale read capacity?
Re: Amazon Aurora Limitless Database
#63Earlier quoted context omitted.
I want to better understand the reasoning behind the "scale to zero" point that many people mention. only case I could think of is that businesses that want "scale to zero" have very low total expenditure on the database. is this the case for you as well? because serverless without going to zero still solves a major problem for a lot of companies with some decent scale, where there is some decent traffic and its too…
Yeah, can’t really see any overlap in the camps of folk who want “scale to zero” and those who want the “automatically horizontally sharded ginormous transaction scale” that this post is about.
Why not have both? That's exactly what DynamoDB offers, and people want the SQL equivalent promise of "it'll keep up no matter what scale you're at".
The point is: You don't want to need to switch technology at a later stage, you'd like to build with the right technology from the get-go.
I almost always go for DynamoDB first, since for small projects it's essentially free, and for huge projects I do not have to worry about the ops overhead that follows normal SQL databases.
A lot of people also have traffic patterns that go:
- Very high volume during the day - Almost no or little volume during the night when your users are sleeping
Aurora Serverless v2 gets us closer to this for SQL, but does not scale to 0 so it's not too nice for the initial phases of projects.
Re: Amazon Aurora Limitless Database
#64Earlier quoted context omitted.
I appreciate the context, but this just adds so many more questions. How are you integrating hardware clocks into the Postgres transaction model? Is this just postgres wire compatible like Cockroach or Yugabyte, or have you modified Postgres MVCC implementation to not use the standard TID. Does this support all of postgres at the Transaction coordinator ? Super exciting announcement, and I am really looking forwards…
They’re probably using it similar to how Spanner uses TrueTime: https://cloud.google.com/spanner/docs/true-time-external-con...
Re: Amazon Aurora Limitless Database
#65Earlier quoted context omitted.
Aurora Limitless Database is based on our own investments in database-optimized virtualization (Caspian), in scale-out log-first database storage (Grover)[2], and in a custom approach to cross-shard transactions that makes use of the high-quality hardware clocks available in EC2[1]. We'll be talking more about the internals over the coming months. [1] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ti... [2…
> makes use of the high-quality hardware clocks available in EC2 So are you using vector clocks in the backend to handle transaction ordering and maintain consensus?
vector (or matrix) clocks are awfully slow in real life. have shipped product with this, will not recommend for future.
Re: Amazon Aurora Limitless Database
#66Earlier quoted context omitted.
I can't wait until people find its limits... in production!
Did you know that Postgres has a max table size of 32TB? Its really really fun to find that out the Wednesday evening before Thanksgiving. Make sure to prune old data from your tables. This one got to this limit because it eventually got too large that queries to delete old data would time out... so it just kept growing. https://www.postgresql.org/docs/current/limits.html
it is partitioned underneath with limit of 32TB for single partition. (hello ctid). there could be many of them.
Re: Amazon Aurora Limitless Database
#67Earlier quoted context omitted.
What are you referring to? I didn’t see anything described that’s exclusive to Spanner. CockroachDB is a Postgres-compatible-ish distributed DB that does multi shard transactions. YugabyteDB is another that comes to mind. CitusDB is a Postgres extension that manages multi shard tables. Or are you referring to automatic scale-out?
The scale of Spanner is unparalleled. They had to use atomic clocks to manage time in a distributed set up.
Aurora uses Amazon's equivalent of TrueTime using a more pragmatic high-quality GPS time solution
Re: Amazon Aurora Limitless Database
#68AWS team needs to automate migrations from RDS to Aurora
Isn't it more or less automated? Create an Aurora replica cluster from your normal RDS database, switch your application over and then delete the old RDS instance.
what if your postgres instance has replicas?
They support blue/green for RDS postgres, but not for RDS/Aurora.
It's slated for later next year is what I heard.
Re: Amazon Aurora Limitless Database
#69I used aurora for a while and liked it quite a bit (especially compared to Alloydb which was absolutely horrible).
I'm curious to hear more about your troubles with alloydb. It's something I just started to test out at work
https://news.ycombinator.com/item?id=34304376
Maybe they improved since. Let me know how you find it:)
Re: Amazon Aurora Limitless Database
#70I can't wait until people find its limits!
I'm going to guess that Amazon.com will be one of the first to find all of the limits.