41–50 of 71 posts
Ah yes, Reinvent season, when all of the AWS announcements make the front page of HN but most won't be generally available for 6-12 months.
We are using the ~equivalent SQL Server architecture in Azure right now (Hyperscale tier) and really enjoy the experience. Not having to worry about specific machines and being able to treat it the same way as an express instance (from an app perspective) are extremely compelling selling points.
The only downside I saw in the Microsoft offering was the fact that there was a throttle for transaction log writes (100 megabytes/s) which is required to maintain resilience across the cluster. You don't want your replicas to wind up 3 weeks behind, etc. My hope is this limit will only increase with time, but even so it is quite workable (for us) today.
I may spin up an instance of Aurora (when available) to take a look at how it behaves around our worst-case serialized txn paths. Our use of SQL Server is very "light" in terms of its weird claws/hooks, so we could pretty easily shift to another RDBMS vendor in the future, assuming we can sell our clients on the 3rd party mix.
Earlier quoted context omitted.
Did you know that Postgres has a max table size of 32TB? Its really really fun to find that out the Wednesday evening before Thanksgiving. Make sure to prune old data from your tables. This one got to this limit because it eventually got too large that queries to delete old data would time out... so it just kept growing. https://www.postgresql.org/docs/current/limits.html
I just put a maximum Postgres table size I’m willing to manage in my employment contract. By the time we’re measuring in Terabytes I’m out of there.
This sounds very exciting, but I would have greatly appreciated some actual technical details. From this statement > Sharded tables – These tables are distributed across multiple shards. Data is split among the shards based on the values of designated columns in the table, called shard keys. It sounds like this is very much managed CitusDB on top of Aurora, but without any details about the implementation its impossi…
Aurora Limitless Database is based on our own investments in database-optimized virtualization (Caspian), in scale-out log-first database storage (Grover)[2], and in a custom approach to cross-shard transactions that makes use of the high-quality hardware clocks available in EC2[1]. We'll be talking more about the internals over the coming months. [1] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ti... [2…
From what I hear from folks using things like Vitess, if you're used to a monolithic SQL database there's often some things to learn to mentally model your query cost well after you move to a sharded world, and understanding more up front can save heartburn later. Writing up those details well is a good thing that AWS could do.
AWS team needs to automate migrations from RDS to Aurora
Earlier quoted context omitted.
I appreciate the context, but this just adds so many more questions. How are you integrating hardware clocks into the Postgres transaction model? Is this just postgres wire compatible like Cockroach or Yugabyte, or have you modified Postgres MVCC implementation to not use the standard TID. Does this support all of postgres at the Transaction coordinator ? Super exciting announcement, and I am really looking forwards…
They’re probably using it similar to how Spanner uses TrueTime: https://cloud.google.com/spanner/docs/true-time-external-con...
edit: More details make it seem like this _isn't_ Postgres on the frontend, its just mostly postgres compatible, which would explain how they got away from the xid based MVCC. So it seems like this is an entirely new distributed database, rather then a modification to existing postgres.
Earlier quoted context omitted.
I can't wait until people find its limits... in production!
Production is the best testing environment after all.
For the other 10%, we take them aside and politely explain that they almost certainly have an unusable staging environment per the scope of our B2B project.
Testing in production is a wonderful path if you are comfortable talking to business people and making lots of compromises.
> a new capability supporting automated horizontal scaling to process millions of write transactions per second and manage petabytes of data in a single Aurora database. Limitless horizontal scaling is really cool and all, but does anyone not running a Fortune-500 Tech company actually need this kind of firepower?
Maybe it's because most projects will never need it, but when you eventually do it feels like you're just left to yourself