"With limitless databases come limitless costs" (c)
Amazon Aurora Limitless Database
51–60 of 71 posts
Re: Amazon Aurora Limitless Database
#52Earlier quoted context omitted.
I can't wait until people find its limits... in production!
Did you know that Postgres has a max table size of 32TB? Its really really fun to find that out the Wednesday evening before Thanksgiving. Make sure to prune old data from your tables. This one got to this limit because it eventually got too large that queries to delete old data would time out... so it just kept growing. https://www.postgresql.org/docs/current/limits.html
Wow. That is still pretty large for one table.
Re: Amazon Aurora Limitless Database
#53This is some seriously impressive engineering. Spanner is the only DB that can achieve this.
What are you referring to? I didn’t see anything described that’s exclusive to Spanner. CockroachDB is a Postgres-compatible-ish distributed DB that does multi shard transactions. YugabyteDB is another that comes to mind. CitusDB is a Postgres extension that manages multi shard tables. Or are you referring to automatic scale-out?
The scale of Spanner is unparalleled. They had to use atomic clocks to manage time in a distributed set up.
Re: Amazon Aurora Limitless Database
#54> a new capability supporting automated horizontal scaling to process millions of write transactions per second and manage petabytes of data in a single Aurora database. Limitless horizontal scaling is really cool and all, but does anyone not running a Fortune-500 Tech company actually need this kind of firepower?
At Notion we run ~100 Postgres instances and we’re pretty far off from being Fortune 500, although we have many Fortune 500 clients. We could have used this Aurora thing instead of doing sharding in the application ourselves: https://www.notion.so/blog/sharding-postgres-at-notion
Separation of compute and storage, and storage having tables being sharded is now in many modern databases (cockroachdb, clickhouse and singlestore come to mind).
This being part of native aws fully postgres compatible is exciting though.
I wish AWS brings columnstore or custom storage engine features.
But I can see why AWS wouldn’t want to bring columnstore tables - it would directly compete with redshift.
Re: Amazon Aurora Limitless Database
#55Earlier quoted context omitted.
They’re probably using it similar to how Spanner uses TrueTime: https://cloud.google.com/spanner/docs/true-time-external-con...
but Spanner isn't Postgres compatible, they have their own transaction processing layer that is built for TrueTime. If this is using the Postgres frontend how are they incorporating TrueTime into the normal Postgres MVCC model that uses the monotonic xid. edit: More details make it seem like this _isn't_ Postgres on the frontend, its just mostly postgres compatible, which would explain how they got away from the xid…
I think they are internally using postgres only. aurora uses a custom storage process on host which internally supports the custom storage engine (s3)
given the mentions of "log as a database", I believe that depending on the transaction the storage responds differently. like how mysql mvcc uses undolog to essentially rollback database internally so that transaction sees data consistency. they could be doing something similar i.e regardless of whatever postgres uses, if they can get a reference of transaction and its start time then they can use the custom storage engine to rollback the log and respond in that way
Re: Amazon Aurora Limitless Database
#56Earlier quoted context omitted.
They’re probably using it similar to how Spanner uses TrueTime: https://cloud.google.com/spanner/docs/true-time-external-con...
but Spanner isn't Postgres compatible, they have their own transaction processing layer that is built for TrueTime. If this is using the Postgres frontend how are they incorporating TrueTime into the normal Postgres MVCC model that uses the monotonic xid. edit: More details make it seem like this _isn't_ Postgres on the frontend, its just mostly postgres compatible, which would explain how they got away from the xid…
remember it is aws, "leverage" is built into their DNA, I don't believe that they would go on the route of building a totally new SQL database like google did
Re: Amazon Aurora Limitless Database
#57Things I want in a serverless database: 1. Scales to zero, no cost when not using 2. Allows for SQL over API like aurora v1. This point is important, it allows for faster access from other serverless technologies, i.e. lambda
I want to better understand the reasoning behind the "scale to zero" point that many people mention.
only case I could think of is that businesses that want "scale to zero" have very low total expenditure on the database. is this the case for you as well?
because serverless without going to zero still solves a major problem for a lot of companies with some decent scale, where there is some decent traffic and its too hard to implement database autoscaling and most businesses have unequal traffic (daytime and nighttime etc,...)
Re: Amazon Aurora Limitless Database
#58Things I want in a serverless database: 1. Scales to zero, no cost when not using 2. Allows for SQL over API like aurora v1. This point is important, it allows for faster access from other serverless technologies, i.e. lambda
I want to better understand the reasoning behind the "scale to zero" point that many people mention. only case I could think of is that businesses that want "scale to zero" have very low total expenditure on the database. is this the case for you as well? because serverless without going to zero still solves a major problem for a lot of companies with some decent scale, where there is some decent traffic and its too…
Yeah, can’t really see any overlap in the camps of folk who want “scale to zero” and those who want the “automatically horizontally sharded ginormous transaction scale” that this post is about.
Re: Amazon Aurora Limitless Database
#59This is some seriously impressive engineering. Spanner is the only DB that can achieve this.
Not sure i would say Spanner is the only database that can do this... first of all, this new thing from AWS can't do what spanner can do and second, there are few that can deliver distributed transactions in addition to Spanner, such as CockroachDB (ex-emplyer), Yagbyte and TiDB.
Re: Amazon Aurora Limitless Database
#60limitless is hyperbole