Amazon Aurora Limitless Database • Hacker News Board

Re: Amazon Aurora Limitless Database

#21

Post by bdcravens » Tue, Nov 28, 2023, 5:03 PM UTC

mdaniel wrote: post #5

> Join the preview of Amazon Aurora Limitless Database would be the non-editorialized submission, and would have advised potential readers that one cannot currently play with it without a ton of hoopjumpery one will also want to watch out for this, buried 7 paragraphs in: > The preview runs in a new Aurora PostgreSQL cluster with version 15 in the AWS US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Paci…

Keep in mind that the California region has been devalued for a while. AWS would turn it off if they could.

https://www.lastweekinaws.com/blog/us-west-1-the-flagship-aw...

Re: Amazon Aurora Limitless Database

#22

Post by mjb » Tue, Nov 28, 2023, 5:04 PM UTC

dkhenry wrote: post #8

This sounds very exciting, but I would have greatly appreciated some actual technical details. From this statement > Sharded tables – These tables are distributed across multiple shards. Data is split among the shards based on the values of designated columns in the table, called shard keys. It sounds like this is very much managed CitusDB on top of Aurora, but without any details about the implementation its impossi…

Aurora Limitless Database is based on our own investments in database-optimized virtualization (Caspian), in scale-out log-first database storage (Grover)[2], and in a custom approach to cross-shard transactions that makes use of the high-quality hardware clocks available in EC2[1]. We'll be talking more about the internals over the coming months.

[1] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ti... [2] https://assets.amazon.science/dc/2b/4ef2b89649f9a393d37d3e04...

Re: Amazon Aurora Limitless Database

#23

Post by rilindo » Tue, Nov 28, 2023, 5:04 PM UTC

nusmella wrote: post #19

Agent766 wrote: post #3

I can't wait until people find its limits!

I can't wait until people find its limits... in production!

Production is the best testing environment after all.

Re: Amazon Aurora Limitless Database

#24

Post by bdcravens » Tue, Nov 28, 2023, 5:06 PM UTC

penguinpower wrote: post #10

> a new capability supporting automated horizontal scaling to process millions of write transactions per second and manage petabytes of data in a single Aurora database. Limitless horizontal scaling is really cool and all, but does anyone not running a Fortune-500 Tech company actually need this kind of firepower?

Most AWS products outside of the core group (EC2, S3, RDS, Lambda, etc) are for the enterprise or venture-backed startups.

Re: Amazon Aurora Limitless Database

#25

Post by mperham » Tue, Nov 28, 2023, 5:06 PM UTC

mdaniel wrote: post #5

> Join the preview of Amazon Aurora Limitless Database would be the non-editorialized submission, and would have advised potential readers that one cannot currently play with it without a ton of hoopjumpery one will also want to watch out for this, buried 7 paragraphs in: > The preview runs in a new Aurora PostgreSQL cluster with version 15 in the AWS US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Paci…

Thank you for the word "hoopjumpery".

Re: Amazon Aurora Limitless Database

#26

Post by vasco » Tue, Nov 28, 2023, 5:07 PM UTC

dkhenry wrote: post #8

This sounds very exciting, but I would have greatly appreciated some actual technical details. From this statement > Sharded tables – These tables are distributed across multiple shards. Data is split among the shards based on the values of designated columns in the table, called shard keys. It sounds like this is very much managed CitusDB on top of Aurora, but without any details about the implementation its impossi…

This does look exactly like CitusDB even in the naming of some things (reference / distributed tables) and how they describe it being added to an existing cluster as extra nodes with an extra manager node that will redirect requests!

Re: Amazon Aurora Limitless Database

#27

Post by icpmacdo » Tue, Nov 28, 2023, 5:07 PM UTC

Sharing here a wonderful MIT lecture on the architecture of Aurora I recently watched https://www.youtube.com/watch?v=jJSh54J1s5o

Re: Amazon Aurora Limitless Database

#28

Post by tristor » Tue, Nov 28, 2023, 5:08 PM UTC

mjb wrote: post #22

dkhenry wrote: post #8

This sounds very exciting, but I would have greatly appreciated some actual technical details. From this statement > Sharded tables – These tables are distributed across multiple shards. Data is split among the shards based on the values of designated columns in the table, called shard keys. It sounds like this is very much managed CitusDB on top of Aurora, but without any details about the implementation its impossi…

Aurora Limitless Database is based on our own investments in database-optimized virtualization (Caspian), in scale-out log-first database storage (Grover)[2], and in a custom approach to cross-shard transactions that makes use of the high-quality hardware clocks available in EC2[1]. We'll be talking more about the internals over the coming months. [1] https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-ti... [2…

> makes use of the high-quality hardware clocks available in EC2

So are you using vector clocks in the backend to handle transaction ordering and maintain consensus?

Re: Amazon Aurora Limitless Database

#29

Post by recursive » Tue, Nov 28, 2023, 5:09 PM UTC

nusmella wrote: post #19

Agent766 wrote: post #3

I can't wait until people find its limits!

I can't wait until people find its limits... in production!

Everyone has a testing environment. Some people even have separate production environments.

Re: Amazon Aurora Limitless Database

#30

Post by adamgordonbell » Tue, Nov 28, 2023, 5:10 PM UTC

What type of consistency guarantees are involved?

Are we giving away something to get multiple writers to same table? Or does the routing to the correct shard somehow solve for that?