Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

github.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

#21

Post by jpgvm »

chenyang wrote: post #15
jpgvm wrote: post #7

Earlier quoted context omitted.

If you need to ask you aren't ready for something like this. That said, it's basically Kafka with the topic-partition persistence swapped out. So the libraries etc are just the standard Kafka libraries because the frontend listener is unmodified.

Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream ( https://github.com/AutoMQ/automq/tree/main/s3stream ) library, thereby harnessing the benefits of cloud EBS and S3.

One thing that isn't made clear is when writes are acknowledged.

Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage?

If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated in a similar fashion to traditional Kafka storage?

Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

#22

Post by west0n »

I bet that all message queues and log databases will support S3, as these types of data generally have a large volume and aren't as economically valuable (don't get me wrong, what I mean is that these databases won't be frequently read and processed).

Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

#24

Post by chenyang »

jpgvm wrote: post #21
chenyang wrote: post #15

Earlier quoted context omitted.

Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream ( https://github.com/AutoMQ/automq/tree/main/s3stream ) library, thereby harnessing the benefits of cloud EBS and S3.

One thing that isn't made clear is when writes are acknowledged. Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage? If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated i…

Yes, acknowledgments for writes occur once the data is committed to the EBS WAL, with each write operation bypassing the cache via Direct IO. Data is then asynchronously uploaded to S3.

Given that EBS already ensures various levels of data durability, AutoMQ does not replicate data. Addressing your last question regarding the scenario when an EBS volume becomes unavailable:

- AutoMQ maintains a minimal amount of data on EBS, for example, only 500MB, which can be easily cached in memory. If an EBS volume goes offline, we promptly upload all data to S3 and close all partitions on the affected broker. Subsequently, we redistribute the closed partitions to other brokers.

Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

#25

Post by chenyang »

jpgvm wrote: post #21
chenyang wrote: post #15

Earlier quoted context omitted.

Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream ( https://github.com/AutoMQ/automq/tree/main/s3stream ) library, thereby harnessing the benefits of cloud EBS and S3.

One thing that isn't made clear is when writes are acknowledged. Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage? If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated i…

You might be interested in our strategies for managing different types of failures:

- In case of an EC2 instance failure, we take advantage of EBS's ability to be attached to multiple instances(https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes...). This allows us to quickly mount the EBS volume from the failed EC2 instance onto another Broker, facilitating a seamless failover process. - For failures that affect an entire availability zone, we utilize Regional EBS which is available in Azure and GCP: https://cloud.google.com/compute/docs/disks/regional-persist...

Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

#26

Post by chenyang »

west0n wrote: post #22

I bet that all message queues and log databases will support S3, as these types of data generally have a large volume and aren't as economically valuable (don't get me wrong, what I mean is that these databases won't be frequently read and processed).

Can't agree more! S3 will be the modern data storage primitive. Also, the move towards shared storage and separating compute from storage is a key trend in cloud-native architecture, enhancing scalability and cost-efficiency.

Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds

#27

Post by jpgvm »

chenyang wrote: post #25
jpgvm wrote: post #21

Earlier quoted context omitted.

One thing that isn't made clear is when writes are acknowledged. Specifically is a write acknowledged when it's written to Delta WAL or when it's uploaded to object storage? If writes are acknowledged when written to Delta WAL is it possible to lose acknowledged writes when an EBS volume becomes unavailable or does that whole partition become unwritable until the volume comes back? Or is Delta WAL itself replicated i…

You might be interested in our strategies for managing different types of failures: - In case of an EC2 instance failure, we take advantage of EBS's ability to be attached to multiple instances( https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes... ). This allows us to quickly mount the EBS volume from the failed EC2 instance onto another Broker, facilitating a seamless failover process. - For failures that…

Ok yeah, multi-attach was the magic I was looking for to handle failure of instances.

Thanks!

Post reply on HN