AutoMQ Reduces Nearly 100% of Kafka Cross-Zone Data Transfer Cost

#1

Post by ByteBard1979 » Wed, Oct 23, 2024, 3:02 AM UTC

AutoMQ Reduces Nearly 100% of Kafka Cross-Zone Data Transfer Cost
medium.com

Re: AutoMQ Reduces Nearly 100% of Kafka Cross-Zone Data Transfer Cost

#2

Post by AlbertoGP » Wed, Oct 23, 2024, 5:22 AM UTC

This is specific to Amazon Web Services: if I understand correctly, this essentially consists on passing the bulk of the data not directly form a server in one Availability Zone (data center?) to another, but to store it in S3 and pass only messages about it (metadata) which are tiny in comparison.

Apparently, transmission between any AZ and S3 is flat rate while AZAZ is metered which is where the savings come from.

Re: AutoMQ Reduces Nearly 100% of Kafka Cross-Zone Data Transfer Cost

#3

Post by ByteBard1979 » Mon, Oct 28, 2024, 4:17 AM UTC

AlbertoGP wrote: post #2

This is specific to Amazon Web Services: if I understand correctly, this essentially consists on passing the bulk of the data not directly form a server in one Availability Zone (data center?) to another, but to store it in S3 and pass only messages about it (metadata) which are tiny in comparison. Apparently, transmission between any AZ and S3 is flat rate while AZ AZ is metered which is where the savings come from.

Not only AWS, but GCP also charges for network communication across availability zones. Moreover, your understanding is correct.The key here is the use of S3 as a shared storage feature. Reading and writing S3-like object storage across AZs will not incur cross-AZ traffic fees. Cloud vendors charge for API call fees and storage fees for object storage.