[dead]
Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
11–20 of 30 posts
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#12The Kafka API and S3 API have each become the de facto standard in the stream processing and object storage domains, respectively. Compared to other Kafka solutions, AutoMQ has embraced them more effectively. I believe AutoMQ has a long way to go.
It's a bit unfortunate IMO because Kafka while perfectly adequate for streaming often winds up with someone wanting to use it like a work queue which predictably ends horribly. Ideally something like Pulsars API (which more resembles something like GCP Pub/Sub) would be the de facto as it is capable of handling both cases seamlessly. There was Kafka-on-Pulsar until recently until the developers essentially made it no…
Can you elaborate more on the part where using Kafka as a work queue ends horribly? Genuinely interested!
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#13The Kafka API and S3 API have each become the de facto standard in the stream processing and object storage domains, respectively. Compared to other Kafka solutions, AutoMQ has embraced them more effectively. I believe AutoMQ has a long way to go.
It's a bit unfortunate IMO because Kafka while perfectly adequate for streaming often winds up with someone wanting to use it like a work queue which predictably ends horribly. Ideally something like Pulsars API (which more resembles something like GCP Pub/Sub) would be the de facto as it is capable of handling both cases seamlessly. There was Kafka-on-Pulsar until recently until the developers essentially made it no…
Agree that Kafka is best suited for stream scenarios. However, we also see Kafka being extensively used in online business scenarios.
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#14Is it truly possible for AutoMQ to achieve 'scale-in/out' within seconds?
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#15Seems nice, but are there any publicly available documentation to be able to evaluate its readiness and what languages are there libraries for? I may be blind but I cannot find any.
If you need to ask you aren't ready for something like this. That said, it's basically Kafka with the topic-partition persistence swapped out. So the libraries etc are just the standard Kafka libraries because the frontend listener is unmodified.
Yes, thank you for the clarification. AutoMQ has replaced the topic-partition storage with cloud-native S3Stream (https://github.com/AutoMQ/automq/tree/main/s3stream) library, thereby harnessing the benefits of cloud EBS and S3.
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#16Earlier quoted context omitted.
It's a bit unfortunate IMO because Kafka while perfectly adequate for streaming often winds up with someone wanting to use it like a work queue which predictably ends horribly. Ideally something like Pulsars API (which more resembles something like GCP Pub/Sub) would be the de facto as it is capable of handling both cases seamlessly. There was Kafka-on-Pulsar until recently until the developers essentially made it no…
Can you elaborate more on the part where using Kafka as a work queue ends horribly? Genuinely interested!
Essentially the problem is that Kafka doesn't have a way of managing acknowledgements on a per-message basis. This means that consumers from Kafka topics are assigned exclusive access on a per-partition basis and the consumer group manager only tracks acknowledged offset of each partition.
As such you end up with a few main problems. The first is head of line blocking, what this means is that if a consumer reads a message from a topic it's unable to process or will take an inordinate amount of time to process it can't move forward without potentially having to replay every message since the problematic message if it doesn't want to risk not replaying a message that wasn't processed correctly. Secondly it means that you can end up with hot partitions if the "cost" of messages isn't uniformly distributed across partitions because load isn't balanced across consumers, i.e there is no work stealing or other mechanism for other consumers to help out processing a hot partition.
Log systems with queue/subscription overlays like Pulsar and GCP Pub/Sub solve this by doing per-message acknowledgement (sometimes referred to as selective acknowledgement vs cumulative acknowledgement that Kafka does) usually by layering a persistent subscription abstraction over the top of the underlying log.
This is in contrast to pure queue systems like RabbitMQ, SQS etc that use a heap or mailbox approach where messages are simply emptied out as they are processed and don't share the log style struction of systems like Kafka.
So TLDR. If you use Kafka like a job queue you will end up in situations where queue processing gets stuck behind a single or patches of unprocessable messages.
The mitigations for it aren't pretty. They either involve building your own selective acknowledgement layer or a series of retry queues that messages are pushed onto using Kafka transactions with a final dead-letter queue at the end etc. Instead either wait for https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A... if you really want Kafka or use something that already does this, i.e Pulsar.
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#17Seems nice, but are there any publicly available documentation to be able to evaluate its readiness and what languages are there libraries for? I may be blind but I cannot find any.
jpgvm is right. AutoMQ didn't modify the computation layer of Kafka, but only revamped the storage layer. This means you can use AutoMQ just like Kafka. This is very user-friendly for those who are already using Kafka and applying tools within the Kafka ecosystem.
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#18They're making a lot of claims and in the readme present zero measurements. The official Kafka distro is pretty bulletproof since 7+ years ago. It'd be great if they could include more information for the (apparently). initiated! I am open to new things.. with at least a tad of supporting data attached :D Ymmv.
Hi, thank you for your interest in AutoMQ. If you have any questions regarding cost-related data, here is an analysis report for your reference: https://docs.automq.com/docs/automq-s3kafka/EJBvwM3dNic6uYkZ...
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#19They're making a lot of claims and in the readme present zero measurements. The official Kafka distro is pretty bulletproof since 7+ years ago. It'd be great if they could include more information for the (apparently). initiated! I am open to new things.. with at least a tad of supporting data attached :D Ymmv.
Thank you for your openness to new things. Your support is indeed very important to a new project. BTW, AutoMQ itself provides real comparison data tested on AWS, and the related code is open for users to reproduce these tests. For a cost comparison with Apache Kafka, you can refer to: https://www.automq.com/blog/automq-vs-apache-kafka-a-real-aw.... If you care about performance data, you can refer to our performance white paper: https://docs.automq.com/docs/automq-s3kafka/CYxlwqDBHitThCkx....
Re: Show HN: AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds
#20Be interesting to see how well they do. Most of the Kafka deployments I've seen in recent years have been using cloud versions e.g. AWS Managed Kafka. So it's great that you save money on compute but it does mean you are now operating it yourself which then costs money.
Indeed, operating Kafka can be challenging and complex due to its nature as a stateful and distributed system.
However, AutoMQ has adopted a cloud-native architecture, offloading storage to EBS and S3(https://docs.automq.com/docs/automq-s3kafka/Q8fNwoCDGiBOV6k8...), eliminating the need for replication and rendering the Broker stateless, which simplifies operations significantly.