For anyone wanting to play with Kafka, Spotify's Kafka container was an invaluable resource for getting me up and running with Kafka. All the Zookeeper dependencies are taken care of allowing you to just start playing with Kafka right away.
https://github.com/spotify/docker-kafka
https://hub.docker.com/r/spotify/kafka/
Heroku Kafka
31–40 of 120 posts
Re: Heroku Kafka
#32Earlier quoted context omitted.
It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers. Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.
The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer. We use it to ingest ~40mb/s and fan it out to a number of consuming applications. I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also th…
It's very easy to setup multiple consumers with RabbitMQ topics though and SQS is a very basic queueing system that just doesn't support much.
To me, Kafka is just meant for much larger magnitudes of scale and persistence (of the entire log of messages for however long you need) as a core feature.
Google's PubSub is still the best blend of traditional queue semantics with Kafka scale and persistence though.
Re: Heroku Kafka
#33For anyone wanting to play with Kafka, Spotify's Kafka container was an invaluable resource for getting me up and running with Kafka. All the Zookeeper dependencies are taken care of allowing you to just start playing with Kafka right away. https://github.com/spotify/docker-kafka https://hub.docker.com/r/spotify/kafka/
I never tried spotify's container. Tried wurstmeister's, and was able to run it on a single server for testing purposes, but kept running into issues while clustering on different servers. Decided to use Ambari and have it do all the work for me instead.
Re: Heroku Kafka
#34Earlier quoted context omitted.
Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?
Kafka, Redis and Logstash+ElasticSearch have really nothing to do with each other. Kafka is a distributed, fault-tolerant and highly scalable message broker. Redis is a very fast key/value (another other data types) store. I suppose that at a high level Logstash can be compared to Kafka but IME Logstash can't handle scale. It's trivially easy to bring Logstash to its knees. Elasticsearch is, well, a search engine.
Redis does a lot more than just store keys and values. Functionally speaking Redis Pub/Sub and Kafka are interchangable up to a certain level of throughput.
Re: Heroku Kafka
#35Is the pricing public?
Not yet. We're working it during our early access program. Well be looking for lots of feedback from customers.
Re: Heroku Kafka
#36Earlier quoted context omitted.
Kafka, Redis and Logstash+ElasticSearch have really nothing to do with each other. Kafka is a distributed, fault-tolerant and highly scalable message broker. Redis is a very fast key/value (another other data types) store. I suppose that at a high level Logstash can be compared to Kafka but IME Logstash can't handle scale. It's trivially easy to bring Logstash to its knees. Elasticsearch is, well, a search engine.
Redis does a lot more than just store keys and values. Functionally speaking Redis Pub/Sub and Kafka are interchangable up to a certain level of throughput.
They really serve very different use cases. Redis pub/sub consumers only receive messages while connected, whereas Kafka consumers can pick up where they left off. Event ordering, back pressure, etc.
There are many use cases Redis pub/sub can't serve beyond just scalability.
Re: Heroku Kafka
#37As a former user of Kafka, this is awesome and it would have been a huge help for our company if this was available then. I'm glad to hear that a company is offering Kafka as opposed to other propriety versions(AWS Kinesis etc).
One thing is odd though, there is no mention of disk space at all and only a configuration of retention time. One of Kafka's best features is the use of disk to store large amounts of messages, you are not RAM bound. Heroku seems to only allows you to set retention times? This could be awesome if they are giving you "unlimited" disk space, but could also be a beta oversight. Interested to see how this progresses.
Re: Heroku Kafka
#38> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?
It's a distributed message queue.
It can be used as a queue but the bigger benefit is for streaming use cases. One of the key differences, among others, is that streaming assumes somewhat faster consumers as opposed to queueing. There's also the pub-sub use-case which is generally considered separate from that of a queue (considered a point to point transport).
Re: Heroku Kafka
#39I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.
> For smaller operations Redis is more than enough for pub/sub.
I didn't find that to be so at my last job, one of those smaller operations.
With Redis you're forced to pick between two severely constrained options:
1. Use PUBLISH/SUBSCRIBE. This is nice if you want to have several listeners all receive the same message. But if a listener is down, there's no way for it to recover a message that it missed. If there is no one listening, messages are just dropped.
2. Use LPUSH/BRPOP. This is nice if you want to have several workers all pulling from the same queue, but isn't sufficient if you want to have several queues streaming from the same topic. (E.g. one listener is responsible for syncing to ElasticSearch and another one is syncing to your analytics DB.)
I strongly prefer RabbitMQ. Its model of exchanges and queues supports mixing and matching these semantics much more flexibly.
Re: Heroku Kafka
#40I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle".