That is more descriptive, but it still sounds like queue functionality. Streaming processing is just a queue that gets emptied quickly and pub-sub is just a set of queues.
Kafka doesn't generally get emptied quickly, but rather retains messages for a configured time/size. Because of this, consumers can choose to replay previously consumed messages, if they wish to do so.
You're right. I was mostly commenting on the common idiomatic ways ppl differentiate streams vs queues. Indeed, it can be used in both scenarios.
As a former user of Kafka, this is awesome and it would have been a huge help for our company if this was available then. I'm glad to hear that a company is offering Kafka as opposed to other propriety versions(AWS Kinesis etc). One thing is odd though, there is no mention of disk space at all and only a configuration of retention time. One of Kafka's best features is the use of disk to store large amounts of message…
Hi, I'm Maciek and I work on the Heroku Kafka team. You don't have to think about disk space--it's on us to make sure there's enough to satisfy the retention settings you configure. We're excited to provide another great open-source project as a managed service!
Thanks for the update. That is awesome. Excited to see what people do with it.
ElasticSearch is a database optimized for searching, not related at all. You can somewhat compare Kafka to Logstash but Kafka has no processing, it's purely a distributed log writing/reading/storage system that also scales far more than logstash can. You write data to it and then read from it with a basic messaging abstraction of topics and partitions.
ElasticSearch can store sequenced number data, which is really all that Kafka is doing, so I don't think it is fair to say it isn't related at all.
That's like saying they all write data to disk so they're related.
Elasticsearch is all about saving, inspecting, indexing and retrieving your data through a rich document-based model and search-optimized methods.
ES might be able to do the same thing functionally because it operates at a higher level but ultimately will never scale or be as simple in access as Kafka.
I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/
It looks like those are static sites. Why host them on Heroku? You could stick them on GitHub Pages [1] for free. [1] https://pages.github.com/
now they are static sites built with grunt. Before they were dynamic so the documentation and tests could be joined and runned dynamically. Now I just do that before push, which had me change the organization of few things. I converted them and I'm now hosting them in Github Pages for that same reason. I learned a lot about grunt, phantomjs and SSL with cloudflare on the process though, so I'm happy with the result.
I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/
> For smaller operations Redis is more than enough for pub/sub. I didn't find that to be so at my last job, one of those smaller operations. With Redis you're forced to pick between two severely constrained options: 1. Use PUBLISH/SUBSCRIBE. This is nice if you want to have several listeners all receive the same message. But if a listener is down, there's no way for it to recover a message that it missed. If there is…
How stable is RabbitMQ? I've been looking into moving from away from redis pub/sub for a bit now.
RabbitMQ is ok in single server and has lots of flexibility but struggles at high throughput ( > 100k/sec) and the clustering setup is not great. There are also lots of edge case bugs.
If you don't need persistence, look at using nats.io which is a much more stable and reliable pub/sub system. You can build persistence on top of it or wait a few months until they finish their new project STAN.
This will be interesting to try out. I've used all the major cloud event/logging systems (Kinesis, Azure EventHubs, etc) and so far Google PubSub is the best in features and performance. Only downside with Google Pubsub can be latency (which I'm working on fixing by building a gRPC driver) but Kafka has proven to be too complicated to maintain in-house. If heroku can provide the speed without the ops overhead, it'll…
Not sure why you are comparing Google Cloud Pub/Sub to Kinesis - the former is a MQ system, not a distributed commit log. When creating a Kinesis consumer, I can specify whether I want to start reading a stream from a) TRIM_HORIZON (which is the earliest events in the stream which haven't yet been expired aka "trimmed"), b) LATEST which is the Cloud Pub/Sub capability, c) AT_SEQUENCE_NUMBER {x} which means from the e…
Is there something like Kinesis' AT_TIMESTAMP in Kafka?
I think the only way in to replay events in Google Cloud Pub/Sub is to create multiple subscriptions in advance, right after topic creation. But then I think you need to pay for the storage and event traversal requests.
I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/
Their pricing for hobby sites is 7$ + 10$ DB, which is very comparable with a self setup IaaS like DO and AWS. Personally I think the developer experience is much better on Heroku and quite worth it. SSL is a pain point, though I do empathize with them - I think they're doing something expensive for that. What I do is to use AWS Cloudfront and ACM for a free cert and site speedup - if they are personal projects the C…
Yeah, but I started with PHP I could just choose among many hosting companies for 5-10$/month and you get your shared space with unlimited domains, which was perfectly suited for my needs at that point. Of course as I learned more, Node and the such I needed better technology and that's why I moved to Heroku. So I'd love to see a "shared hosting for heroku" or similar. I think it will happen given some time, when the Node.js environment stabilizes more and more big players come.
How stable is RabbitMQ? I've been looking into moving from away from redis pub/sub for a bit now.
RabbitMQ is ok in single server and has lots of flexibility but struggles at high throughput ( > 100k/sec) and the clustering setup is not great. There are also lots of edge case bugs. If you don't need persistence, look at using nats.io which is a much more stable and reliable pub/sub system. You can build persistence on top of it or wait a few months until they finish their new project STAN.
Thanks! 100k is far more than I need, but I couldn't find something that would fit exactly my needs, so I ended up rolling my own.