Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Kafka

heroku.com

Post reply on HN
Print view View original

41–50 of 120 posts

Re: Heroku Kafka

#41

Post by sethammons »

cachemiss wrote: post #5
sethammons wrote: post #4

Earlier quoted context omitted.

Would love to hear what you don't care for in Kafka and what alterative solution(s) you prefer.

To clarify, my feelings towards Kafka are from the POV of someone who has had to build a managed service on top of it, which is not the common use case (for which many people seem to be happy with). Other people may have more positive experiences. In my experience, Kafka is a solid system when you work in its wheelhouse, which is a relatively static set of servers / topics, that you add to slowly and deliberately. If…

Thanks for the thoughtful and detailed response! Very helpful.

Re: Heroku Kafka

#42

Post by andreasklinger »

For those wondering (all imo and only best guess)

The biggest advantage of kafka is that all of the heroku marketplace all of a sudden becomes "plug and play"

Essentially it's the "backend data" equivalent of what segment does for "frontend data".

Example: What's the benefit of having a graphDB service in the marketplace if most people dont want to / cant invest engineering in keeping the data in (realtime) sync.

With kafka they can establish standards that all partners can adapt to, they will simply offer piping of all heroku postgres/redis changes.

Re: Heroku Kafka

#43

Post by superuser2 »

jonahx wrote: post #7

> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?

You send a message (for example some JSON) to a Kafka topic. Any number of clients subscribe to that topic with a specific start time-stamp. Pluck a message off the queue, compute with it, send an acknowledgement. Kafka provides strong assurances that all readers get all the messages and report success (it retries otherwise), even if some participants come and go.

Very useful if, say, you have some real world event and dozens of different micro services need to do something about that event, independently.

You can also just use it for logging.

Re: Heroku Kafka

#44

Post by emfree »

cachemiss wrote: post #5
sethammons wrote: post #4

Earlier quoted context omitted.

Would love to hear what you don't care for in Kafka and what alterative solution(s) you prefer.

To clarify, my feelings towards Kafka are from the POV of someone who has had to build a managed service on top of it, which is not the common use case (for which many people seem to be happy with). Other people may have more positive experiences. In my experience, Kafka is a solid system when you work in its wheelhouse, which is a relatively static set of servers / topics, that you add to slowly and deliberately. If…

Thanks for the insightful comment!

> The alternative if you are at a company with the resources to do so (mine is), is to build something that fits your use case better than Kafka

I'd love to hear more about this :) What did you end up doing differently from Kafka? How's it working out for you?

Re: Heroku Kafka

#45

Post by ec109685 »

manigandham wrote: post #22
nodesocket wrote: post #19

Earlier quoted context omitted.

Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?

ElasticSearch is a database optimized for searching, not related at all. You can somewhat compare Kafka to Logstash but Kafka has no processing, it's purely a distributed log writing/reading/storage system that also scales far more than logstash can. You write data to it and then read from it with a basic messaging abstraction of topics and partitions.

ElasticSearch can store sequenced number data, which is really all that Kafka is doing, so I don't think it is fair to say it isn't related at all.

Re: Heroku Kafka

#46

Post by hmottestad »

Does anyone know if Kafka has improved on their data loss issues since tested by Aphyr? https://aphyr.com/posts/293-jepsen-kafka

A quote from the article: "At the end of the run, Kafka typically acknowledges 98–100% of writes. However, half of those writes (all those made during the partition) are lost."

Re: Heroku Kafka

#47

Post by manigandham »

bjt wrote: post #39
koolba wrote: post #9

I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.

> For smaller operations Redis is more than enough for pub/sub. I didn't find that to be so at my last job, one of those smaller operations. With Redis you're forced to pick between two severely constrained options: 1. Use PUBLISH/SUBSCRIBE. This is nice if you want to have several listeners all receive the same message. But if a listener is down, there's no way for it to recover a message that it missed. If there is…

Agreed, and this is because Redis is a database first, with some pub/sub and nice lists functionality. RabbitMQ is a proper message queue (mq) which provides the necessary features for bigger applications.

However RabbitMQ is also pretty fragile and terrible at scaling. NATS.io is another system that's got the messaging right and is working on persistence soon.

Re: Heroku Kafka

#48

Post by asymmetric »

saryant wrote: post #30
manigandham wrote: post #28

Earlier quoted context omitted.

It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers. Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.

The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer. We use it to ingest ~40mb/s and fan it out to a number of consuming applications. I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also th…

It's incorrect to say that RMQ sends message to consumers. What it does is it routes messages to queues/exchanges. It's then entirely up to you to decide how many consumers will effectively consume them, I.e. You can have as many consumers as you wish.

Re: Heroku Kafka

#49

Post by kinkdr »

bjt wrote: post #39
koolba wrote: post #9

I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.

> For smaller operations Redis is more than enough for pub/sub. I didn't find that to be so at my last job, one of those smaller operations. With Redis you're forced to pick between two severely constrained options: 1. Use PUBLISH/SUBSCRIBE. This is nice if you want to have several listeners all receive the same message. But if a listener is down, there's no way for it to recover a message that it missed. If there is…

How stable is RabbitMQ? I've been looking into moving from away from redis pub/sub for a bit now.

Re: Heroku Kafka

#50

Post by [deleted] »

bjt wrote: post #39
koolba wrote: post #9

I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.

> For smaller operations Redis is more than enough for pub/sub. I didn't find that to be so at my last job, one of those smaller operations. With Redis you're forced to pick between two severely constrained options: 1. Use PUBLISH/SUBSCRIBE. This is nice if you want to have several listeners all receive the same message. But if a listener is down, there's no way for it to recover a message that it missed. If there is…

[deleted]
Post reply on HN