Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Kafka

heroku.com

Post reply on HN
Print view View original

21–30 of 120 posts

Re: Heroku Kafka

#21

Post by manigandham »

This will be interesting to try out. I've used all the major cloud event/logging systems (Kinesis, Azure EventHubs, etc) and so far Google PubSub is the best in features and performance.

Only downside with Google Pubsub can be latency (which I'm working on fixing by building a gRPC driver) but Kafka has proven to be too complicated to maintain in-house. If heroku can provide the speed without the ops overhead, it'll be some good competition to Google's option.

Also want to note that Jay Kreps who helped build Kafka at LinkedIn is now behind http://www.confluent.io/ which is like a better/enterprise version of Kafka.

Re: Heroku Kafka

#22

Post by manigandham »

nodesocket wrote: post #19
manigandham wrote: post #18

Earlier quoted context omitted.

Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312 Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.

Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?

ElasticSearch is a database optimized for searching, not related at all.

You can somewhat compare Kafka to Logstash but Kafka has no processing, it's purely a distributed log writing/reading/storage system that also scales far more than logstash can. You write data to it and then read from it with a basic messaging abstraction of topics and partitions.

Re: Heroku Kafka

#23

Post by saryant »

nodesocket wrote: post #19
manigandham wrote: post #18

Earlier quoted context omitted.

Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312 Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.

Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?

Kafka, Redis and Logstash+ElasticSearch have really nothing to do with each other.

Kafka is a distributed, fault-tolerant and highly scalable message broker.

Redis is a very fast key/value (another other data types) store.

I suppose that at a high level Logstash can be compared to Kafka but IME Logstash can't handle scale. It's trivially easy to bring Logstash to its knees.

Elasticsearch is, well, a search engine.

Re: Heroku Kafka

#24

Post by manigandham »

amock wrote: post #12
koolba wrote: post #9

I've wondered why there isn't a "big player" in the cloud space for this. Felt like a hole. My operating theory is that the people who would really make use of something like this have grown beyond managed offerings and would take it in house. For smaller operations Redis is more than enough for pub/sub. Ditto for SQS for externally triggered eventing.

Depending on what you mean by "this" there are offerings by the big players. Google has Cloud Pub/Sub and AWS has Kinesis in addition to SQS, so two of the big players do have offerings. I'm not familiar enough with Azure to know what it has.

Azure has Event Hubs that are very similar to Kinesis/Kafka.

https://azure.microsoft.com/en-us/services/event-hubs/

They also have simpler Queues and Service Bus for RPC/lightweight message handling.

Re: Heroku Kafka

#25

Post by tschellenbach »

jonahx wrote: post #7

> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?

It's a message queue. You use it for everything you want to do outside of the general request cycle. IE: Making API calls, priming cache, sending emails, etc..

Biggest competitors of Kafka are RabbitMQ and amazon SQS.

Re: Heroku Kafka

#26

Post by yolesaber »

nodesocket wrote: post #17

Can somebody provide a real-life use case for Kafka? I've seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?

Let's say you have a CMS which pushes content to your site. You also want to make the whole site searchable, so you index your content into (e.g) Elasticsearch. Kafka is great for this because you can put the content onto Kafka's message queue and then have a service reading from it which then put's it into Elasticsearch. It scales well, too. So let's say your site takes off and you have hundreds of articles published a day (not to mention updates, deletions etc) - these events can all be sent to kafka and it will maintain the order as well as still be fast. You can also have many many services reading (consuming) from it simultaneously and it will handle it nicely.

Basically, if you want to get data from one place to another and care about order, Kafka is a good solution. It acts as a middleman between services.

Re: Heroku Kafka

#27

Post by manigandham »

koolba wrote: post #20
amock wrote: post #12

Earlier quoted context omitted.

Depending on what you mean by "this" there are offerings by the big players. Google has Cloud Pub/Sub and AWS has Kinesis in addition to SQS, so two of the big players do have offerings. I'm not familiar enough with Azure to know what it has.

By "this" I meant a managed Kafka cloud offering. I generally a fan of these types of services as there isn't as tight a binding as proprietary ones. Migrating from Heroku Postgres to RDS or self hosted is well defined. Ditto for Redis migrations. SQS, Kinesis, and other proprietary ones not so much. You can insulate your code base but if you're really going to leverage the ecosystem of those services then you're goi…

There really isn't much lock-in when it comes to event logging systems. Just change the interface your code uses to whatever service you need. There might be a little refactoring to handle topics in the different ways but it's all ultimately the same thing.

Since logging by nature offers asynchronous processing, you can migrate your publishers first and then the consumers without any downtime.

Re: Heroku Kafka

#28

Post by manigandham »

tschellenbach wrote: post #25
jonahx wrote: post #7

> What is Kafka? > Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions Can anyone translate this into meaningful English for me?

It's a message queue. You use it for everything you want to do outside of the general request cycle. IE: Making API calls, priming cache, sending emails, etc.. Biggest competitors of Kafka are RabbitMQ and amazon SQS.

It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers.

Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.

Re: Heroku Kafka

#29

Post by ChartsNGraffs »

nodesocket wrote: post #17

Can somebody provide a real-life use case for Kafka? I've seen comparisons between Redis, but what specifically does Kafka solve that Redis cannot?

I'd say it's biggest differentiator from a typical messaging system is the ability to rewind and reconsume messages. It's meant to offload a large volume of data quickly and then retain it for some time so that it can processed later on. Data is published to topics and it is entirely feasible to read from one (or more topics), process that data and then publish the results to a different topic. In comparison to Redis, I would say that while they overlap they're each better suited for different problems. Redis is blazing fast, but it's parallelism/replication story isn't as great as Kafka's. Redis is a lot easier to get running though.

Re: Heroku Kafka

#30

Post by saryant »

manigandham wrote: post #28
tschellenbach wrote: post #25

Earlier quoted context omitted.

It's a message queue. You use it for everything you want to do outside of the general request cycle. IE: Making API calls, priming cache, sending emails, etc.. Biggest competitors of Kafka are RabbitMQ and amazon SQS.

It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers. Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.

The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer.

We use it to ingest ~40mb/s and fan it out to a number of consuming applications.

I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also that "immutable" isn't necessarily true, it's common for Kafka topics to roll off messages based on time or size. (That's just to clarify for those not familiar with Kafka. I realize that you mean messages are not deleted or modified once written to a topic, other than by topic retention settings)

Post reply on HN