Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Kafka

heroku.com

Post reply on HN
Print view View original

71–80 of 120 posts

Re: Heroku Kafka

#71

Post by balamaci »

sudhirj wrote: post #51
franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

Their pricing for hobby sites is 7$ + 10$ DB, which is very comparable with a self setup IaaS like DO and AWS. Personally I think the developer experience is much better on Heroku and quite worth it. SSL is a pain point, though I do empathize with them - I think they're doing something expensive for that. What I do is to use AWS Cloudfront and ACM for a free cert and site speedup - if they are personal projects the C…

DO has 5$ + VAT price for 512MB instance. I have no problem accommodating mysql + web on that.

Re: Heroku Kafka

#72

Post by rtehfm »

manigandham wrote: post #21

This will be interesting to try out. I've used all the major cloud event/logging systems (Kinesis, Azure EventHubs, etc) and so far Google PubSub is the best in features and performance. Only downside with Google Pubsub can be latency (which I'm working on fixing by building a gRPC driver) but Kafka has proven to be too complicated to maintain in-house. If heroku can provide the speed without the ops overhead, it'll…

What are your thoughts on Kafka vs Flume?

Re: Heroku Kafka

#73

Post by dkersten »

nickfrostatx wrote: post #68
mintplant wrote: post #66

Earlier quoted context omitted.

It looks like those are static sites. Why host them on Heroku? You could stick them on GitHub Pages [1] for free. [1] https://pages.github.com/

The S3 Free Tier can be a good alternative as well. https://aws.amazon.com/s3/

AWS free tier is free for 12 months only. Its basically a "new customer" special. Github pages is not time limited.

Re: Heroku Kafka

#74

Post by allengeorge »

ec109685 wrote: post #45
manigandham wrote: post #22

Earlier quoted context omitted.

ElasticSearch is a database optimized for searching, not related at all. You can somewhat compare Kafka to Logstash but Kafka has no processing, it's purely a distributed log writing/reading/storage system that also scales far more than logstash can. You write data to it and then read from it with a basic messaging abstraction of topics and partitions.

ElasticSearch can store sequenced number data, which is really all that Kafka is doing, so I don't think it is fair to say it isn't related at all.

So can a RDBMS... But that doesn't mean that Kafka and databases are related.

As multiple comments have stated above, Kafka is really a distributed message subsystem. Its core interface is a set of topics that one can publish to, and that consumers can read from (in other words, a pub-sub system). Kafka doesn't inspect the message payload at all.

Elasticsearch is a unstructured (to some extent) document store that's optimized around document search. So at the very least, the payload is important when using Elasticsearch.

Re: Heroku Kafka

#75

Post by sethammons »

balamaci wrote: post #60
yolesaber wrote: post #26

Earlier quoted context omitted.

Let's say you have a CMS which pushes content to your site. You also want to make the whole site searchable, so you index your content into (e.g) Elasticsearch. Kafka is great for this because you can put the content onto Kafka's message queue and then have a service reading from it which then put's it into Elasticsearch. It scales well, too. So let's say your site takes off and you have hundreds of articles publishe…

Hm but why would you not send it directly to ElasticSearch?

Kafka shines when you have multiple services that have data to publish and multiple services that need to read that data stream. If you have three services and they write to ES, publish metrics to some other store, and log events to the db, you could instead write that all to Kafka, and individual consumers can use the data (for instance, to put into ES). On the origin-service side, it has one integration point; it does not need to know about ES. Now let's say that your users want a near real-time dashboard of their data changes on your multiple services. All you do is make a new consumer from Kafka. You don't add it to your three services. Kafka simplifies your service relation graph.

Re: Heroku Kafka

#76

Post by dkersten »

saryant wrote: post #30
manigandham wrote: post #28

Earlier quoted context omitted.

It's not a message queue, it's a logging system. Queues are meant for ephemeral messages that expire once consumed. Kafka is immutable log storage that can be read as many times as necessary by consumers. Biggest competitors would be AWS Kinesis, Azure EventHubs and Google PubSub.

The biggest difference, IMO, is that Kafka is typically used when a message will be consumed by multiple consumers, whereas RabbitMQ or SQS generally send a message to a single consumer. We use it to ingest ~40mb/s and fan it out to a number of consuming applications. I'll also add that if you put some thought behind your topic replication and partitioning you can build some incredibly resilient applications. Also th…

messages are not deleted or modified once written to a topic, other than by topic retention settings

Except if you have log compaction turned on, I guess.

A key "selling point" of Kafka for me is that each consumer can decide from when they wish to receive messages. That is, you can replay the messages.

Re: Heroku Kafka

#77

Post by dkersten »

amock wrote: post #53
gshx wrote: post #38

Earlier quoted context omitted.

It can be used as a queue but the bigger benefit is for streaming use cases. One of the key differences, among others, is that streaming assumes somewhat faster consumers as opposed to queueing. There's also the pub-sub use-case which is generally considered separate from that of a queue (considered a point to point transport).

That is more descriptive, but it still sounds like queue functionality. Streaming processing is just a queue that gets emptied quickly and pub-sub is just a set of queues.

Kafka doesn't generally get emptied quickly, but rather retains messages for a configured time/size. Because of this, consumers can choose to replay previously consumed messages, if they wish to do so.

Re: Heroku Kafka

#78

Post by kasey_junk »

elcct wrote: post #59

My impression of Kafka was that this thing is bloated. How it compares to something like NSQ?

Its a completely different use case. Many times people call Kafka a "message queue" but its not. It's a distributed log service. Its possible to build a message queue on top of a distributed log service but there are reasons not to.

Its better to think of Kafka as a database for events, not as a transport mechanism for those events.

As for being bloated, Kafka lives in a very empty space, that is it supports fully ordered events to all consumers (and it has good HA options). The only other tool that I've come across that gives you the same data guarantees is Kinesis and it requires AWS.

I've found that yes Kafka is complex, but its complex because its solving a complex problem, not because its bloated.

That said, if you want a non-ordered message queue, use NSQ instead of Kafka.

Re: Heroku Kafka

#79

Post by njudah »

franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

There will be news on this front soon; stay tuned.

Re: Heroku Kafka

#80

Post by [deleted] »

franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

[deleted]
Post reply on HN