Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Kafka

heroku.com

Post reply on HN
Print view View original

61–70 of 120 posts

Re: Heroku Kafka

#61

Post by balamaci »

nodesocket wrote: post #19
manigandham wrote: post #18

Earlier quoted context omitted.

Kafka and Redis are very different things - see this: https://news.ycombinator.com/item?id=11577312 Redis is a database, Kakfa is a data logging system built for scale and throughput. Event processing (of any kind like stocks, ad impressions, ecommerce purchases) are a great fit. Also good as a message queue unless you need ultra low-latency RPC.

Gotcha, so then advantage of Kafka over Logstash + ElasticSearch?

Well Logstash can output data into Kafka or ElasticSearch. So you could for example transform logs to json or do simple text processing in Logstash and put it to ElasticSearch for logsearching but you can also put it in Kafka and then have a stream parsing with all sorts of tools like Flink, Spark, etc. You could then have the possibility do so some realtime analysis on what the user do all over your stack. Too many "Login Failed" events and maybe you have an attacker trying to bruteforce a passsword and maybe you need to present him with a captcha screen.

Re: Heroku Kafka

#62

Post by redtuesday »

franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

Check out Red Hat's Openshift Online [1] if you haven't already. They offer 3 Gears for free, each with 512 MB Ram, 1 GB Disk space (install e.g Postgres into one gear and you have a DB with 1 GB) and you can use Lets Encrypt with their Bronze plan (which is free if you only use the 3 free gears). Depending on what your hobby sites do this could be enough.

[1] https://www.openshift.com/pricing/

Re: Heroku Kafka

#63

Post by mateuszf »

franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

If you have multiple applications though - it's possible to use one ssl terminator per domain.

Re: Heroku Kafka

#64

Post by ktamura »

mbseid wrote: post #37

As a former user of Kafka, this is awesome and it would have been a huge help for our company if this was available then. I'm glad to hear that a company is offering Kafka as opposed to other propriety versions(AWS Kinesis etc). One thing is odd though, there is no mention of disk space at all and only a configuration of retention time. One of Kafka's best features is the use of disk to store large amounts of message…

Don't forget that Heroku is the original multi-tenant shop. I wouldnt be surprised if a single Kafka instance stores multiple customers's messages and elastically scale as more customers/data is added.

Re: Heroku Kafka

#65

Post by kubek2k »

balamaci wrote: post #60
yolesaber wrote: post #26

Earlier quoted context omitted.

Let's say you have a CMS which pushes content to your site. You also want to make the whole site searchable, so you index your content into (e.g) Elasticsearch. Kafka is great for this because you can put the content onto Kafka's message queue and then have a service reading from it which then put's it into Elasticsearch. It scales well, too. So let's say your site takes off and you have hundreds of articles publishe…

Hm but why would you not send it directly to ElasticSearch?

because: * you can do ES indexing async * having articles index instantly is not critical (I guess)

Re: Heroku Kafka

#66

Post by mintplant »

franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

It looks like those are static sites. Why host them on Heroku? You could stick them on GitHub Pages [1] for free.

[1] https://pages.github.com/

Re: Heroku Kafka

#67

Post by flurdy »

sudhirj wrote: post #51
franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

Their pricing for hobby sites is 7$ + 10$ DB, which is very comparable with a self setup IaaS like DO and AWS. Personally I think the developer experience is much better on Heroku and quite worth it. SSL is a pain point, though I do empathize with them - I think they're doing something expensive for that. What I do is to use AWS Cloudfront and ACM for a free cert and site speedup - if they are personal projects the C…

The $7 is comparable to one app per DO or AWS micro/nano server. So Heroku wins on convenience.

If you have say 10 apps then Heroku costs 10*$7, but you might still only have used 1-3 servers depending on memory use of apps etc so then Heroku looses on cost.

Naturally I got a total mix of quite a few on Heroku's classic or new free plan, some on their hobby plan, some on AWS, some on docker cloud, most proxied behind a SSL certificate running on AWS..... (https://flurdy.com/docs/letsencrypt/nginx.html)

Re: Heroku Kafka

#68

Post by nickfrostatx »

mintplant wrote: post #66
franciscop wrote: post #40

I love Heroku and everything they are doing, it's doubtless a push forward for the web as a whole. However, the pricing for hobby sites (including SSL) is crazy from a personal point of view so I'm slowly moving my projects out of it [1][2]. I wish they had some kind of "Hobby Bundle". [1] http://umbrellajs.com/ [2] http://picnicss.com/

It looks like those are static sites. Why host them on Heroku? You could stick them on GitHub Pages [1] for free. [1] https://pages.github.com/

The S3 Free Tier can be a good alternative as well. https://aws.amazon.com/s3/

Re: Heroku Kafka

#70

Post by balamaci »

kubek2k wrote: post #65
balamaci wrote: post #60

Earlier quoted context omitted.

Hm but why would you not send it directly to ElasticSearch?

because: * you can do ES indexing async * having articles index instantly is not critical (I guess)

Well ES is pretty fast by itself - lots of people use it to store log entries(ELK stack) and every log line triggers an indexing event in ES. Introducing Kafka into the mix just seems like an unnecessary complication.
Post reply on HN