Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hermes – A message broker built on top of Kafka

allegro.tech

Post reply on HN
Print view View original

11–20 of 58 posts

Re: Hermes – A message broker built on top of Kafka

#11

Post by lllr_finger »

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP.

Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they struggle with the nuances, behavior, and maintenance of Kafka/ZooKeeper more than anything.

I also didn't see how it dealt with concepts like exactly once delivery - any experiences in that area?

Re: Hermes – A message broker built on top of Kafka

#12

Post by napsterbr »

lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

As someone evaluating Kafka for the first time, it would be useful to know what Hermes provides other than Kafka already does. After glancing at the homepage I see the REST api and the fact it is push based. Honestly I don't see how it would fit on my use case but interesting project nonetheless.

> exactly once delivery

Kafka is known to provide exactly once semantics - given your producers and consumers follow some rules, notably being idempotent. When ingesting from Kafka Stream API, it is actually exactly once delivery. There are a couple posts on confluent.io explaining how they achieve this (sorry, currently on mobile, can't copy-paste without having an outburst on how unusable touch devices are for me).

Re: Hermes – A message broker built on top of Kafka

#13

Post by chimen »

vowelless wrote: post #6
codeduck wrote: post #4

Earlier quoted context omitted.

It's also the name of a hilariously bad courier in the UK - so bad in fact that most people call it Herpes instead.

It’s also the name of a bureaucrat on a certain tv show.

curiously, nobody mentions the clothing designer :)

Re: Hermes – A message broker built on top of Kafka

#14

Post by infecto »

lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

I am not too familiar with Hermes but there is a lot of power exposing a http endpoint. I think you are thinking too inside of the box here. The benefit here is not adoption from developers or inability to understand how Kafka works with more native libs.

The value here is 1) publishing messages from more unique sources. Perhaps allowing your clients to publish messages. 2) You can enforce additional guarantees. Does the message conform to the a defined schema?

Re: Hermes – A message broker built on top of Kafka

#15

Post by twa927 »

Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ.

I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.

Re: Hermes – A message broker built on top of Kafka

#16

Post by chrisjc »

twa927 wrote: post #15

Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.

This is worth a read.

https://engineering.linkedin.com/distributed-systems/log-wha...

Re: Hermes – A message broker built on top of Kafka

#17

Post by lllr_finger »

napsterbr wrote: post #12
lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

As someone evaluating Kafka for the first time, it would be useful to know what Hermes provides other than Kafka already does. After glancing at the homepage I see the REST api and the fact it is push based. Honestly I don't see how it would fit on my use case but interesting project nonetheless. > exactly once delivery Kafka is known to provide exactly once semantics - given your producers and consumers follow some…

Sure, I'm aware of how Kafka works - this is an abstraction over all that, so I'm curious if and how Hermes can provide the same behaviors. I didn't see any relevant documentation at a quick glance.

Re: Hermes – A message broker built on top of Kafka

#18

Post by twa927 »

chrisjc wrote: post #16
twa927 wrote: post #15

Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.

This is worth a read. https://engineering.linkedin.com/distributed-systems/log-wha...

I skimmed over it but it's again mostly about internal design. The high level use case I see is "publish-subscribe" which is handled by RabbitMQ and a dozen of other solutions.

One use case I see is that the events published into Kafka are persisted so e.g. some component can see a history of some events (so this is something not handled by RabbitMQ). Is it right?

Re: Hermes – A message broker built on top of Kafka

#19

Post by thibauts »

lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

Exactly what I thought. It can alleviate the need to use flaky Kafka clients in some languages, but kind of disappointing in that it doesn't soften the main pain point of Kafka: operational and cognitive load.

Re: Hermes – A message broker built on top of Kafka

#20

Post by lllr_finger »

infecto wrote: post #14
lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

I am not too familiar with Hermes but there is a lot of power exposing a http endpoint. I think you are thinking too inside of the box here. The benefit here is not adoption from developers or inability to understand how Kafka works with more native libs. The value here is 1) publishing messages from more unique sources. Perhaps allowing your clients to publish messages. 2) You can enforce additional guarantees. Does…

Developers can do both pretty easily today without too much effort. The TCO of Kafka clusters is considerably higher than the costs in developing an API that marries a POST to a Producer. It feels like you're saving time on one of the easier parts of the solution.

Totally open to hearing more out of the box ideas. I don't think Hermes is bad for existing, I'm just not envisioning why I'd recommend it to someone.

Post reply on HN