Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hermes – A message broker built on top of Kafka

allegro.tech

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Hermes – A message broker built on top of Kafka

#21

Post by thanatos_dem »

lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

Exactly once delivery is not a thing, and confluent needs to stop openly lying to people about it. At least once delivery and idempotence is not the same, and has existed forever.

Calling it “exactly once” is marketing BS. It’s the same as Oracle claiming for years to support serializable transactions when they didn’t, except that one is technically possible, they just didn’t support it in actuality.

Re: Hermes – A message broker built on top of Kafka

#22

Post by lllr_finger »

twa927 wrote: post #18
chrisjc wrote: post #16

Earlier quoted context omitted.

This is worth a read. https://engineering.linkedin.com/distributed-systems/log-wha...

I skimmed over it but it's again mostly about internal design. The high level use case I see is "publish-subscribe" which is handled by RabbitMQ and a dozen of other solutions. One use case I see is that the events published into Kafka are persisted so e.g. some component can see a history of some events (so this is something not handled by RabbitMQ). Is it right?

Event streams between decoupled systems is kind of the sweet spot for Kafka. It's extremely easy to scale horizontally, and handles distributed work and network partitions in an easy to reason about way. I've also seen Rabbit be the bottleneck before where I've never really seen Kafka be the bottleneck in an architecture - it's very analogous to a firehose. For organizations shuttling messages and events between teams, it makes a very convenient lingua franca.

Re: Hermes – A message broker built on top of Kafka

#23

Post by josephg »

twa927 wrote: post #15

Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.

I’ve long had the inverse view - I’m not sure what good use cases there are for Rabbitmq that couldn’t be handled better by a Kafka cluster.

One company I worked with used Kafka as their central source of truth across the organisation. All events generated by users were thrown into a massive Kafka cluster. Each team in the organisation cared about a different view into that data (financials, marketing, fraud, what we display to that user on the website, etc). Each team would ingest the same kafka queue and do different things with it - often consuming certain events into their own Postgres instance, or other things like that.

I used Kafka when I made my reddit r/place clone a few years ago because it gives great read and write amplification. With Postgres as a central source of truth, you can only handle thousands of writes per second. And reads will slow down the instance. With Kafka you can handle about 2M/sec. And reads can really easily be serviced from other machines - you can just have a bunch of downstream Kafka instances consuming from the root, and serving your readers in turn.

It may be that you can also solve all these problems with a well configured rabbitmq cluster. But coming from a database world I find it more comfortable to reason about architecture, performance and correctness with Kafka.

Re: Hermes – A message broker built on top of Kafka

#24

Post by chrisjc »

twa927 wrote: post #18
chrisjc wrote: post #16

Earlier quoted context omitted.

This is worth a read. https://engineering.linkedin.com/distributed-systems/log-wha...

I skimmed over it but it's again mostly about internal design. The high level use case I see is "publish-subscribe" which is handled by RabbitMQ and a dozen of other solutions. One use case I see is that the events published into Kafka are persisted so e.g. some component can see a history of some events (so this is something not handled by RabbitMQ). Is it right?

I often do the same thing... skim through articles and papers to get the gist. Trust me, this is one to ready all the way through.

Re: Hermes – A message broker built on top of Kafka

#25

Post by ryanworl »

thanatos_dem wrote: post #21
lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

Exactly once delivery is not a thing, and confluent needs to stop openly lying to people about it. At least once delivery and idempotence is not the same, and has existed forever. Calling it “exactly once” is marketing BS. It’s the same as Oracle claiming for years to support serializable transactions when they didn’t, except that one is technically possible, they just didn’t support it in actuality.

They do not use the phrase exactly-once delivery as far as I’ve seen, they say “exactly-once semantics”. And this is referring to a specific set of Kafka features that you previously had to build yourself if you used Kafka. I have not seen any reference to them claiming their solution is somehow novel. In fact, it just wraps up patterns people were doing anyway.

Oracle claiming to support serializable transactions is also false (as you say), but calling it a lie is not the whole story. “ANSI serializable” and actually serializable are not the same thing. Oracle is “ANSI SQL-92 serializable”, as in no named anomalies from the spec. There just happen to be more anomalies like write skew which are not in the spec.

Re: Hermes – A message broker built on top of Kafka

#26

Post by mancini0 »

twa927 wrote: post #15

Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.

A financial exchange - order messages are routed to Kafka and partitioned by the instruments symbol, match engines associated with a given set of symbols consume from their assigned partition. When a match-engine goes down it can reconstruct the order book by replaying from a given offset.

Re: Hermes – A message broker built on top of Kafka

#27

Post by Plugawy »

twa927 wrote: post #18
chrisjc wrote: post #16

Earlier quoted context omitted.

This is worth a read. https://engineering.linkedin.com/distributed-systems/log-wha...

I skimmed over it but it's again mostly about internal design. The high level use case I see is "publish-subscribe" which is handled by RabbitMQ and a dozen of other solutions. One use case I see is that the events published into Kafka are persisted so e.g. some component can see a history of some events (so this is something not handled by RabbitMQ). Is it right?

That's correct - being able to "rewind" a history of a topic (queue) is a powerful concept. But, Kafka is a bit harder to operate than RabbitMQ in my experience. (Somewhat related, one of our subsystems originally was built around Kafka, but later was migrated to RabbitMQ and Postgres)

Re: Hermes – A message broker built on top of Kafka

#28

Post by pgwhalen »

thanatos_dem wrote: post #21
lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

Exactly once delivery is not a thing, and confluent needs to stop openly lying to people about it. At least once delivery and idempotence is not the same, and has existed forever. Calling it “exactly once” is marketing BS. It’s the same as Oracle claiming for years to support serializable transactions when they didn’t, except that one is technically possible, they just didn’t support it in actuality.

You’d be served well by reading confluent/Kafka documentation a bit more closely. They do not claim exactly once delivery, they claim (and achieve) exactly once processing (“semantics”) which is still very useful.

Re: Hermes – A message broker built on top of Kafka

#29

Post by guhcampos »

lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

I see a bit of their reasoning. Taking from the article:

"When you have an environment with 20+ services, code sharing, maintenance and following updates become problematic. At Allegro we had the chance to find it out. It’s better to take out dependencies from business services as much as possible."

By adopting something like Hermes you are not really "taking out" a dependency, but abstracting it - as you said. Yet, by talking HTTP to the message broker you are abstracting away Kafka from your developers and your code. One less lib to depend on for each language you have in your architecture. One less version to control among your services, etc.

Re: Hermes – A message broker built on top of Kafka

#30

Post by bpicolo »

lllr_finger wrote: post #20
infecto wrote: post #14

Earlier quoted context omitted.

I am not too familiar with Hermes but there is a lot of power exposing a http endpoint. I think you are thinking too inside of the box here. The benefit here is not adoption from developers or inability to understand how Kafka works with more native libs. The value here is 1) publishing messages from more unique sources. Perhaps allowing your clients to publish messages. 2) You can enforce additional guarantees. Does…

Developers can do both pretty easily today without too much effort. The TCO of Kafka clusters is considerably higher than the costs in developing an API that marries a POST to a Producer. It feels like you're saving time on one of the easier parts of the solution. Totally open to hearing more out of the box ideas. I don't think Hermes is bad for existing, I'm just not envisioning why I'd recommend it to someone.

It's a scale thing when you want a ton of different endpoints.

This is one of the options that Google Pub/Sub and SNS can do for you (and if you use e.g. App Engine, this is also queueing model). It can sometimes be an easier model for bootstrapping message passing.

Post reply on HN