Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hermes – A message broker built on top of Kafka

allegro.tech

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Hermes – A message broker built on top of Kafka

#41

Post by Groxx »

lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

re HTTP specifically: one major benefit I've seen to even "just" HTTP wrappers for systems is that the HTTP ecosystem is extremely mature, even on relatively exotic languages / platforms / coding patterns / design constraints / etc.

You want load balancing, context propagation, multiplexing, proxying, authentication, request tracing, [anything from a truly gigantic list, both in-code and around-your-system]? HTTP has it. Probably several. And they probably already work with everything you already have, and happily run unattended for years.

Kafka... might? Kafka for language X.... might? But probably not.

You want to extend Kafka to add X between Y and Z? Does the protocol even allow it? HTTP does, choose your flavor. Odds are even decent that a fair number of your engineers have already heard of or used it.

---

There are benefits to specialized protocols, absolutely. But there are also benefits to letting everything just use the same robust HTTP client as everything else.

Re: Hermes – A message broker built on top of Kafka

#42

Post by thanatos_dem »

ryanworl wrote: post #25
thanatos_dem wrote: post #21

Earlier quoted context omitted.

Exactly once delivery is not a thing, and confluent needs to stop openly lying to people about it. At least once delivery and idempotence is not the same, and has existed forever. Calling it “exactly once” is marketing BS. It’s the same as Oracle claiming for years to support serializable transactions when they didn’t, except that one is technically possible, they just didn’t support it in actuality.

They do not use the phrase exactly-once delivery as far as I’ve seen, they say “exactly-once semantics”. And this is referring to a specific set of Kafka features that you previously had to build yourself if you used Kafka. I have not seen any reference to them claiming their solution is somehow novel. In fact, it just wraps up patterns people were doing anyway. Oracle claiming to support serializable transactions is…

Then they should call it "idempotent processing". They know that when people, especially middle or upper management, ask about queues, they insist on needing "exactly once" delivery, and will never understand the FLP impossibility and its implications on messaging systems.

They are intentionally using this phrasing to obscure the fact that nothing novel is happening to push sales, and it is absolutely infuriating how well it works.

Re: Hermes – A message broker built on top of Kafka

#43

Post by [deleted] »

Groxx wrote: post #41
lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

re HTTP specifically: one major benefit I've seen to even "just" HTTP wrappers for systems is that the HTTP ecosystem is extremely mature, even on relatively exotic languages / platforms / coding patterns / design constraints / etc. You want load balancing, context propagation, multiplexing, proxying, authentication, request tracing, [anything from a truly gigantic list, both in-code and around-your-system]? HTTP has…

[deleted]

Re: Hermes – A message broker built on top of Kafka

#44

Post by thanatos_dem »

tiew9Vii wrote: post #36
ryanworl wrote: post #25

Earlier quoted context omitted.

They do not use the phrase exactly-once delivery as far as I’ve seen, they say “exactly-once semantics”. And this is referring to a specific set of Kafka features that you previously had to build yourself if you used Kafka. I have not seen any reference to them claiming their solution is somehow novel. In fact, it just wraps up patterns people were doing anyway. Oracle claiming to support serializable transactions is…

No they do, I completely agree with the original poster on this. Confluent over egg the marketing, I think they may of changed to the phrase “exactly once schematics” as Apache Pulsar labelled it that from the start and people calling out exactly once.

Even from their announcement blog post, they're intentionally mixing words - https://www.confluent.io/blog/exactly-once-semantics-are-pos...

They may never describe Kafka itself without the word "semantics", but here are some other snippets:

- "I know what some of you are thinking. Exactly once delivery is impossible" - "While some have outright said that exactly once delivery is probably impossible!"

They mix their phrasing depending on what they're talking about, and whether they are referring to Kafka directly or indirectly.

Re: Hermes – A message broker built on top of Kafka

#45

Post by pablasso »

nullwasamistake wrote: post #37
josephg wrote: post #23

Earlier quoted context omitted.

I’ve long had the inverse view - I’m not sure what good use cases there are for Rabbitmq that couldn’t be handled better by a Kafka cluster. One company I worked with used Kafka as their central source of truth across the organisation. All events generated by users were thrown into a massive Kafka cluster. Each team in the organisation cared about a different view into that data (financials, marketing, fraud, what we…

Sounds like your org used Kafka for event sourcing. This is almost always a bad idea, event sourcing and aggregate reconstruction is a nightmare IMO. Kafka used as a pure FIFO cache for regular CRUD endpoints works fine

Event sourcing was one of LinkedIn use cases when they created it, Kafka is fine for all logging needs.

Re: Hermes – A message broker built on top of Kafka

#46

Post by ww520 »

twa927 wrote: post #15

Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.

It's basically a high speed transaction log, persistent, distributed, easily scalable, that happens to store messages to do messaging brokering very well.

Re: Hermes – A message broker built on top of Kafka

#48

Post by joesb »

gcb0 wrote: post #33
guhcampos wrote: post #29

Earlier quoted context omitted.

I see a bit of their reasoning. Taking from the article: "When you have an environment with 20+ services, code sharing, maintenance and following updates become problematic. At Allegro we had the chance to find it out. It’s better to take out dependencies from business services as much as possible." By adopting something like Hermes you are not really "taking out" a dependency, but abstracting it - as you said. Yet,…

if you already have http clients on all you components. and it's not like bindings for kafka is that much uncommon than a decent http client.

You can use HTTP client to contact any HTTP compatible service. You can only use Kafka client to connect to Kafka.

That sounds like a tautology but it's practical. There is way higher chance that you can make any of your future service response speak HTTP, than all of them speaking Kafka. If you add HTTP client libs today, there's higher chance of reuse than adding Kafka libs.

Re: Hermes – A message broker built on top of Kafka

#49

Post by rswail »

lllr_finger wrote: post #11

Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…

Having just completed a project using Kafka as an event pipeline and a data store, one of the issues we found was that consumer polling takes a large chunk of resources.

Having a push model for consumption would certainly remove some of the complexity we had to deal with for scaling out consumption.

Re: Hermes – A message broker built on top of Kafka

#50

Post by josephg »

nullwasamistake wrote: post #37
josephg wrote: post #23

Earlier quoted context omitted.

I’ve long had the inverse view - I’m not sure what good use cases there are for Rabbitmq that couldn’t be handled better by a Kafka cluster. One company I worked with used Kafka as their central source of truth across the organisation. All events generated by users were thrown into a massive Kafka cluster. Each team in the organisation cared about a different view into that data (financials, marketing, fraud, what we…

Sounds like your org used Kafka for event sourcing. This is almost always a bad idea, event sourcing and aggregate reconstruction is a nightmare IMO. Kafka used as a pure FIFO cache for regular CRUD endpoints works fine

Yes; they did. It worked pretty well actually.

Why do you think it’s a bad idea? Most of the arguments against event sourcing that I’ve read seem to be “yes but the tooling isn’t very good”. That might be true, but maybe we solve that problem with more investment into event sourcing; not less of it.

Post reply on HN