Earlier quoted context omitted.
It’s also the name of a bureaucrat on a certain tv show.
curiously, nobody mentions the clothing designer :)
thought it was a handbag designer.
Hermes – A message broker built on top of Kafka
31–40 of 58 posts
Re: Hermes – A message broker built on top of Kafka
#32Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.
[deleted]
Re: Hermes – A message broker built on top of Kafka
#33Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…
I see a bit of their reasoning. Taking from the article: "When you have an environment with 20+ services, code sharing, maintenance and following updates become problematic. At Allegro we had the chance to find it out. It’s better to take out dependencies from business services as much as possible." By adopting something like Hermes you are not really "taking out" a dependency, but abstracting it - as you said. Yet,…
if you already have http clients on all you components.
and it's not like bindings for kafka is that much uncommon than a decent http client.
Re: Hermes – A message broker built on top of Kafka
#34Very nice with a simple wrapper. That said I’m wondering if 9 out of 10 use cases could do with something simpler, ie zeromq, which scales really well.
Re: Hermes – A message broker built on top of Kafka
#35Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.
Kafka is a high-throughput, horizontally-scalable blob data store for data streams. The data store part of that is my favorite part.
You can use it as a simple message broker, but since it keeps the message history as a timeseries, you can also do things like run batch analysis jobs on the day's message or replay the last X hours of messages because your DB died and your backup is old.
It is a good way to decouple data producers and data consumer, particularly in an enterprise context - producers push to Kafka and anyone can consume that data, whether they are an operations team that wants a realtime data stream, a BI team that needs periodic data dumps, or a team that wants a long-term audit trail (the duration of the history is going to depend on your scale, but for many users a long history is realistic).
Kafka also has a nice ecosystem including streaming analytics (KSQL), clients that make reading from Kafka easily horizontally scalable (have many machines acting as a single client, automatically rebalancing if one of those machines dies), exactly once processing and probably more since I last worked with it.
I'm not familiar enough with RabbitMQ to say how it compares to Kafka, but I haven't found a use case yet where Kafka isn't a good choice (except for the 'I need to set up a message broker quick and painlessly' use case because it is not a particularly fun technology to manage yourself)
Re: Hermes – A message broker built on top of Kafka
#36Earlier quoted context omitted.
Exactly once delivery is not a thing, and confluent needs to stop openly lying to people about it. At least once delivery and idempotence is not the same, and has existed forever. Calling it “exactly once” is marketing BS. It’s the same as Oracle claiming for years to support serializable transactions when they didn’t, except that one is technically possible, they just didn’t support it in actuality.
They do not use the phrase exactly-once delivery as far as I’ve seen, they say “exactly-once semantics”. And this is referring to a specific set of Kafka features that you previously had to build yourself if you used Kafka. I have not seen any reference to them claiming their solution is somehow novel. In fact, it just wraps up patterns people were doing anyway. Oracle claiming to support serializable transactions is…
No they do, I completely agree with the original poster on this. Confluent over egg the marketing, I think they may of changed to the phrase “exactly once schematics” as Apache Pulsar labelled it that from the start and people calling out exactly once.
Re: Hermes – A message broker built on top of Kafka
#37Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.
I’ve long had the inverse view - I’m not sure what good use cases there are for Rabbitmq that couldn’t be handled better by a Kafka cluster. One company I worked with used Kafka as their central source of truth across the organisation. All events generated by users were thrown into a massive Kafka cluster. Each team in the organisation cared about a different view into that data (financials, marketing, fraud, what we…
Sounds like your org used Kafka for event sourcing. This is almost always a bad idea, event sourcing and aggregate reconstruction is a nightmare IMO.
Kafka used as a pure FIFO cache for regular CRUD endpoints works fine
Re: Hermes – A message broker built on top of Kafka
#38Can someone help me understand the value proposition for Hermes? The only thing I can see is that it abstracts away producing to and consuming from Kafka. The use cases provided answer why you'd use a message broker system, but not why you'd want to do it over HTTP. Edit: I understand HTTP is easier than Kafka, but is this something developers really struggle with when adopting Kafka? My experience is that they strug…
I only skimmed the write up on it, but knowing kafka fairly well, the following could potentially work better than what is included by kafka:
- push model (dependent on use case)
- filtering
- throttling/rate negotiation
- exactly once (kafka out of the box does not dedupe on the broker)
Re: Hermes – A message broker built on top of Kafka
#39Ironically: Hermes was the name of a JMS user interface/testing tool.
My thoughts exactly. (I immediately remembered the old-time JMS UI. I think it was on SourceForge.)
Re: Hermes – A message broker built on top of Kafka
#40Can someone provide actual high-level use cases for using Kafka? Prefereably use-cases not handled by RabbitMQ. I've seen a few talks about Kafka but they focused on the internals. My guess is that Kafka is for large systems for which managing a multi-node RabbitMQ cluster is too much trouble.
I’ve long had the inverse view - I’m not sure what good use cases there are for Rabbitmq that couldn’t be handled better by a Kafka cluster. One company I worked with used Kafka as their central source of truth across the organisation. All events generated by users were thrown into a massive Kafka cluster. Each team in the organisation cared about a different view into that data (financials, marketing, fraud, what we…
Size? If you’re getting less than a few hundred events a minute is it worth setting up Kafka?