Hermes – A message broker built on top of Kafka

Is Rabbit much easier to set up for small projects? I haven’t used it much.

Our value proposition is built on four main aspects: * ease of integration * easier Kafka management * centralised management and validation * increased stability / reliability

Mind that some of the points don't make much sense unless you have a lot of services managed by a lot of independent teams. Thus Łukasz remark about "20+ microservices" in the original post. We run 700 microservices on prod managed by something close to 70 teams.

Ease of integration has been nicely summed up by others in this thread. HTTP tends to be the simplest way to integrate anything nowadays, at least in our case. While this comes at a cost, being able to get projects started up very quickly, without getting into details of proper handling of Kafka producer/consumer clients provided great value for us. Also while history might not be so important considering using Hermes in 2019 because Kafka matured, gained traction and recognisability among Software Engineers, it wasn't so easy to handle Kafka in 0.7/0.8 days when we started.

Of course switching to HTTP comes at a cost. I think the biggest one is using pure HTTP in push model. This makes it impossible to take advantage of Kafka data model, which guarantees event ordering at partition level. Zalando took a different approach with Nakadi (https://github.com/zalando/nakadi). I would say that at some point Hermes should consider following this path for more advanced users.

Easier Kafka management. Since we abstract away Kafka and hide it behind HTTP/REST API, we can easily introduce many changes to Kafka clusters. One of them was splitting Kafka cluster into two operate ones (one per our DC) without clients noticing. They were still publishing to same old Hermes instances, discovered via Consul. While doing it with clients might seem like a trivial thing to do when you have just a few services that use Kafka, with a few hundreds of clients it generates a lot of unnecessary work for developers.

Now whenever we need to do some maintenance with Kafka clusters (rebalance partitions, change cluster/hosts etc) we just route the traffic at Hermes level and no interaction with clients/developers is necessary.

Centralised management and validation. We started with publishing JSON. Along the way, as more and more people started consuming data offline (from Hadoop), it turned out that moving to some structured/schema based format is necessary, thus Avro. Hermes helped us a lot with this. It enables us to fail fast when someone starts publishing malformed requests for whatever reason, instead of relying on consumer (online and offline) to be hit and have to communicate with producer. Secondly support for Avro in JVM (our main microservice platform) is not that great and we put a lot effort into making it better (including publishing https://github.com/allegro/json-avro-converter). By having Hermes to do on-the-fly conversion for both publishers and subscribers we made it possible to only define schema and deal as little with Avro as possible in simple cases when it might not be beneficial for the team.

We also have Hermes integrated with our Service Catalog, so we can easily track ownership of topics and subscriptions. People publishing have easy access to information about who not only subscribes to online data, but also who accesses data offline (via Hadoop) using our offline clients feature. This way Hermes provides central place to manage our data streams.

Increased stability/reliability. This last one might be controversial, but in practice it did save us a few times. Mind, that I mean increased (more nines), not totally bulletproof. Kafka is a great, resilient piece of software. it is also complex and incidents happen. It might not even be that cluster is down - but increasing response times from few ms to 1second can be just as deadly. Hermes Frontend on the other hand is really simple. By putting it in front of Kafka together with built-in buffering support, we added a layer which increased our reliability. Now even if Kafka cluster has huge problems, we can accept incoming events for 2-3 hours, having time to either resolve the issue or reroute traffic to other cluster. This means that microservices don’t have to deal with data buffering on their own. Of course Hermes is still pretty much stateless by itself, so when traffic to Kafka flows normally, we can restart, spin up and spin down instances at will.

Entering danger zone: if both Kafka goes down and Hermes hosts blow up - the data is lost. This is a trade off and we are happy to say that for years running Hermes + Kafka on production, it never failed and saved us a few times.

I hope that I managed to clarify why we are using Hermes as main message bus powering our microservice architecture. We open sourced it, as we wanted to do our work in the open, sharing it with anyone who finds it useful and beneficial :)

* s3 - use kinesis firehose to take the contents of your kinesis stream and time partition it into files for either ingestion into redshift, elastic search, etc... or later batch analysis for ML or just to treat as cold searchable storage with something like Athena

* dynamoDB - spit out the data into kinesis from dynamoDB as it changes to create a change stream used elsewhere in your platform. (dynamo-streams)

* real time analysis - perform real time sql analysis (kinesis analytics) on what's in your stream over a given window of time or data, and react as events/situations occur.

Looking at all the services that amazon has built around kinesis might help you understand some of the differences between kafka and something like RMQ.