From Kafka to ZeroMQ for real-time log aggregation • Hacker News Board

Re: From Kafka to ZeroMQ for real-time log aggregation

#21

Post by ryanmaclean » Sat, Sep 17, 2016, 7:54 AM UTC

oblio wrote: post #10

Nimimi wrote: post #7

You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default. Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.

Do you mean this? https://dcos.io/get-started/ aka DC/OS? From what I can see it doesn't really support database deployments except for ArangoDB and Cassandra.

Riak and MySQL are in Universe, for example: https://github.com/mesosphere/universe/tree/version-3.x/repo...

Re: From Kafka to ZeroMQ for real-time log aggregation

#22

Post by abritishguy » Sat, Sep 17, 2016, 7:56 AM UTC

htn wrote: post #3

FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io on AWS and also Azure, GCE and DigitalOcean. But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.

Kinesis is very poor

Re: From Kafka to ZeroMQ for real-time log aggregation

#23

Post by hashmp » Sat, Sep 17, 2016, 7:58 AM UTC

ZenoArrow wrote: post #9

janczukt wrote: post #4

Earlier quoted context omitted.

We need an on-premise and cloud story, so cloud only solutions did not cut it for us.

I'm in a similar boat. I'm hoping to propose Kafka to help with some data replication and consolidation tasks, but it has to be both on-premise and as low maintenance as possible (low maintenance in the sense of the work local developers would do). To anyone reading this with Kafka experience, do you have any tips/advice when it comes to maintaining a Kafka service?

Use 5 zookeepers, on a separate set of servers.

Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well for auto DNS registration and auto discovery.

Use the yahoo Kafka-manager app to manage the cluster and to see what is going on.

Don't use the Kafka default of storing data in /tmp/. Your OS will periodically clean it.

Re: From Kafka to ZeroMQ for real-time log aggregation

#24

Post by bachback » Sat, Sep 17, 2016, 8:02 AM UTC

With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - much more powerful and the sane default. It seems at some point people confused internal networking solutions with the Internet.

Re: From Kafka to ZeroMQ for real-time log aggregation

#25

Post by thomaslee » Sat, Sep 17, 2016, 8:16 AM UTC

I used to be on a team responsible for a single small-ish Kafka cluster (between 6-12 nodes) doing non-trivial throughput on bare metal. Without commenting on whether ZeroMQ is the right alternative: I can understand being scared off. Our hand was forced such that we had to go the other way and understand what was going on in Kafka.

The kicker is that Kafka can be rock solid in terms of handling massive throughput and reliability when the wheels are well greased, but there are a lot of largely undocumented lessons to learn along the way RE: configuration and certain surprising behavior that can arise at scale (such as https://issues.apache.org/jira/browse/KAFKA-2063, which our team ran into maybe a year ago & is only being fixed now).

Symptoms of these issues can cause additional knock-on effects with respect to things like leader election (we wound up with a "zombie leader" in our cluster that caused all sorts of bizarre problems) and graceful shutdowns.

Add to that the fact the software is still very much under active development (sporadic partition replica drops after an upgrade from 0.8.1 to 0.8.2; we had to apply some small but crucial patches from Uber's fork) & that it needs a certain level of operational maturity to monitor it all ... it's easy to get nervous about what the next "surprise" will be.

Having said all that, I'd use Kafka again in a heartbeat for those high volume use cases where reliability matters. Not sure I'd advise others without similar operational experience to do the same for anything mission critical, though -- unless you like stress. That stress is why Confluent is in business. :)

Re: From Kafka to ZeroMQ for real-time log aggregation

#26

Post by PieterH » Sat, Sep 17, 2016, 8:20 AM UTC

raarts wrote: post #20

PieterH wrote: post #19

Earlier quoted context omitted.

See http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.

From the blog: "Crazy Idea: Clone nanomsg, move to zeromq organization, relicense as MPL, support ZMTP, only new socket types and expose CZMQ API." Did that ever happen? I still like the idea behind nanomsg.

It never happened because (IMO, I'm guessing), no-one actually needs nanomsg. I've never really understood the hostility some people had towards e.g. the ZeroMQ protocols, since these RFCs are as plastic and open to contributions as any part of the project. Yet people like to react against things, and this was a large part of nano's reason for existence. In reality, people who need stable working code just take ZeroMQ and that's it.

Which is all kind of a shame since it would have been so nice to see a new C engine in the community. We have new engines in C#, Erlang, Java, yet the old core project is still that somewhat clunky C++ engine.

Re: From Kafka to ZeroMQ for real-time log aggregation

#27

Post by PieterH » Sat, Sep 17, 2016, 8:32 AM UTC

bachback wrote: post #24

With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - m…

If you try to use ZeroMQ to replace TCP with the same semantics, then no, it won't work.

Re: From Kafka to ZeroMQ for real-time log aggregation

#28

Post by kchoudhu » Sat, Sep 17, 2016, 8:32 AM UTC

bachback wrote: post #24

With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - m…

The ZeroMQ documentation is pretty up front about the need for you to build those pieces yourself. It would appear you chose the wrong tool for your requirements.

Re: From Kafka to ZeroMQ for real-time log aggregation

#29

Post by bogomipz » Sat, Sep 17, 2016, 8:36 AM UTC

Nimimi wrote: post #7

You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default. Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.

>"DC/IO is quickly becoming the go-to solution for database deployments."

It is? Can you provide any evidence supporting this claim?

Mesos is mostly used to deploy stateless services.

Re: From Kafka to ZeroMQ for real-time log aggregation

#30

Post by buster » Sat, Sep 17, 2016, 8:55 AM UTC

To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed.

I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. How can you claim to run a serious business and say "babies will die" when you can't be sure to be able to find problems?

Because, when will you lose logs? Not in normal operation, but when weird things happen. When networking has a hiccup. When Load on the system is too high, so most likely when many people are using your service. Exactly when shit hits the fan. And you just made the decision that it's ok to drop log messages in such cases? That's not good.

I think you should either dive into Kafka/Zookeeper and fix your problems or switch to another logging solution. You should probably just drop that non-sense "streaming and real-time logs" requirement and live with a log delay of a few seconds and build something really stable instead of building something inherently unstable. Honestly, just collecting syslogs on the core vm and sending them to a central server would have been the better solution. Better then looking into fancy real-time, streaming logs on a sunday night because the system is having a breakdown and you can't even be sure that you are not missing essential logs.