From Kafka to ZeroMQ for real-time log aggregation • Hacker News Board

Re: From Kafka to ZeroMQ for real-time log aggregation

#61

Post by fauigerzigerk » Sat, Sep 17, 2016, 2:08 PM UTC

_qc3o wrote: post #18

wcdolphin wrote: post #17

Did you ever try running 5 ZK's in the ensemble? 3 is the absolute minimum to survive a single machine failure. If you are having trouble with availability, it seems natural to increase your safety factor there. I was surprised by the contrasting sense of importance of delivery guarantees in the article. At the start, losing a message was akin to the death of a child. At the end, shrug . Now every single machine fail…

They said availability was "death of a child", not dropping log messages. The trade-off they've made here in terms of being available with some potential loss of visibility is the right one. The system overall is clearly simpler and simpler systems have simpler failure modes and so it is easier to add mitigation components on top that can recover from those failure modes to guarantee higher uptime. I've never heard a…

>They said availability was "death of a child", not dropping log messages.

True, but it appears to me that availability problems and dropped log messages often have the same root cause - network issues.

So whenever they do have availability issues (and dying babies) they won't be able to investigate properly because log messages are being lost as well.

That's obviously a very general observation. It may well be that in their architecture availability issues are mostly caused by something unrelated to networking (e.g. the database).

Re: From Kafka to ZeroMQ for real-time log aggregation

#62

Post by Gigablah » Sat, Sep 17, 2016, 2:13 PM UTC

hashmp wrote: post #23

ZenoArrow wrote: post #9

Earlier quoted context omitted.

I'm in a similar boat. I'm hoping to propose Kafka to help with some data replication and consolidation tasks, but it has to be both on-premise and as low maintenance as possible (low maintenance in the sense of the work local developers would do). To anyone reading this with Kafka experience, do you have any tips/advice when it comes to maintaining a Kafka service?

Use 5 zookeepers, on a separate set of servers. Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well…

> Don't use the Kafka default of storing data in /tmp/.

That seems like MySQL level of bad defaults.

Re: From Kafka to ZeroMQ for real-time log aggregation

#63

Post by packetized » Sat, Sep 17, 2016, 2:14 PM UTC

agentgt wrote: post #50

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

Re: From Kafka to ZeroMQ for real-time log aggregation

#64

Post by agentgt » Sat, Sep 17, 2016, 2:29 PM UTC

packetized wrote: post #63

agentgt wrote: post #50

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace.

BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it.

In fact Elastic has been pretty darn speedy as of lately particularly since we purge some of the data after 6 months (we still have permanent filesystem storage of logs of course).

Re: From Kafka to ZeroMQ for real-time log aggregation

#65

Post by ZenoArrow » Sat, Sep 17, 2016, 2:36 PM UTC

hashmp wrote: post #23

ZenoArrow wrote: post #9

Earlier quoted context omitted.

I'm in a similar boat. I'm hoping to propose Kafka to help with some data replication and consolidation tasks, but it has to be both on-premise and as low maintenance as possible (low maintenance in the sense of the work local developers would do). To anyone reading this with Kafka experience, do you have any tips/advice when it comes to maintaining a Kafka service?

Use 5 zookeepers, on a separate set of servers. Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well…

Five zookeepers? Seems like a lot. Why five? Is it hard to keep them active?

Thanks for the tips.

Re: From Kafka to ZeroMQ for real-time log aggregation

#66

Post by AYBABTME » Sat, Sep 17, 2016, 2:56 PM UTC

buster wrote: post #30

To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but rea…

One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a viable choice. So dropping logs seems to be the only sane choice here. There's no other alternative really so I'm not sure about the consternation.

Re: From Kafka to ZeroMQ for real-time log aggregation

#67

Post by vegabook » Sat, Sep 17, 2016, 3:07 PM UTC

kchoudhu wrote: post #57

mej10 wrote: post #49

Earlier quoted context omitted.

This would be great, but I don't think it works on AWS -- I don't think they support broadcast.

My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query. Lots of ways to skin this cat...

indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.

Re: From Kafka to ZeroMQ for real-time log aggregation

#68

Post by trimbo » Sat, Sep 17, 2016, 3:09 PM UTC

agentgt wrote: post #64

packetized wrote: post #63

Earlier quoted context omitted.

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace. BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it. In…

> cluster failing with some network issues we had with Rackspace

Were you using Zen Discovery at the time?

I haven't kept up with ES development in the last year so maybe they fixed this, but a flaky network can cause a cluster using Zen to freak out a lot.

Re: From Kafka to ZeroMQ for real-time log aggregation

#69

Post by trimbo » Sat, Sep 17, 2016, 3:16 PM UTC

packetized wrote: post #63

agentgt wrote: post #50

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

> We run an average of 14k logs/sec through a two-node RMQ cluster

How many MB/s are you indexing?

Re: From Kafka to ZeroMQ for real-time log aggregation

#70

Post by sidlls » Sat, Sep 17, 2016, 3:32 PM UTC

bachback wrote: post #35

kchoudhu wrote: post #28

Earlier quoted context omitted.

The ZeroMQ documentation is pretty up front about the need for you to build those pieces yourself. It would appear you chose the wrong tool for your requirements.

ZeroMQ wants to be a neutral wrapper in any language but in the end its a C++ library enforcing C++ concepts. You can't map in a straightforward way process concepts from C++ to other languages (also the OS and VM sits in between that). In the end its just mapping programming logic to state machines. There are much better ways to do this and end up with something much more powerful (with first class meta-programming)…

I've found ZeroMQ to be an immensely useful and powerful library for certain kinds of distributed queueing applications. It's just a library, not an entire application or web framework to drop practically pre-dictated code into.

Also what "c++ concepts" does it enforce and how? And why is that a bad thing?