From Kafka to ZeroMQ for real-time log aggregation • Hacker News Board

Re: From Kafka to ZeroMQ for real-time log aggregation

#81

Post by janczukt » Sat, Sep 17, 2016, 6:00 PM UTC

vegabook wrote: post #78

kchoudhu wrote: post #77

Earlier quoted context omitted.

For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works. As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.

that's a very fair point. There are few applications where if the DB goes down, there's any point for the rest of the system to stay up. My use case happens to be distribution of financial tick info without persistence - ie: no cares on temporary data losses. Therefore it's very peer-oriented and doesn't care about lost messages or duplicated messages. All it cares about is throughput of latest data samples (note the…

This is exactly how we solved our "discoverability" problem at the end of the day. It was pragmatic enough for us to have each node register itself in our production MongoDB in a collection with a TTL index. If the node went down, the DB itself removed its registration.

Re: From Kafka to ZeroMQ for real-time log aggregation

#82

Post by gerakinis » Sat, Sep 17, 2016, 6:17 PM UTC

agentgt wrote: post #64

packetized wrote: post #63

Earlier quoted context omitted.

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace. BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it. In…

You can turn off multicast discovery and write in unicast peering addresses. If you are in the cloud and you are clustering this is step 1 =)

Re: From Kafka to ZeroMQ for real-time log aggregation

#83

Post by gerakinis » Sat, Sep 17, 2016, 6:19 PM UTC

packetized wrote: post #63

agentgt wrote: post #50

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

Are you using HA functionality and also on disk backing? These two things bring down performance roughly 5-10x and are mostly required for situations that can't afford message loss. I still like the rabbitmq solution, it is my own, but i've found it takes more hardware than you are suggesting.

Re: From Kafka to ZeroMQ for real-time log aggregation

#84

Post by im_down_w_otp » Sat, Sep 17, 2016, 6:22 PM UTC

TheHydroImpulse wrote: post #13

FYI, Kafka doesn't need to fetch from disk every time as it caches the logs pretty aggressively, as long as you have enough memory. Running Zk and Kafka on the same nodes is likely not the best thing.

Why? I would think that, as long as there wasn't massive I/O contention between the two, that co-locating Kafka and Zookeeper on the same machines would mitigate a whole massive class of weird edge cases by removing one of the failure modes; the network boundary between the two critical components.

Though for my part I still don't understand why Zookeeper wasn't built as a library to add distributed strongly consistent coordination to software that needs/benefits from it rather than being an external service that needs to be connected to, and thus introduces a gnarly mess of new failure modes that make Zookeeper client behavior extremely critical and often fragile. Something that's more like a "libpaxos/libraft" (e.g. serf for Go-lang or riak_ensemble for Erlang) seems a lot more valuable. /shrug

Re: From Kafka to ZeroMQ for real-time log aggregation

#85

Post by gerakinis » Sat, Sep 17, 2016, 6:23 PM UTC

agentgt wrote: post #50

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…

Heka is really stupid to configure, really fast, and now deprecated... I've used it for log tailing and metrics forwarding extensively and can't recommend it enough if you need to use amqps out.

If you don't need amqps out there are more modern, better supported projects.

Re: From Kafka to ZeroMQ for real-time log aggregation

#86

Post by darkr » Sat, Sep 17, 2016, 6:25 PM UTC

ZenoArrow wrote: post #65

hashmp wrote: post #23

Earlier quoted context omitted.

Use 5 zookeepers, on a separate set of servers. Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well…

Five zookeepers? Seems like a lot. Why five? Is it hard to keep them active? Thanks for the tips.

Zab [the distributed consensus algorithm that powers ZK] shares some similarities with Paxos, and requires a quorum of nodes to be online.

If you want highly available ZK, your choices are 3, 5, 7... nodes, for which you can have 1, 2, or 3 nodes offline at any one time.

If you have one node fully down on a 3 node cluster, and there is even a tiny network blip or partition (as often happens in cloud environments) then you are down.

Re: From Kafka to ZeroMQ for real-time log aggregation

#87

Post by wanderr » Sat, Sep 17, 2016, 6:35 PM UTC

wanderr wrote: post #74

I came up with a very different solution for real time access to logs: tail them to slack. It's not an aggregation solution and doesn't work well if you have chatty logs with nothing to filter on, but if you just want to be notified when things are happening in the logs it's pretty nice and doesn't need any infrastructure. http://wanderr.com/jay/tail-error-logs-to-slack-for-fun-and-...

why the downvote? the article says "Real-time access to server-side logs is what makes backend development palatable in the era of cloud computing. As a developer you want to be able to get real-time feedback from your server side code deployed to the actual execution environment in the cloud, especially during active development or staging." and this is another solution that provides that.

Re: From Kafka to ZeroMQ for real-time log aggregation

#88

Post by _halgari » Sat, Sep 17, 2016, 6:42 PM UTC

ZMQ's default behavior (and in some cases only behavior) of dropping new messages when buffers are full, made it a no-go for my client. We ended up switching away from ZMQ to a more traditional durable queue and ended up saving a ton of code complexity and got a lot of reliability in the process. Having now researched it I can't think of a reason I'd ever use ZMQ again. I'll either use a durable queue when I care about message delivery, or something much more traditional when I don't.

Re: From Kafka to ZeroMQ for real-time log aggregation

#89

Post by tjholowaychuk » Sat, Sep 17, 2016, 7:25 PM UTC

weitzj wrote: post #11

Did you look at nsq.io or NATS?

+1 for NSQ, it's not a magic bullet in terms of scalability but you can get quite far. When I was at Segment we were pushing an easy 2-3B messages per day through it, if not more with message "amplification" internally.

Re: From Kafka to ZeroMQ for real-time log aggregation

#90

Post by smetj » Sat, Sep 17, 2016, 7:29 PM UTC

agentgt wrote: post #50

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…

100K msg/s going through RabbitMQ ... Would you mind commenting on how your Rabbitmq is setup? Is it a cluster? Distributed queue(s)? Synced queues? What kind of exchange? How many queues your messages end up in? (because 1 queue is bound to 1 core), persistent queue? lazy queue? What is the "Consumer utilisation" value when doing 100K msg/s?

I'd be really interested to hear how you can achieve such a thoughput with rabbitmq