Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From Kafka to ZeroMQ for real-time log aggregation

tomasz.janczuk.org

Post reply on HN
Print view View original

91–100 of 110 posts

Re: From Kafka to ZeroMQ for real-time log aggregation

#91

Post by smetj »

gerakinis wrote: post #83
packetized wrote: post #63

Earlier quoted context omitted.

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

Are you using HA functionality and also on disk backing? These two things bring down performance roughly 5-10x and are mostly required for situations that can't afford message loss. I still like the rabbitmq solution, it is my own, but i've found it takes more hardware than you are suggesting.

Yes that's my experience as well.

Re: From Kafka to ZeroMQ for real-time log aggregation

#92

Post by dkarapetyan »

fauigerzigerk wrote: post #61
_qc3o wrote: post #18

Earlier quoted context omitted.

They said availability was "death of a child", not dropping log messages. The trade-off they've made here in terms of being available with some potential loss of visibility is the right one. The system overall is clearly simpler and simpler systems have simpler failure modes and so it is easier to add mitigation components on top that can recover from those failure modes to guarantee higher uptime. I've never heard a…

>They said availability was "death of a child", not dropping log messages. True, but it appears to me that availability problems and dropped log messages often have the same root cause - network issues. So whenever they do have availability issues (and dying babies) they won't be able to investigate properly because log messages are being lost as well. That's obviously a very general observation. It may well be that…

It would be quite simple to have a two tiered approach to the logging problem since they have separated it into 2 components. One can just write and ship files while the other is what they have described in terms of providing real time streaming.

So the question then becomes what are the failures modes of their logging setup in terms of misbehaving clients? I don't know how kafka handles misbehaving clients. I suspect it would lead to global effects and slowdown of the entire cluster because of 1 or 2 misbehaving clients whereas in the current set up local misbehavior will be localized to the nearest aggregator dropping messages. Simple memory usage and other kinds of monitoring can then be used to find these issues and then mitigate them accordingly.

This is still a heck of lot simpler setup than using kafka and worrying about all sorts of weird distributed system failure modes. I'm sure kafka got them started initially but continuing to use it is like using a sledgehammer to kill a fly. For the use case they have this setup is the correct one and migrating to kafka if it becomes necessary will be possible. So in my view this is proper engineering. They've made all the right trade-offs instead of just chasing fads and trends.

Re: From Kafka to ZeroMQ for real-time log aggregation

#94

Post by TheHydroImpulse »

im_down_w_otp wrote: post #84
TheHydroImpulse wrote: post #13

FYI, Kafka doesn't need to fetch from disk every time as it caches the logs pretty aggressively, as long as you have enough memory. Running Zk and Kafka on the same nodes is likely not the best thing.

Why? I would think that, as long as there wasn't massive I/O contention between the two, that co-locating Kafka and Zookeeper on the same machines would mitigate a whole massive class of weird edge cases by removing one of the failure modes; the network boundary between the two critical components. Though for my part I still don't understand why Zookeeper wasn't built as a library to add distributed strongly consiste…

But co-locating them won't actually remove a class of errors because Zk is not HA. The Kafka brokers need to communicate with the leader in the Zk cluster.

If we have K1,Z1 -- K2,Z2 -- K3,Z3 -- and one node goes down, you've now taken down both a broker and a Zk node. Remember, the brokers don't care about connecting to any Zk node, they want the leader. So you aren't gaining any more fault tolerant by co-locating them.

If there's a network partition between the leader Zk node and other nodes, the local Kafka broker won't actually be able to do much because the Zk cluster will elect a new leader, on another node, so again, you aren't gaining anything.

Moreover, you're now tying the scalability of Kafka with Zk. Zk doesn't scale linearly, so there's only so many nodes you may have in a cluster. Kafka, on the other hand, scales linearly. So if you're colocating them and you have to bump up Kafka, do you still start up Zk for those nodes (but they don't actually join the cluster)? You're now special casing and adding more edge cases.

Re: From Kafka to ZeroMQ for real-time log aggregation

#95

Post by buster »

AYBABTME wrote: post #66
buster wrote: post #30

To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but rea…

One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a viable choice. So dropping logs seems to be the only sane choice here. There's no other alternative really so I'm not sure about the consternation.

What? No. You can just save logs on the disk and buffer them. Just dropping logs or blocking because some network resource is not available are both terrible choices and that's not how logging worked for the last decades. Throwing away logs ist a major step backwards.

Re: From Kafka to ZeroMQ for real-time log aggregation

#96

Post by vegabook »

mej10 wrote: post #49
vegabook wrote: post #47

Earlier quoted context omitted.

it is trivially easy for any node to broadcast its IP address to the whole network periodically (in my case every 2 seconds) using a separate thread and UDP. Using this technique I have rock solid ZeroMQ topology that reconnects with max downtime about 2.5 seconds (because I broadcast every 2 sdconds) for any single node failure. I agree that this functionality could be better implemented in zmq but using this simple…

This would be great, but I don't think it works on AWS -- I don't think they support broadcast.

That's very interesting and I did not know that. I happen to run my own modest (12-node) cluster, and had planned to move to cloud if I grew beyond 20 nodes. The fact that I would not own the transport layer in my case is quite severely problematic. Going to have to investigate if this is the case at Azure and the rest too. Can't I partition my instances into a VPN and UDP broadcast within that? If anybody has any insight, very happy to hear about it.

Re: From Kafka to ZeroMQ for real-time log aggregation

#97

Post by AYBABTME »

buster wrote: post #95
AYBABTME wrote: post #66

Earlier quoted context omitted.

One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a viable choice. So dropping logs seems to be the only sane choice here. There's no other alternative really so I'm not sure about the consternation.

What? No. You can just save logs on the disk and buffer them. Just dropping logs or blocking because some network resource is not available are both terrible choices and that's not how logging worked for the last decades. Throwing away logs ist a major step backwards.

If you buffer to disk, the same problem will eventually show up. Queues (in memory, on disk, anywhere) are all ultimately bounded, and when they are full, you have 2 choices: block or drop. Somehow you need to make the choice, there's no getting away from it.

If you don't make the choice consciously, say by assuming that you can buffer to disk and avoid the problem, at some point you'll fill up your disks and your system will block: you'll have unknowingly picked the "block" option. If you decide to rotate logs and delete old rotations when too many logs are present, then you're picking the "drop" option...

Re: From Kafka to ZeroMQ for real-time log aggregation

#98

Post by fauigerzigerk »

dkarapetyan wrote: post #92
fauigerzigerk wrote: post #61

Earlier quoted context omitted.

>They said availability was "death of a child", not dropping log messages. True, but it appears to me that availability problems and dropped log messages often have the same root cause - network issues. So whenever they do have availability issues (and dying babies) they won't be able to investigate properly because log messages are being lost as well. That's obviously a very general observation. It may well be that…

It would be quite simple to have a two tiered approach to the logging problem since they have separated it into 2 components. One can just write and ship files while the other is what they have described in terms of providing real time streaming. So the question then becomes what are the failures modes of their logging setup in terms of misbehaving clients? I don't know how kafka handles misbehaving clients. I suspec…

>It would be quite simple to have a two tiered approach to the logging problem since they have separated it into 2 components. One can just write and ship files while the other is what they have described in terms of providing real time streaming.

Yes, they absolutely could do that, but they apparently don't. And maybe that's because they would lose a lot of the simplicity they won by ditching Kafka.

Anyway, I didn't want to defend Kafka specifically. The one time I considered it, I ended up not using it because it seemed too heavy weight for my use case in terms of memory usage and complexity.

Re: From Kafka to ZeroMQ for real-time log aggregation

#99

Post by buster »

AYBABTME wrote: post #97
buster wrote: post #95

Earlier quoted context omitted.

What? No. You can just save logs on the disk and buffer them. Just dropping logs or blocking because some network resource is not available are both terrible choices and that's not how logging worked for the last decades. Throwing away logs ist a major step backwards.

If you buffer to disk, the same problem will eventually show up. Queues (in memory, on disk, anywhere) are all ultimately bounded, and when they are full, you have 2 choices: block or drop. Somehow you need to make the choice, there's no getting away from it. If you don't make the choice consciously, say by assuming that you can buffer to disk and avoid the problem, at some point you'll fill up your disks and your sy…

That's why you aggregate logs in a central service. I was writing about sending logs to a central service and not about how your disks fill up with more logs. There is log rotation for that and usually your logs will have been sent way before any log rotates. If your log rotation deletes logs before you aggregated them or if you let your disks fill up with logs you have a much bigger problem you should fix, of course.

Re: From Kafka to ZeroMQ for real-time log aggregation

#100

Post by cbsmith »

asasidh wrote: post #73

So you used Kafka for something that should have been handled by a MQTT or ZeroMQ in the first place ?

MQTT is just a protocol, so not sure how that helps.

0MQ doesn't sound like it is the right solution either, but yeah... often you pick the wrong tool and learn something in the process.

Post reply on HN