Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From Kafka to ZeroMQ for real-time log aggregation

tomasz.janczuk.org

Post reply on HN
Print view View original

101–110 of 110 posts

Re: From Kafka to ZeroMQ for real-time log aggregation

#101

Post by cbsmith »

hashmp wrote: post #23
ZenoArrow wrote: post #9

Earlier quoted context omitted.

I'm in a similar boat. I'm hoping to propose Kafka to help with some data replication and consolidation tasks, but it has to be both on-premise and as low maintenance as possible (low maintenance in the sense of the work local developers would do). To anyone reading this with Kafka experience, do you have any tips/advice when it comes to maintaining a Kafka service?

Use 5 zookeepers, on a separate set of servers. Use configuration management such as chef to allow you to quickly build new nodes and to roll out changes accross the cluster. You will need to make tweaks. The chef Kafka cookbook which is the top result on Google has means of coordinating restarts of brokers accross the cluster. Use consul as a locking mechanism for this. You could use zookeeper, but consul works well…

Five kind of kills performance compared to three, and doesn't map well into AWS, where you generally have 3 or 4 AZ's. I tend to go with three but make sure you've got fully automated responses towards failures.

Re: From Kafka to ZeroMQ for real-time log aggregation

#102

Post by AYBABTME »

buster wrote: post #99
AYBABTME wrote: post #97

Earlier quoted context omitted.

If you buffer to disk, the same problem will eventually show up. Queues (in memory, on disk, anywhere) are all ultimately bounded, and when they are full, you have 2 choices: block or drop. Somehow you need to make the choice, there's no getting away from it. If you don't make the choice consciously, say by assuming that you can buffer to disk and avoid the problem, at some point you'll fill up your disks and your sy…

That's why you aggregate logs in a central service. I was writing about sending logs to a central service and not about how your disks fill up with more logs. There is log rotation for that and usually your logs will have been sent way before any log rotates. If your log rotation deletes logs before you aggregated them or if you let your disks fill up with logs you have a much bigger problem you should fix, of course…

That's again the same problem; if your centralized service isn't reachable for whatever reason, your nodes can buffer for a while (in memory or on disk) but eventually the problem always will boil down to 'drop' or 'block'. However you construct it, somewhere you need to make that call. They made the call to drop logs, it's totally fine.

Re: From Kafka to ZeroMQ for real-time log aggregation

#103

Post by buster »

AYBABTME wrote: post #102
buster wrote: post #99

Earlier quoted context omitted.

That's why you aggregate logs in a central service. I was writing about sending logs to a central service and not about how your disks fill up with more logs. There is log rotation for that and usually your logs will have been sent way before any log rotates. If your log rotation deletes logs before you aggregated them or if you let your disks fill up with logs you have a much bigger problem you should fix, of course…

That's again the same problem; if your centralized service isn't reachable for whatever reason, your nodes can buffer for a while (in memory or on disk) but eventually the problem always will boil down to 'drop' or 'block'. However you construct it, somewhere you need to make that call. They made the call to drop logs, it's totally fine.

If your central infrastructure is down for many days you have other problems. Buffering logs on disks. Rotating and zipping files doesn't take much space. You can buffer a lot before you run into trouble. That's my point. You sound like buffering on the node is only possible for seconds whereas in real world scenarios log files are written for days, weeks or even months. Even in very large deployments and lots of logs. You would make the decision to don't use this advantage and throw logs away for no good reason.

Re: From Kafka to ZeroMQ for real-time log aggregation

#104

Post by macns »

buster wrote: post #45
macns wrote: post #43

Earlier quoted context omitted.

[..] i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. ZMQ silently drops messages when subscribers fail or not listening or when buffers fill up, but as they describe later on "access to historical logs", it's much easier to set up separate process/es for just that. It seems that when shit hits the fan for this reason ZMQ really is a more reliable cho…

No. When you can't rely on your collected ZMQ logs and need to "access historical logs" by some other means, why use the ZMQ logs at all? You usually don't know that something was not logged. Also, as he describes in the article, historical logs are scoped out and it is "likely" they they will develop something for those logs in the future. Again it looks like the plan is to use ZMQ and a subscriber to put those logs…

AFAIK there's no ZMQ logs, ZMQ is just messaging patterns over various protocols, logging is your responsibility.

Re: From Kafka to ZeroMQ for real-time log aggregation

#105

Post by agentgt »

smetj wrote: post #90
agentgt wrote: post #50

I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…

100K msg/s going through RabbitMQ ... Would you mind commenting on how your Rabbitmq is setup? Is it a cluster? Distributed queue(s)? Synced queues? What kind of exchange? How many queues your messages end up in? (because 1 queue is bound to 1 core), persistent queue? lazy queue? What is the "Consumer utilisation" value when doing 100K msg/s? I'd be really interested to hear how you can achieve such a thoughput with…

I'm not sure how much I can help because I didn't setup the RMQ cluster so I don't know the configuration details but I know it is fairly powerful (its also partly why I can't entirely be critical of Auth0 because ZMQ is probably far less expensive infrastructure wise).

I do know we use multiple queues and even exchanges (and I did not know about the one core to queue).

A simple googling shows though folks have achieved far greater throughput[1] than 100k (and by the way this wasn't sustained.. it was spikes).

[1]: https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-m...

Re: From Kafka to ZeroMQ for real-time log aggregation

#106

Post by hinkley »

AYBABTME wrote: post #66
buster wrote: post #30

To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but rea…

One has only two choices in those situations: drop logs or block receiving more logs. Given their availability requirements, I don't think that blocking is a viable choice. So dropping logs seems to be the only sane choice here. There's no other alternative really so I'm not sure about the consternation.

I come to this discussion much like one would sit down at a bar and find their two friends are deep in a coding discussion they didn't hear the beginning of.

How do you end up with a system design that can generate logs so fast that you can't keep up? It seems to be that some fundamental element of capacity planning was missed long ago and we're trying to fix the symptoms and not the cause.

If I have a geographically distributed system, I'm going to have bandwidth and latency issues if I try this, sure. But why do I care? A request to the SF data center shouldn't involve the Munich data center. That is, if I care about response times, and if I don't care about that, then why do I care about instantaneous log availability?

I think sometimes we get so bored with the problems we have that we invent new things to get upset about. Or management does, which is always worse.

Re: From Kafka to ZeroMQ for real-time log aggregation

#107

Post by oldmanjay »

vegabook wrote: post #79
oldmanjay wrote: post #71

Earlier quoted context omitted.

What if they crash without registering an exit?

They broadcast every 2 seconds. No heartbeat = dead.

So there's more machinery waiting in the wings, not just the bog-simple query. Presumably you've barely described all of the mechanisms, and I won't bother socratically making the point that handwaving complexity away isn't the same thing as simplifying. I'll simply state it.

Re: From Kafka to ZeroMQ for real-time log aggregation

#108

Post by vegabook »

oldmanjay wrote: post #107
vegabook wrote: post #79

Earlier quoted context omitted.

They broadcast every 2 seconds. No heartbeat = dead.

So there's more machinery waiting in the wings, not just the bog-simple query. Presumably you've barely described all of the mechanisms, and I won't bother socratically making the point that handwaving complexity away isn't the same thing as simplifying. I'll simply state it.

The above code is wrapped in a thread and runs non stop in each node. It's really not very complicated. It's basically sticking a def around the top block of code and then thread.start() in the main. It's extremely cheap because it sends then sleeps for a few seconds. Any node than then listen to the broadcasts on a known port (but no IP address needed) and will know exactly what the state of the system is and all the IPs. Then in the main loop you just make sure that you've gotten a recent pulse otherwise you reconnect. If you're using zmq it's just another short poller entry amongst the others that you will already inevitably have. Literally a couple of lines. It allows you to bring nodes up and down at will for robustness and scalability.

I think it's much less complex than running an entire database node just for this, which btw will also require you constantly to poll, and will require you to bring in an (often heavyweight) client library into each node too, as opposed to standard-library sockets which if you're running multinode you're almost certainly already importing. If you're looking for "simple" distributed computing my sense is that that has yet to be invented.

Re: From Kafka to ZeroMQ for real-time log aggregation

#109

Post by [deleted] »

vegabook wrote: post #108
oldmanjay wrote: post #107

Earlier quoted context omitted.

So there's more machinery waiting in the wings, not just the bog-simple query. Presumably you've barely described all of the mechanisms, and I won't bother socratically making the point that handwaving complexity away isn't the same thing as simplifying. I'll simply state it.

The above code is wrapped in a thread and runs non stop in each node. It's really not very complicated. It's basically sticking a def around the top block of code and then thread.start() in the main. It's extremely cheap because it sends then sleeps for a few seconds. Any node than then listen to the broadcasts on a known port (but no IP address needed) and will know exactly what the state of the system is and all th…

[deleted]
Post reply on HN