I'm a total message queue noob. What are the usecases for them?
I used MQTT but only as a message bus.
From Kafka to ZeroMQ for real-time log aggregation
41–50 of 110 posts
Re: From Kafka to ZeroMQ for real-time log aggregation
#42[deleted]
Re: From Kafka to ZeroMQ for real-time log aggregation
#43To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but rea…
[..] i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations.
ZMQ silently drops messages when subscribers fail or not listening or when buffers fill up, but as they describe later on "access to historical logs", it's much easier to set up separate process/es for just that.
It seems that when shit hits the fan for this reason ZMQ really is a more reliable choice because it's more flexible.
Re: From Kafka to ZeroMQ for real-time log aggregation
#44I'm a total message queue noob. What are the usecases for them? I used MQTT but only as a message bus.
You should have a look at http://zguide.zeromq.org/page:all
It's a great read and describes most scenarios well and easy to understand.
Re: From Kafka to ZeroMQ for real-time log aggregation
#45To me it sounds like Kafka was not understood in full detail (maybe because missing documentation or the high complexity) and they switched to a system they build themselves. Naturally they know in full detail what is going on and can set up the system as needed. I am wondering if working on solving the actual problems with Kafka would have been the better route. I've never used Kafka and i find ZeroMQ great, but rea…
[..] i find ZeroMQ great, but reading that their logging solution does drop log messages is a huge no-go for operations. ZMQ silently drops messages when subscribers fail or not listening or when buffers fill up, but as they describe later on "access to historical logs", it's much easier to set up separate process/es for just that. It seems that when shit hits the fan for this reason ZMQ really is a more reliable cho…
No. When you can't rely on your collected ZMQ logs and need to "access historical logs" by some other means, why use the ZMQ logs at all? You usually don't know that something was not logged.
Also, as he describes in the article, historical logs are scoped out and it is "likely" they they will develop something for those logs in the future. Again it looks like the plan is to use ZMQ and a subscriber to put those logs into logstash. That doesn't solve the problem i mentioned at all. ZMQ may still drop the logs! So, as far as i understand they don't have a plan for reliable logging. Even if they would, they'd have one reliable solution and an unreliable solution. The unreliable ZMQ based approach is probably neat and leads to fancy realtime dashboard stuff, but since it's not a reliable source of information it's not a good solution for operating a system where "babies will die".
Re: From Kafka to ZeroMQ for real-time log aggregation
#46The author correctly points out that he is comparing apples to oranges.
Kafka gives you features that certain systems cannot live without, like on disk persistence (saved my life couple of times) and topics. Filtering messages on the client side like ZeroMQ does it not an option in many cases, just think about security. I think Kafka has a long way to go before it can be used as a general message queue (many features are not there yet like visibility timeout for example) but if you can manage Zookeeper and have means to work with it (somebody understands it and knows its quirks) it can provide a reliable platform for distributing a large number of messages with low latency and high throughput, just like it does at LinkedIN.
Re: From Kafka to ZeroMQ for real-time log aggregation
#47With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - m…
it is trivially easy for any node to broadcast its IP address to the whole network periodically (in my case every 2 seconds) using a separate thread and UDP. Using this technique I have rock solid ZeroMQ topology that reconnects with max downtime about 2.5 seconds (because I broadcast every 2 sdconds) for any single node failure. I agree that this functionality could be better implemented in zmq but using this simple technique, the rest of zmq becomes amazing. In Python:
import socket
import time
cs = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
cs.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
cs.setsockopt(SOL_SOCKET, SO_BROADCAST, 1)
while True:
cs.sendto('Node ID', ('255.255.255.255', 54545))
time.sleep(4)
import socket
s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('',54545))
m=s.recvfrom(1024)
print m[0]
Re: From Kafka to ZeroMQ for real-time log aggregation
#48I'm a total message queue noob. What are the usecases for them? I used MQTT but only as a message bus.
From my point of view, the main things behind message queues (Not zMQ specifically) is guaranteed delivery, persistence, multiple-message atomicity, message passing/forwarding, and sometimes guaranteed message ordering. Other than that, all it does is facilitate communication between different actors.
Nothing magical/weird about it, just depends on whether or not you've got a nail to hammer with your MQ-hammer.
Re: From Kafka to ZeroMQ for real-time log aggregation
#49With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - m…
it is trivially easy for any node to broadcast its IP address to the whole network periodically (in my case every 2 seconds) using a separate thread and UDP. Using this technique I have rock solid ZeroMQ topology that reconnects with max downtime about 2.5 seconds (because I broadcast every 2 sdconds) for any single node failure. I agree that this functionality could be better implemented in zmq but using this simple…
This would be great, but I don't think it works on AWS -- I don't think they support broadcast.
Re: From Kafka to ZeroMQ for real-time log aggregation
#50I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics.
And our log messages are ridiculously big at times (15k to as big as 50k).
Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k messages/s when embarrassingly had debug turned on in production and RabbitMQ did not fail but Elastic Search and sadly Flume did as well (I tried to get rid of flume with a custom Rust AMQP to Elastic Search client but at the time had some bugs with the libraries.. Maybe I will recheck out Mozilla Heka someday).
There is this sort of beating of the developer chest with a lot of tech companies.. that hey listen we are ultra important and we are dealing with ridiculously traffic and we need ultra high performance. Please tell/show me these numbers.... Or maybe stop logging crap you don't need to log.
Or maybe I'm wrong and we should log absolutely everything and Auth0 made the right choice given their needs (lets assume they have millions of messages a second), I still think I could make a sharded RabbitMQ go pretty far.
This goes with other technology as well. You don't need to pick hot glamorous NoSQL when Postgresql or MySQL and a tiny bit of engineering will get the job done just fine particularly when mature solutions give you such many things free out of the box (RabbitMQ gives you a ton of stuff like a cool admin UI and routing that you would have to build in ZeroMQ).