You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default. Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.
I wish there was more information on how much work it takes to maintain a DC/OS setup. All the marketing makes it out to be the easiest thing in the world.
I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…
Agree with everything you said, just curious is the GT in your name for Georgia Tech?
I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…
Agree with everything you said, just curious is the GT in your name for Georgia Tech?
Yep... I was the first agent@cc.gatech.edu circa 99-04 (I wonder who owns it now). Advance apologies if you knew me then... I was an inconsiderate a$$hole at the time. I'm still an a$$hole but less inconsiderate.
The ZeroMQ documentation is pretty up front about the need for you to build those pieces yourself. It would appear you chose the wrong tool for your requirements.
ZeroMQ wants to be a neutral wrapper in any language but in the end its a C++ library enforcing C++ concepts. You can't map in a straightforward way process concepts from C++ to other languages (also the OS and VM sits in between that). In the end its just mapping programming logic to state machines. There are much better ways to do this and end up with something much more powerful (with first class meta-programming)…
ZeroMQ has native stacks in Java, C#, Erlang, C. Also C++.
With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - m…
it is trivially easy for any node to broadcast its IP address to the whole network periodically (in my case every 2 seconds) using a separate thread and UDP. Using this technique I have rock solid ZeroMQ topology that reconnects with max downtime about 2.5 seconds (because I broadcast every 2 sdconds) for any single node failure. I agree that this functionality could be better implemented in zmq but using this simple…
it is trivially easy for any node to broadcast its IP address to the whole network periodically (in my case every 2 seconds) using a separate thread and UDP. Using this technique I have rock solid ZeroMQ topology that reconnects with max downtime about 2.5 seconds (because I broadcast every 2 sdconds) for any single node failure. I agree that this functionality could be better implemented in zmq but using this simple…
This would be great, but I don't think it works on AWS -- I don't think they support broadcast.
My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query.
Agree with everything you said, just curious is the GT in your name for Georgia Tech?
Yep... I was the first agent@cc.gatech.edu circa 99-04 (I wonder who owns it now). Advance apologies if you knew me then... I was an inconsiderate a$$hole at the time. I'm still an a$$hole but less inconsiderate.
No apologies needed, lol. I went to Clemson. Just saw it and was curious.
Maybe TANK ( https://github.com/phaistos-networks/TANK ) would have been a good alternative on there. No features parity with Kafka, but setting it up is a matter of running one binary and creating a few topics, and it is faster than Kafka for produce/consume operations. (disclosure: I am involved in its development).
Did you ever try running 5 ZK's in the ensemble? 3 is the absolute minimum to survive a single machine failure. If you are having trouble with availability, it seems natural to increase your safety factor there. I was surprised by the contrasting sense of importance of delivery guarantees in the article. At the start, losing a message was akin to the death of a child. At the end, shrug . Now every single machine fail…
They said availability was "death of a child", not dropping log messages. The trade-off they've made here in terms of being available with some potential loss of visibility is the right one. The system overall is clearly simpler and simpler systems have simpler failure modes and so it is easier to add mitigation components on top that can recover from those failure modes to guarantee higher uptime. I've never heard a…
I've managed a production kafka cluster at my current gig for over two years. It has been easy and trouble free with the exception of one incident, which was ultimately our fault.