Did you look at nsq.io or NATS?
From Kafka to ZeroMQ for real-time log aggregation
11–20 of 110 posts
Re: From Kafka to ZeroMQ for real-time log aggregation
#12You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default. Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.
Do you mean this? https://dcos.io/get-started/ aka DC/OS? From what I can see it doesn't really support database deployments except for ArangoDB and Cassandra.
Ok, database was the wrong term. Maybe "things that big data companies use" or something.
Re: From Kafka to ZeroMQ for real-time log aggregation
#13FYI, Kafka doesn't need to fetch from disk every time as it caches the logs pretty aggressively, as long as you have enough memory.
Running Zk and Kafka on the same nodes is likely not the best thing.
Re: From Kafka to ZeroMQ for real-time log aggregation
#14Probably should have been running ZK and Kafka queues separate to CoreOS/container shenanigans.
If deployed using the Netflix co-processes both are very durable.
Re: From Kafka to ZeroMQ for real-time log aggregation
#15Earlier quoted context omitted.
We need an on-premise and cloud story, so cloud only solutions did not cut it for us.
The article is a little old. How has the system run since you deployed it? Do you have any interesting figures?
It continues to run beautifully. Since we rolled it out back in 2015 we had zero issues with real time logging. I have particularly fond memories of the first week after rollout, it felt like vacation. I finally could get some sleep.
Re: From Kafka to ZeroMQ for real-time log aggregation
#16FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io on AWS and also Azure, GCE and DigitalOcean. But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.
Eh, Kinesis has some pretty significant trade-offs to know about if you are comparing it with Kafka (e.g. data retention time and write latency).
Re: From Kafka to ZeroMQ for real-time log aggregation
#17Did you ever try running 5 ZK's in the ensemble? 3 is the absolute minimum to survive a single machine failure. If you are having trouble with availability, it seems natural to increase your safety factor there.
I was surprised by the contrasting sense of importance of delivery guarantees in the article. At the start, losing a message was akin to the death of a child. At the end, shrug. Now every single machine failure (or even ømq process restart) failure will lose you log messages stored in memory :(.
Glad to hear you found a solution that worked for you though! Would love to hear about difficulties you had with the new system, in particular adding brokers.
Re: From Kafka to ZeroMQ for real-time log aggregation
#18Did you ever try running 5 ZK's in the ensemble? 3 is the absolute minimum to survive a single machine failure. If you are having trouble with availability, it seems natural to increase your safety factor there. I was surprised by the contrasting sense of importance of delivery guarantees in the article. At the start, losing a message was akin to the death of a child. At the end, shrug . Now every single machine fail…
They said availability was "death of a child", not dropping log messages. The trade-off they've made here in terms of being available with some potential loss of visibility is the right one. The system overall is clearly simpler and simpler systems have simpler failure modes and so it is easier to add mitigation components on top that can recover from those failure modes to guarantee higher uptime.
I've never heard anyone say managing a production Kafka cluster was easy or simple. Well, anyone who has had to actually maintain such clusters hasn't said it anyway.
Re: From Kafka to ZeroMQ for real-time log aggregation
#19But why ZeroMQ and not nanomsg?
See http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.
Re: From Kafka to ZeroMQ for real-time log aggregation
#20But why ZeroMQ and not nanomsg?
See http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.
From the blog: "Crazy Idea: Clone nanomsg, move to zeromq organization, relicense as MPL, support ZMTP, only new socket types and expose CZMQ API."
Did that ever happen? I still like the idea behind nanomsg.