From Kafka to ZeroMQ for real-time log aggregation

#1

Post by janczukt » Sat, Sep 17, 2016, 5:33 AM UTC

From Kafka to ZeroMQ for real-time log aggregation
tomasz.janczuk.org

Re: From Kafka to ZeroMQ for real-time log aggregation

#2

Post by jvoorhis » Sat, Sep 17, 2016, 5:59 AM UTC

2015

Re: From Kafka to ZeroMQ for real-time log aggregation

#3

Post by htn » Sat, Sep 17, 2016, 6:07 AM UTC

FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io on AWS and also Azure, GCE and DigitalOcean.

But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.

Re: From Kafka to ZeroMQ for real-time log aggregation

#4

Post by janczukt » Sat, Sep 17, 2016, 6:11 AM UTC

htn wrote: post #3

FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io on AWS and also Azure, GCE and DigitalOcean. But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.

We need an on-premise and cloud story, so cloud only solutions did not cut it for us.

Re: From Kafka to ZeroMQ for real-time log aggregation

#5

Post by siscia » Sat, Sep 17, 2016, 6:27 AM UTC

Did you consider MQTT? Sound to me a more natural choice.

Re: From Kafka to ZeroMQ for real-time log aggregation

#6

Post by bdowling » Sat, Sep 17, 2016, 6:28 AM UTC

But why ZeroMQ and not nanomsg?

Re: From Kafka to ZeroMQ for real-time log aggregation

#7

Post by Nimimi » Sat, Sep 17, 2016, 6:29 AM UTC

You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default.

Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.

Re: From Kafka to ZeroMQ for real-time log aggregation

#8

Post by PieterH » Sat, Sep 17, 2016, 6:34 AM UTC

janczukt wrote: post #4

htn wrote: post #3

FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io on AWS and also Azure, GCE and DigitalOcean. But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.

We need an on-premise and cloud story, so cloud only solutions did not cut it for us.

The article is a little old. How has the system run since you deployed it? Do you have any interesting figures?

Re: From Kafka to ZeroMQ for real-time log aggregation

#9

Post by ZenoArrow » Sat, Sep 17, 2016, 6:35 AM UTC

janczukt wrote: post #4

htn wrote: post #3

FWIW, you can get Kafka packaged as a fully managed and HA service from https://aiven.io on AWS and also Azure, GCE and DigitalOcean. But if the Auth0 runs their entire operations on AWS, maybe Kinesis would have been a more natural transition.

We need an on-premise and cloud story, so cloud only solutions did not cut it for us.

I'm in a similar boat. I'm hoping to propose Kafka to help with some data replication and consolidation tasks, but it has to be both on-premise and as low maintenance as possible (low maintenance in the sense of the work local developers would do).

To anyone reading this with Kafka experience, do you have any tips/advice when it comes to maintaining a Kafka service?

Re: From Kafka to ZeroMQ for real-time log aggregation

#10

Post by oblio » Sat, Sep 17, 2016, 6:35 AM UTC

Nimimi wrote: post #7

You can deploy Kafka using DC/IO and it takes care about HA for you. DC/IO is quickly becoming the go-to solution for database deployments. ArangoDB even recommends it as default. Now about Kafka vs ZeroMQ: you want Kafka if you cannot tolerate the loss of even a single message. The append-only log with committed reader positions is a perfect fit for that.

Do you mean this? https://dcos.io/get-started/ aka DC/OS?

From what I can see it doesn't really support database deployments except for ArangoDB and Cassandra.