Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Confluent, a company for Apache Kafka and realtime data

linkedin.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Confluent, a company for Apache Kafka and realtime data

#21

Post by gelliott »

beagle3 wrote: post #18

Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a b…

Can your endpoints log to syslog so they can be forwarded on to a central collector?

Re: Confluent, a company for Apache Kafka and realtime data

#22

Post by beagle3 »

lrm242 wrote: post #20
beagle3 wrote: post #18

Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a b…

autossh + ssh -C + tail -F ? kafka sounds like an enormous overkill. If you want to store the logs locally while tailing, just add in a tee.

Both this solution (and easytiger's above) work if disconnects happen rarely and for short times.

But it my case, I have "30 minutes on, 5 minutes off, 90 minutes on, 90 minutes off, 2 minutes on" kind of situations, in which anything that doesn't track what was already transferred and what wasn't, will lose data. (zeromq's buffers also have limited capacity and/or are tied to a process on the other side - if it restarts, buffers are gone).

Re: Confluent, a company for Apache Kafka and realtime data

#24

Post by mwarkentin »

beagle3 wrote: post #18

Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a b…

Sounds like a job for logstash: http://logstash.net

If you don't mind using a 3rd party service, you could look into using Papertrail, Loggly, etc.

Re: Confluent, a company for Apache Kafka and realtime data

#25

Post by beagle3 »

gelliott wrote: post #21
beagle3 wrote: post #18

Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a b…

Can your endpoints log to syslog so they can be forwarded on to a central collector?

Probably. Do you know of a syslog/forwarder that deals properly with intermittent connections? (i.e., available for an hour, then gone for an hour, then available again, etc.)

Re: Confluent, a company for Apache Kafka and realtime data

#26

Post by beagle3 »

mwarkentin wrote: post #24
beagle3 wrote: post #18

Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a b…

Sounds like a job for logstash: http://logstash.net If you don't mind using a 3rd party service, you could look into using Papertrail, Loggly, etc.

3rd party is not an option. I'll have to look at logstash, thanks.

Re: Confluent, a company for Apache Kafka and realtime data

#27

Post by possibilistic »

mlhamel wrote: post #23

I'm a bit confused. The more i read the doc, the more i feel it look Just Another Queue system. I don't really see the difference between Kafka and let's say RabbitMQ, Celery or also statsd...

There are differences, the biggest of which is throughput. Kafka can handle incredible load. The messaging semantics are also a bit different. Here's a pretty good comparison:

http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable...

Re: Confluent, a company for Apache Kafka and realtime data

#28

Post by b0ti »

beagle3 wrote: post #25
gelliott wrote: post #21

Earlier quoted context omitted.

Can your endpoints log to syslog so they can be forwarded on to a central collector?

Probably. Do you know of a syslog/forwarder that deals properly with intermittent connections? (i.e., available for an hour, then gone for an hour, then available again, etc.)

Not sure about logstash but a lot of its users use NXLog (http://nxlog.org) as a shipper since it has a much lower resource footprint as there is no java and ruby in there. (I'm affiliated with NXLog.)

Re: Confluent, a company for Apache Kafka and realtime data

#29

Post by imperialWicket »

easytiger wrote: post #19
beagle3 wrote: post #18

Good luck to the Confluent guys! There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution: I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent). I just want to consistently tail their logs in a b…

How about a tiny forwarder on each device. It fopens the file and reads to the end, when the file is appended to it will be able to continue reading. Packet up each line and forward to you server using zeromq PUSH/PULL. If you are disconnected they queue up and send when zeromq automagically reconnects.

Since logstash was already mentioned, it's worth putting https://github.com/elasticsearch/logstash-forwarder here, as the logstash team built it as a solution for logstash-type needs on systems that may not be able to support logstash itself.

Re: Confluent, a company for Apache Kafka and realtime data

#30

Post by otterley »

beagle3 wrote: post #25
gelliott wrote: post #21

Earlier quoted context omitted.

Can your endpoints log to syslog so they can be forwarded on to a central collector?

Probably. Do you know of a syslog/forwarder that deals properly with intermittent connections? (i.e., available for an hour, then gone for an hour, then available again, etc.)

Syslog-NG PE works well for this. It can be configured to use a per-destination disk buffer, so that if the destination goes offline, messages will queue until they can be sent again.
Post reply on HN