Confluent, a company for Apache Kafka and realtime data

In the same way that Hadoop is starting to feel outdated, but HDFS doesn't seem to be going anywhere -- I think we'll see a lot of innovation in stream processing frameworks in the next few years, but Kafka will just keep on going.

There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution:

I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent).

I just want to consistently tail their logs in a bandwidth-efficient, connection-drop resistant way; and I can't find any standard thing that does this.

Kafka would fit the bill in general, but would require a lot of work (reading textual logs into kafka, querying kafka for new stuff across connection, reading from kafka and writing to text files) - and I'm not sure how well it deals with dropped connections.

My existing solution is to rsync the log directories (--append, --inplace) as infrequently as I can from an operational view, which is 1 minute. It is relatively bandwidth efficient (although could be much better), robust with respect to connection issues, and generally works.

However, it is less efficient than it could be: if directories have a lot of files, like /var/log often does, there's a lot of sync overhead. The delay is 1 minute instead of a couple of seconds (which is what you would get with a simple "tail -f" through a TCP connection), and it doesn't play well with common log rotation schemes (though that's relatively easy to work around).

Anyone has a better solution, kafkaesque or otherwise?

kafka sounds like an enormous overkill. If you want to store the logs locally while tailing, just add in a tee.