Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing LoggerFS (2013)

developer.rackspace.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Introducing LoggerFS (2013)

#31

Post by seanewest »

achillean wrote: post #24
mmastrac wrote: post #19

Earlier quoted context omitted.

My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that i…

I actually wrote exactly the same thing years ago (2007), even had the same name :) http://sourceforge.net/projects/loggerfs/

Put it on github -- people might send some Pull Requests.

Re: Introducing LoggerFS (2013)

#34

Post by Twirrim »

e12e wrote: post #12
staunch wrote: post #5

My first concern is what happens to daemons when they can't write to the FUSE mount because this program is down or misbehaving. I believe many daemons would misbehave, block, or die. That scares me but maybe I'm being paranoid. I think all the arguments against log shippers are pretty weak, workarounds are simple, especially if the alternative introduces any instability.

Can hardly be worse than in the event of a full disk/partition?

It talks about buffering in memory. That could be bad.. I'm assuming/hoping they'd put some kind of upper bound on it, or you could easily take out a server.

Re: Introducing LoggerFS (2013)

#35

Post by Rantenki »

e12e wrote: post #26
ams6110 wrote: post #23

Earlier quoted context omitted.

How many logging frameworks don't already support syslog?

Probably none, given: "echo hi|logger".

Take with a grain of salt, since it's anecdotal, but I have seen logger hang when syslog rotates. Logger is really only 100% reliable when you are piping it the output of a command that runs in a bounded amount of time. If you have a real daemon it's always better to use syslog directly, and an inability to use syslog is maybe an indication of the quality of the daemon in question.

Re: Introducing LoggerFS (2013)

#36

Post by cbr »

ithkuil wrote: post #22
e12e wrote: post #12

Earlier quoted context omitted.

Can hardly be worse than in the event of a full disk/partition?

The failure more is very different. On log partition disk full log writes will fail, which you favourite logging library might decide to ignore. A misbehaving FUSE daemon can hang applications that write files and assume it's a local disk. A smart logger can have a writer thread and a buffer and decide to drop log lines if the logger is too slow. But LoggerFS is meant to be a drop in solution for legacy code, so the…

You could set up a watchdog.

Re: Introducing LoggerFS (2013)

#37

Post by derefr »

mmastrac wrote: post #19
mmahemoff wrote: post #17

A little light on detail. Can someone please explain exactly how this works? The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue. Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to un…

My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that i…

Presuming your daemons rotate their logs, what's the benefit of this over mounting a plain-old tmpfs over your logs folder? Either way, the shipper reads virtual files that are actually stored in memory, and which eventually get purged. It's just the thresholds that are different.

Re: Introducing LoggerFS (2013)

#38

Post by rdtsc »

cbr wrote: post #36
ithkuil wrote: post #22

Earlier quoted context omitted.

The failure more is very different. On log partition disk full log writes will fail, which you favourite logging library might decide to ignore. A misbehaving FUSE daemon can hang applications that write files and assume it's a local disk. A smart logger can have a writer thread and a buffer and decide to drop log lines if the logger is too slow. But LoggerFS is meant to be a drop in solution for legacy code, so the…

You could set up a watchdog.

Yap one could put a watchdog. And then a status reporting and monitoring system to log the log system failures. At some point in that complexity just reverting to rsyslog would seem like a breath of fresh air. Even with its shortcomings.

I have been there, over-designed myself into a hole and then looking back at what was started as a simple 3 step idea now turning into exponentially growing number of branches and corner cases that have to be handled..

Sometimes it is easier to just say "ok this was not a good design" and just throw it away. I have done that and looking back it was a good decision.

Re: Introducing LoggerFS (2013)

#39

Post by mmastrac »

derefr wrote: post #37
mmastrac wrote: post #19

Earlier quoted context omitted.

My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that i…

Presuming your daemons rotate their logs, what's the benefit of this over mounting a plain-old tmpfs over your logs folder? Either way, the shipper reads virtual files that are actually stored in memory, and which eventually get purged. It's just the thresholds that are different.

You'd have to be rolling your logs fairly often to avoid filling up tmpfs with log lines that have already been shipped. The advantage of this approach is that log lines that are shipped are no longer on the box at all.

Re: Introducing LoggerFS (2013)

#40

Post by colin_mccabe »

As others have commented, in general, it seems simplest and best to use the remote features of rsyslog or journald rather than going through a filesystem layer.

But this looks like a fun project. Is there an advantage to this over using NFS for logs?

Post reply on HN