Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing LoggerFS (2013)

developer.rackspace.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Introducing LoggerFS (2013)

#21

Post by tlrobinson »

anoother wrote: post #13

This sounds really interesting, but I can't seem to ascertain what it actually does. > The log data is buffered in-memory (potentially journaled for reliability) and sent over a configurable transport. What are the options for this 'configurable transport'? What is/are the endpoint(s)? Does LoggerFS have facilities for storing and reading-back logs, or does it rely on other services for this? The post only seems to /…

From the article:

    Backend/Aggregator agnostic (includes multiple log transports)
      Supports any Syslog-based log manager
        Loggly, Splunk, Logstash, Rsyslog/Syslog-ng
      ZeroMQ
      NSQ transport – used internally at AutoRef.com
      Generic UDP/TCP
      And soon: AMQP and Redis (and later: Scribe? Fluentd?)
It deals with the specific problem of collecting the logs from the applications on your servers and shipping them to an aggregator (of which there are already many, e.x. Loggly, Splunk, Logstash)

Re: Introducing LoggerFS (2013)

#22

Post by ithkuil »

e12e wrote: post #12
staunch wrote: post #5

My first concern is what happens to daemons when they can't write to the FUSE mount because this program is down or misbehaving. I believe many daemons would misbehave, block, or die. That scares me but maybe I'm being paranoid. I think all the arguments against log shippers are pretty weak, workarounds are simple, especially if the alternative introduces any instability.

Can hardly be worse than in the event of a full disk/partition?

The failure more is very different. On log partition disk full log writes will fail, which you favourite logging library might decide to ignore. A misbehaving FUSE daemon can hang applications that write files and assume it's a local disk.

A smart logger can have a writer thread and a buffer and decide to drop log lines if the logger is too slow. But LoggerFS is meant to be a drop in solution for legacy code, so the concern is perfectly valid.

Re: Introducing LoggerFS (2013)

#23

Post by ams6110 »

mmastrac wrote: post #19
mmahemoff wrote: post #17

A little light on detail. Can someone please explain exactly how this works? The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue. Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to un…

My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that i…

How many logging frameworks don't already support syslog?

Re: Introducing LoggerFS (2013)

#24

Post by achillean »

mmastrac wrote: post #19
mmahemoff wrote: post #17

A little light on detail. Can someone please explain exactly how this works? The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue. Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to un…

My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that i…

I actually wrote exactly the same thing years ago (2007), even had the same name :)

http://sourceforge.net/projects/loggerfs/

Re: Introducing LoggerFS (2013)

#25

Post by e12e »

staunch wrote: post #18
e12e wrote: post #12

Earlier quoted context omitted.

Can hardly be worse than in the event of a full disk/partition?

Depending on your disk not to magically fill up is very different from relying on this FUSE daemon to be up and operational.

To be clear, I do think it's a valid concern -- my point was that "what if log location isn't writable/disk is full" is pretty basic concern that any daemon should account for (what happens if somehow mount -oro /var/log?). And writes failing shouldn't be that much of an alien error condition.

In sum, if fuse/loggerfs together can guarantee that every possible state will result in sane (error) state on writes -- this shouldn't be worse than a disk dying/fs corruption/disk-full type failure.

It will obviously be another point of failure.

On a side note, mounting this under eg /var/log, and then having a strong guarantee that failure will result in an unmount, revealing a writable /var/log seems like the best of both worlds. Would probably have to HUP all writers though...

Re: Introducing LoggerFS (2013)

#26

Post by e12e »

ams6110 wrote: post #23
mmastrac wrote: post #19

Earlier quoted context omitted.

My first impression is that this is a fantastic and novel development for centralized log analysis. Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc). The application doesn't know that i…

How many logging frameworks don't already support syslog?

Probably none, given: "echo hi|logger".

Re: Introducing LoggerFS (2013)

#27

Post by disbelief »

divoxx wrote: post #7

I can't find a link to the actual project on the article. Is it just me? lol

A quick Google turned up a repository on Sourceforge: http://sourceforge.net/projects/loggerfs/

Started in 2007, last update in 2013. Not sure if it's the same thing, but the description is similar:

> LoggerFS is a fuse-based virtual file system that allows you to store log files from apache, syslog and more directly in a database instead of a regular file.

Re: Introducing LoggerFS (2013)

#28

Post by jhardcastle »

disbelief wrote: post #27
divoxx wrote: post #7

I can't find a link to the actual project on the article. Is it just me? lol

A quick Google turned up a repository on Sourceforge: http://sourceforge.net/projects/loggerfs/ Started in 2007, last update in 2013. Not sure if it's the same thing, but the description is similar: > LoggerFS is a fuse-based virtual file system that allows you to store log files from apache, syslog and more directly in a database instead of a regular file.

Another commenter upthread[0] said that was his project, same name and same idea.

> I actually wrote exactly the same thing years ago (2007), even had the same name :)

[0] https://news.ycombinator.com/item?id=7855060

Re: Introducing LoggerFS (2013)

#29

Post by cyounkins »

infosectosser wrote: post #14

This article was written in September 2013, just a few days before AutoRef - the company the author was working for at the time - announced they were shutting down [1]. Does anybody know if development continued on the project? [1] http://www.bizjournals.com/pittsburgh/blog/techflash/2013/09...

Hi, previous CEO of AutoRef here.

I've spoken with Eric (the engineer who spearheaded this), and while code exists it needs some work for release. It hasn't been worked on since the closure of AutoRef.

Eric and I will be working on getting a release ready within a week. If you'd like to be notified when a release is ready, please sign up for this mailchimp list and we'll email you: http://eepurl.com/WlqXz

Re: Introducing LoggerFS (2013)

#30

Post by seanewest »

I'm just waiting for us all to make a FS for practically everything through FUSE (a virtual file system which is used by LoggerFS).

I learned about FUSE while getting into Plan 9 and the "everything as a file" mentality. Lets make everything a file!

Post reply on HN