Introducing LoggerFS (2013) • Hacker News Board

Re: Introducing LoggerFS (2013)

#11

Post by rch » Thu, Jun 05, 2014, 10:09 PM UTC

staunch wrote: post #5

My first concern is what happens to daemons when they can't write to the FUSE mount because this program is down or misbehaving. I believe many daemons would misbehave, block, or die. That scares me but maybe I'm being paranoid. I think all the arguments against log shippers are pretty weak, workarounds are simple, especially if the alternative introduces any instability.

It needs testing, like anything new, but I think the concept is sound enough to warrant at least a modicum of attention. It's something I've been wanting for a while actually.

Re: Introducing LoggerFS (2013)

#12

Post by e12e » Thu, Jun 05, 2014, 10:09 PM UTC

staunch wrote: post #5

My first concern is what happens to daemons when they can't write to the FUSE mount because this program is down or misbehaving. I believe many daemons would misbehave, block, or die. That scares me but maybe I'm being paranoid. I think all the arguments against log shippers are pretty weak, workarounds are simple, especially if the alternative introduces any instability.

Can hardly be worse than in the event of a full disk/partition?

Re: Introducing LoggerFS (2013)

#13

Post by anoother » Thu, Jun 05, 2014, 10:11 PM UTC

This sounds really interesting, but I can't seem to ascertain what it actually does.

> The log data is buffered in-memory (potentially journaled for reliability) and sent over a configurable transport.

What are the options for this 'configurable transport'? What is/are the endpoint(s)? Does LoggerFS have facilities for storing and reading-back logs, or does it rely on other services for this?

The post only seems to /hint/ at answers to these questions.

Re: Introducing LoggerFS (2013)

#14

Post by infosectosser » Thu, Jun 05, 2014, 10:12 PM UTC

This article was written in September 2013, just a few days before AutoRef - the company the author was working for at the time - announced they were shutting down [1]. Does anybody know if development continued on the project?

[1] http://www.bizjournals.com/pittsburgh/blog/techflash/2013/09...

Re: Introducing LoggerFS (2013)

#15

Post by Xorlev » Thu, Jun 05, 2014, 10:13 PM UTC

godisdad wrote: post #2

Buh? http://en.wikipedia.org/wiki/Log-structured_file_system

Log-structured != log-centric.

Log-centric (LoggerFS) is a filesystem around managing log files.

Log-structured is a way of structuring data such that it's written to sequentially and is always appending (while dropping from the head).

Log files are most similar to log-structured, but not a filesystem dedicated to shipping log files for centralization.

Re: Introducing LoggerFS (2013)

#16

Post by imperialWicket » Thu, Jun 05, 2014, 10:14 PM UTC

This is a pretty old article, and not much seems to have happened in LoggerFS development. Maybe log shippers aren't so bad after all?

Re: Introducing LoggerFS (2013)

#17

Post by mmahemoff » Thu, Jun 05, 2014, 10:14 PM UTC

A little light on detail. Can someone please explain exactly how this works?

The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue.

Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to understand it.)

Re: Introducing LoggerFS (2013)

#18

Post by staunch » Thu, Jun 05, 2014, 10:15 PM UTC

e12e wrote: post #12

staunch wrote: post #5

My first concern is what happens to daemons when they can't write to the FUSE mount because this program is down or misbehaving. I believe many daemons would misbehave, block, or die. That scares me but maybe I'm being paranoid. I think all the arguments against log shippers are pretty weak, workarounds are simple, especially if the alternative introduces any instability.

Can hardly be worse than in the event of a full disk/partition?

Depending on your disk not to magically fill up is very different from relying on this FUSE daemon to be up and operational.

Re: Introducing LoggerFS (2013)

#19

Post by mmastrac » Thu, Jun 05, 2014, 10:17 PM UTC

mmahemoff wrote: post #17

A little light on detail. Can someone please explain exactly how this works? The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue. Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to un…

My first impression is that this is a fantastic and novel development for centralized log analysis.

Basically, it's a virtual filesystem that pretends to create files, but actually intercepts writes to log-style files in this virtual location and buffers those in memory, eventually shipping them off to a place where you can run central analytics on them (ie: Splunk, Logstash, etc).

The application doesn't know that it's writing to a virtual file: it just opens a descriptor, dumps a line or two every few seconds and continues along its merry way. The logs never touch the disk, which means that they don't content for limited disk I/O bandwidth.

Re: Introducing LoggerFS (2013)

#20

Post by [deleted] » Thu, Jun 05, 2014, 10:18 PM UTC

mmahemoff wrote: post #17

A little light on detail. Can someone please explain exactly how this works? The best I can figure is it's a shipper replacement. So apps write to a log file as usual, but it's actually buffered in memory and pushed to a central server via a FIFO queue. Given it's all in-memory, it will be small and transient, so you're completely relying on the central server to store it reliably. (Not a criticism, just trying to un…

[deleted]