Real-Time Log Collection with Fluentd and MongoDB

#1

Post by kzk_mover » Mon, Dec 05, 2011, 4:26 AM UTC

Real-Time Log Collection with Fluentd and MongoDB
blog.treasure-data.com

Re: Real-Time Log Collection with Fluentd and MongoDB

#2

Post by nodesocket » Mon, Dec 05, 2011, 5:02 AM UTC

Do you have a parser for nginx and/or lighttpd? Would like push logs from these to MongoDB.

Re: Real-Time Log Collection with Fluentd and MongoDB

#3

Post by hoop » Mon, Dec 05, 2011, 5:13 AM UTC

nodesocket wrote: post #2

Do you have a parser for nginx and/or lighttpd? Would like push logs from these to MongoDB.

For lighttpd, try the following in your config to log in a format identifical to that of Apache's combined log format.

    accesslog.format = "%h %V %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

See the docs on ModAccessLog for more information: http://redmine.lighttpd.net/wiki/1/Docs:ModAccesslog

Re: Real-Time Log Collection with Fluentd and MongoDB

#4

Post by nl » Mon, Dec 05, 2011, 5:32 AM UTC

Very nice & all, but... MongoDB?

MongoDB is very popular, but all the (limited) criticisms of it seem to related to insert performance once the dataset it too big to fit in RAM.

Normally the easy-of-development arguments make up for that, but log files is one of those areas that has a tendency to expand quickly beyond any expectations.

There is a reason why most companies are using HDFS and/or Cassandra for structured log file storage.

Re: Real-Time Log Collection with Fluentd and MongoDB

#5

Post by kzk_mover » Mon, Dec 05, 2011, 5:38 AM UTC

nl wrote: post #4

Very nice & all, but... MongoDB? MongoDB is very popular, but all the (limited) criticisms of it seem to related to insert performance once the dataset it too big to fit in RAM. Normally the easy-of-development arguments make up for that, but log files is one of those areas that has a tendency to expand quickly beyond any expectations. There is a reason why most companies are using HDFS and/or Cassandra for structure…

fluentd's greatest advantage is, it's written in Ruby. So, it's really easy to write the plugin for any datastores.

This is HDFS (Hoop, the HDFS REST gateway) plugin.

* https://github.com/tagomoris/fluent-plugin-hoop

And also, Cassandra plugin are now in under development.

* https://github.com/tomitakazutaka/fluent-plugin-cassandra

You can see the user contributed plugin list here.

* https://github.com/tomitakazutaka/fluent-plugin-cassandra

Re: Real-Time Log Collection with Fluentd and MongoDB

#6

Post by kzk_mover » Mon, Dec 05, 2011, 5:41 AM UTC

nodesocket wrote: post #2

Do you have a parser for nginx and/or lighttpd? Would like push logs from these to MongoDB.

as hoop suggested, it can parse Apache's combined format :-) And the contribution is welcome!

Re: Real-Time Log Collection with Fluentd and MongoDB

#7

Post by doryokujin » Mon, Dec 05, 2011, 5:44 AM UTC

Great! If we use Fluentd and MongoDB, we can collect realtime event without writing some codes, but only configuration setting. I also think about more flexible aggregation system using them: "An Introduction to Fluent & MongoDB Plugins" http://www.slideshare.net/doryokujin/an-introduction-to-flue... . Please tell me if there exists more powerful use-case using Fluentd & Mongo!

Re: Real-Time Log Collection with Fluentd and MongoDB

#8

Post by kordless » Mon, Dec 05, 2011, 6:11 AM UTC

I'd also suggest looking at both Logstash and Greylog2. They both can use MongoDB as the storage engine for logs, and can also do the field extractions.