What software do you use for analyzing apache logs? I would prefer open source ones that will work on Linux, although this is not a deal breaker.
Ask HN: what software do you use for analyzing apache logs?
1–10 of 22 posts
Re: Ask HN: what software do you use for analyzing apache logs?
#2I use analog.
Re: Ask HN: what software do you use for analyzing apache logs?
#3Use Awk or Perl. They are available on every Linux system already so it's highly portable and very powerful.
Re: Ask HN: what software do you use for analyzing apache logs?
#4I use analog.
I use a combination of analog and some custom scripts. analog is written by a profession statistician and is a steadfast tool. The custom scripts look for interesting searches. Classics include "What does the fur on a rat do?" and "How do I connect an airbrush to a scubadiving tank?"
Re: Ask HN: what software do you use for analyzing apache logs?
#5Stone Steps Webalizer: http://www.stonesteps.ca/projects/webalizer/
along with cronolog and a bunch of custom Python scripts to autogenerate webalizer configs.
Re: Ask HN: what software do you use for analyzing apache logs?
#6AWStats with GeoIPFree is pretty good, but a major hassle to configure.
Re: Ask HN: what software do you use for analyzing apache logs?
#7A Windows solution, but WebLog Expert works great for us: http://www.weblogexpert.com/
Re: Ask HN: what software do you use for analyzing apache logs?
#8Started with Webalizer, switched to Google Analytics.
Webalizer's ok, but it's missing a bunch of features like sane user agent string parsing (to give an overview of the browsers accessing your site). It also displays most of the stats by page hits (such as country and user agent) rather than "visits" or unique IPs, which I think is a better way to group.
I've been pretty happy with Google Analytics so far - it has a ton of options for sorting and grouping data (like viewing users' paths through the links on the site) and good IP geolocation. Plus, the JavaScript tracker gives you stats on visitors' screen resolution, which can be handy. On the downside, all our data is belong to Google.
Re: Ask HN: what software do you use for analyzing apache logs?
#9apachelog is a nice python module for parsing log lines from apache. it works as a great base for doing your own analysis.
http://code.google.com/p/apachelog/
It's based on this perl module: http://cpan.uwinnipeg.ca/~peterhi/Apache-LogRegex
To manage log files, use cronolog
edit: I use cronolog to break up the logfiles daily and then I run a python script (which uses apachelog to handle the nasty parsing) to create a summary dictionary of parameters from that day. For example:
{num_unique_ips:140, num_pageviews:532, ...}
I pickle that dictionary and save it as a file. So every day has a raw log file and a "summary dictionary" file. To make plots I go through the summary files and unpickle them to extract the quantities of interest.
Re: Ask HN: what software do you use for analyzing apache logs?
#10#!/bin/bash
sudo awk '{print $11}' access_log | grep -v 'yourowndomain.com' | grep -v 'bloglines.com' | grep -v '"-"' | grep -v 'feedburner.com' | sort | uniq -c | sort -rn | head -20