Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GoAccess – Visual Web Log Analyzer

goaccess.io

Post reply on HN
Print view View original

11–20 of 20 posts

Re: GoAccess – Visual Web Log Analyzer

#11

Post by majodev »

elboulangero wrote: post #9
noxvilleza wrote: post #6

Earlier quoted context omitted.

What's quite nice with the HTML output view is that it stores stats even if the underlying log files are rotated/deleted - however if the goaccess process ends (like if your server needs to restart) you lose all the historic context.

I run goaccess once a day to analyze log files. There's an option to store the result in goaccess database. So nothing is never lost and I accumulate stats for as long as I want. I detailed that in a (lengthy) blog post if you're interested: https://arnaudr.io/2020/08/10/goaccess-14-a-detailed-tutoria...

I can confirm that goaccess databases works great for incrementally regenerating these HTML reports. We currently pipe all logs from all our ingress-nginx LBs every 15 minutes (and grep by virtual host) to goaccess and thus update its reports periodically. Each month, all reports get archived so we can start with a fresh report again.

Semi live and super useful without having to use 3rd party services like Google Analytics. The HTML report is self-contained (single file) and thus can easily be shared (or just statically hosted).

Re: GoAccess – Visual Web Log Analyzer

#12

Post by ComputerGuru »

jgalt212 wrote: post #8
legrande wrote: post #3

Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years and had to slice roughly 70% percent of my traffic away because most of it was automated. Most bots were courteous to state they were bots typically using a useragent with `-bot` found in the string. Some used generic browser useragents but were scanning for…

Here's a tricky one. Common Crawl runs its bots from AWS. AWS has like a jillion IP addresses. How do you tell which traffic are legit Common Crawl bots and which are imposters?

I mean, they’re all bots one way or the other. The only exception would be a personal VPN running off of AWS, but that’s a bad idea given how many sites block that range.

Re: GoAccess – Visual Web Log Analyzer

#13

Post by tombrossman »

loloquwowndueo wrote: post #2

Despite the emphasis on goaccess’s visual mode, keep in mind that “ While the terminal output is the default output, it has the capability to generate a complete, self-contained real-time HTML report (great for analytics, monitoring and data visualization)”. It was a great replacement for my aging webalizer setup which in turn had been replaced by google analytics. This is for a personal site and at some point I real…

The HTML reports are my preferred way of looking at stats, but to make them more useful it's worth taking some additional steps to filter all the garbage traffic.

What works for me is:

- Use ipset to drop all traffic from certain countries (you pick which works best for you)

- Configure fail2ban to 'automagically' drop all IPs requesting .php and wp-admin URLs for a few days

- Integrate Piwik/Matomo's 'referrer spam' blocklist into your list of ignored referrers.

- Use per-site logging and only log .html hits with a static site to see page views.

This approach won't work for everyone and it takes extra sysadmin & Bash scripting skills to achieve, but it works really well with my Jekyll site.

I don't receive much traffic on my personal website but my stats page is public and updates hourly with a cronjob. https://www.tombrossman.com/stats/

Re: GoAccess – Visual Web Log Analyzer

#14

Post by nkrisc »

elboulangero wrote: post #9
noxvilleza wrote: post #6

Earlier quoted context omitted.

What's quite nice with the HTML output view is that it stores stats even if the underlying log files are rotated/deleted - however if the goaccess process ends (like if your server needs to restart) you lose all the historic context.

I run goaccess once a day to analyze log files. There's an option to store the result in goaccess database. So nothing is never lost and I accumulate stats for as long as I want. I detailed that in a (lengthy) blog post if you're interested: https://arnaudr.io/2020/08/10/goaccess-14-a-detailed-tutoria...

I love when something I had once been looking for, but seemed so specific I didn't know how to search for it, drops right into my lap. Thank you for this, it is exactly what I've been looking for.

Re: GoAccess – Visual Web Log Analyzer

#18

Post by smnscu »

tombrossman wrote: post #13
loloquwowndueo wrote: post #2

Despite the emphasis on goaccess’s visual mode, keep in mind that “ While the terminal output is the default output, it has the capability to generate a complete, self-contained real-time HTML report (great for analytics, monitoring and data visualization)”. It was a great replacement for my aging webalizer setup which in turn had been replaced by google analytics. This is for a personal site and at some point I real…

The HTML reports are my preferred way of looking at stats, but to make them more useful it's worth taking some additional steps to filter all the garbage traffic. What works for me is: - Use ipset to drop all traffic from certain countries (you pick which works best for you) - Configure fail2ban to 'automagically' drop all IPs requesting .php and wp-admin URLs for a few days - Integrate Piwik/Matomo's 'referrer spam'…

Nice photos, thank you for piquing my interest in Jersey https://www.tom.je/

Also, pretty good advice in this post, bookmarked it.

Re: GoAccess – Visual Web Log Analyzer

#20

Post by fasteo »

Worth mentioning that it comes with an embedded WebSocket server, available as a standalone classical unix server - write programs that do one thing and do it well -

"Very simple, just redirect the output from your application (stdout) to a file (named pipe) and let gwsocket transfer the data to the browser — That's it."

[1] https://gwsocket.io/

Post reply on HN