GoAccess – Visual Web Log Analyzer
goaccess.io
GoAccess – Visual Web Log Analyzer
1–10 of 20 posts
Re: GoAccess – Visual Web Log Analyzer
#2Despite the emphasis on goaccess’s visual mode, keep in mind that “ While the terminal output is the default output, it has the capability to generate a complete, self-contained real-time HTML report (great for analytics, monitoring and data visualization)”. It was a great replacement for my aging webalizer setup which in turn had been replaced by google analytics.
This is for a personal site and at some point I realized I cared less about having those very detailed google metrics (which I never checked anyway) than about making my site more responsive and less invasive. I nixed google analytics and haven’t looked back; still have basic metrics thanks to goaccess.
Re: GoAccess – Visual Web Log Analyzer
#3Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years and had to slice roughly 70% percent of my traffic away because most of it was automated.
Most bots were courteous to state they were bots typically using a useragent with `-bot` found in the string. Some used generic browser useragents but were scanning for things like `wp-admin` etc
Most of the genuine human traffic were people on a mobile phone and that was the only heuristic I looked at to determine how many people visited my site. Very few desktop users were present.
Re: GoAccess – Visual Web Log Analyzer
#4Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years and had to slice roughly 70% percent of my traffic away because most of it was automated. Most bots were courteous to state they were bots typically using a useragent with `-bot` found in the string. Some used generic browser useragents but were scanning for…
The --ignore-crawlers flag in GoAccess filters out a decent amount of bot traffic. It's not perfect, but it's good enough for a rough estimate of 'real' traffic.
Re: GoAccess – Visual Web Log Analyzer
#5Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years and had to slice roughly 70% percent of my traffic away because most of it was automated. Most bots were courteous to state they were bots typically using a useragent with `-bot` found in the string. Some used generic browser useragents but were scanning for…
Indeed - about 39% of hits to my site come from crawlers.
How do I know? Thanks to GoAccess!! ;)
Re: GoAccess – Visual Web Log Analyzer
#6Despite the emphasis on goaccess’s visual mode, keep in mind that “ While the terminal output is the default output, it has the capability to generate a complete, self-contained real-time HTML report (great for analytics, monitoring and data visualization)”. It was a great replacement for my aging webalizer setup which in turn had been replaced by google analytics. This is for a personal site and at some point I real…
What's quite nice with the HTML output view is that it stores stats even if the underlying log files are rotated/deleted - however if the goaccess process ends (like if your server needs to restart) you lose all the historic context.
Re: GoAccess – Visual Web Log Analyzer
#7Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years and had to slice roughly 70% percent of my traffic away because most of it was automated. Most bots were courteous to state they were bots typically using a useragent with `-bot` found in the string. Some used generic browser useragents but were scanning for…
You could add $ssl_cipher to your log_format configuration (if nginx), and use that as a TLS fingerprint to find more bots.
Re: GoAccess – Visual Web Log Analyzer
#8Sadly I imagine most of the traffic to a server would be bots or bad actors scanning for common vulns. I used AWStats (Another log file analyzer) for many years and had to slice roughly 70% percent of my traffic away because most of it was automated. Most bots were courteous to state they were bots typically using a useragent with `-bot` found in the string. Some used generic browser useragents but were scanning for…
Here's a tricky one. Common Crawl runs its bots from AWS. AWS has like a jillion IP addresses. How do you tell which traffic are legit Common Crawl bots and which are imposters?
Re: GoAccess – Visual Web Log Analyzer
#9Despite the emphasis on goaccess’s visual mode, keep in mind that “ While the terminal output is the default output, it has the capability to generate a complete, self-contained real-time HTML report (great for analytics, monitoring and data visualization)”. It was a great replacement for my aging webalizer setup which in turn had been replaced by google analytics. This is for a personal site and at some point I real…
What's quite nice with the HTML output view is that it stores stats even if the underlying log files are rotated/deleted - however if the goaccess process ends (like if your server needs to restart) you lose all the historic context.
I run goaccess once a day to analyze log files. There's an option to store the result in goaccess database. So nothing is never lost and I accumulate stats for as long as I want.
I detailed that in a (lengthy) blog post if you're interested: https://arnaudr.io/2020/08/10/goaccess-14-a-detailed-tutoria...
Re: GoAccess – Visual Web Log Analyzer
#10Great work. I guess I'm spoiled already by tools like DataDog because I was clicking on the dashboards and expected to be able to go to the logs who actually generated them. For example, if I see a huge spike in the Requests dashboard at 3am, I would like to be able to go to the logs that generated that spike by just clicking on the spike. Does GoAccess provide access to the logs themselves?