Minimally Invasive (and More Accurate) Analytics: GoAccess and Athena/SQL

lnav[0] can take care of parsing the log files and displaying them in a TUI. There's also a SQLite interface for doing queries. However, you'll need to build the filters/queries yourself, there aren't any built-in ones at the moment.

[0] - https://lnav.org

Looks very interesting.

Yeah I mean, I'm just running a django site, so I imagine I could add a custom middleware that makes an API request on every page load. I guess it would have to try and see if the access token is expired first? and if so grab a new one then make the hit. Is that the recommended setup?

Would I be able to pass extra information to be included in the logs, like e.g. username?

Also, I know you have good privacy policies, but still sending this information through a request makes me nervous nevertheless, even though it's of course miles better than js-based solutions. But what are your thoughts on how possible is it for these requests to be intercepted and this logged data siphoned off by someone else?

> I guess it would have to try and see if the access token is expired first? and if so grab a new one then make the hit. Is that the recommended setup?

Exactly. The token expires after 15 minutes, so you need to check the response and issue a new token should it have expired. You can read our docs on how to do that or take a look at our Go SDK [0] and re-implement it in Python. Unfortunately, I don't have enough time to provide one right now.

> Would I be able to pass extra information to be included in the logs, like e.g. username?

That's not possible right now, but you will be able to send custom events in the future.

> But what are your thoughts on how possible is it for these requests to be intercepted and this logged data siphoned off by someone else?

Highly unlikely. All traffic is SSL encrypted, the internal communication of our server cluster is encrypted, the database, ... I mean, software can always be hacked, but I spend a lot of my time on infrastructure and security.

[0] https://github.com/pirsch-analytics/pirsch-go-sdk/blob/maste...

Logging JSON directly from nginx is what I currently do, and then the log output is ingested straight into ElasticSearch. One neat thing you can do is also log return headers from an upstream HTTP server, such as a username for example or any application-specific piece of data. That way you can interleave your HTTP access logs with application data and have everything available for querying in one index.

I want to second that plug for Athena for ad-hoc analysis. (If you're hosting your own stuff and at the scale where it'd be useful, there's Presto/Hive, which Athena is based on, and/or at Trino, the Presto fork maintained by some of its initial developers.)

It was useful for me when tweaking spam/bot detection rules a while ago; if I could roughly describe a rule in a query, I could back-test it on old traffic and follow up on questionable-looking results (e.g. what other requests did this IP make around the time of the suspicious ones?). We also used Athena on a project looking into performance, and on network flow logs. The lack of recurring charges for an always-on cluster makes it great for occasional use like that.

You can use what the docs call "partition projection" to efficiently limit the date range of logs to look at (https://docs.aws.amazon.com/athena/latest/ug/partition-proje...), so it was free-ish to experiment with a query on the last couple days of data before looking further back.

More generally, Athena/Presto/Hive support various data sources and formats (including applying regexps to text). Compressed plain-text formats like ALB logs can already be surprisingly cheap to store/scan. If you're producing/exporting data, it's worth looking into how these tools "like" to receive it--you may be able to use a more compact columnar format (Parquet or ORC) or take advantage of partitioning/bucketing (https://docs.aws.amazon.com/athena/latest/ug/partitions.html, https://trino.io/blog/2019/05/29/improved-hive-bucketing.htm...) for more efficient querying later.

As the blog post notes, usability was...imperfect, especially during initial setup. Error messages sometimes point at one of the first few tokens of the SQL, nowhere near the mistake, and there are lots of knobs to tweak, some controlled by 'magical.dotted.names.in.strings'. CLIs were sometimes easier than the GUI. But you can get a lot out of it once you've got it working!