Self-host analytics for better privacy and accuracy • Hacker News Board

Re: Self-host analytics for better privacy and accuracy

#11

Post by FiloSottile » Mon, May 09, 2016, 9:10 PM UTC

Animats wrote: post #8

If you're serving the pages yourself (not, as this site does, through Cloudflare's caches), what does this tell you that isn't in the server logs?

Unless you are tracking events, outlinks, time spent on page or something similar, nothing as far as data goes. But obviously server logs lack statistics and aggregation. PIWIK actually supports using the server logs as its data source instead of the javascript tag.

Re: Self-host analytics for better privacy and accuracy

#12

Post by ocdtrekkie » Mon, May 09, 2016, 9:12 PM UTC

aluhut wrote: post #5

I would love to have Sandstorm on my RasPi but they don't support ARM. It sounds like a great home intranet.

Just notes on this for the curious:

1. Sandstorm doesn't support ARM currently because Sandstorm apps run native Linux binaries, and every app would have to be compiled for each architecture.

2. I honestly think you'd be running pretty crippled trying to do Sandstorm on a RasPi. It's a bit smaller scale than Sandstorm seems targeted for. Each open Sandstorm grain commonly uses 100 MB of RAM or more (on top of the RAM used by Linux and the Sandstorm server itself, of course), so with just a couple of Sandstorm grains running simultaneously, you can max out a RasPi pretty quickly.

Re: Self-host analytics for better privacy and accuracy

#13

Post by mwexler » Mon, May 09, 2016, 9:13 PM UTC

There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like Snowplow (http://snowplowanalytics.com/), but it's currently hard-coded to AWS.

Hosting your own analytics data can be great, but there are lots of ways to get better accuracy and control over your data without having to host everything. Still, if you can, it's great.

Re: Self-host analytics for better privacy and accuracy

#14

Post by shermozle » Mon, May 09, 2016, 9:16 PM UTC

Piwik has the problem that it writes directly to MySQL as the activity happens. If your database is down, you lose data. If you have a spike of traffic above what your DB can handle in writes, you lose data.

Snowplow doesn't have this problem.

Re: Self-host analytics for better privacy and accuracy

#15

Post by shermozle » Mon, May 09, 2016, 9:17 PM UTC

Animats wrote: post #8

If you're serving the pages yourself (not, as this site does, through Cloudflare's caches), what does this tell you that isn't in the server logs?

Deduplicated reach (Unique Visitors). IP addresses change and are shared, so they aren't suitable for this.

Re: Self-host analytics for better privacy and accuracy

#16

Post by tirus » Mon, May 09, 2016, 9:20 PM UTC

For those of us that prefer not to be tracked by self-hosted Sandstorm apps, 2 new ublock rules (they're very rough, but I am no ublock wizard):

||sandcats.io/embed.js$script

view.gif?page=$image

Note that you would have to change the filters if the scripts and tracking pixel are renamed of course but this should catch a majority of the push button installs.

Re: Self-host analytics for better privacy and accuracy

#17

Post by moehm » Mon, May 09, 2016, 9:21 PM UTC

shermozle wrote: post #14

Piwik has the problem that it writes directly to MySQL as the activity happens. If your database is down, you lose data. If you have a spike of traffic above what your DB can handle in writes, you lose data. Snowplow doesn't have this problem.

You can query your writes in Redis, so it won't be lost if your database goes down.

https://piwik.org/faq/how-to/faq_19738/

Re: Self-host analytics for better privacy and accuracy

#18

Post by rhizome » Mon, May 09, 2016, 9:21 PM UTC

viana007 wrote: post #10

The biggest problem with Piwik is that is not scalable and the cost ($) with servers to store analytics data. I saw many cases that Piwik not supported the big volume of data.

Why isn't it scalable?

Re: Self-host analytics for better privacy and accuracy

#19

Post by _jomo » Mon, May 09, 2016, 9:22 PM UTC

Another very simple alternative is goaccess [0] which is purely server-side. It gathers quite a lot of information by parsing the server log. It doesn't do all the JavaScript stuff to track every single click, but it gives you stats on how many users visit your site, which parts, when, which sites or which domains they're coming from and also how much bandwidth they use. It also shows which status codes are coming from which paths and a lot more. It supports various output formats such as HTML or an interactive htop-like terminal application. It's also being actively developed. I use it and find it very useful.

0: https://www.goaccess.io/

Re: Self-host analytics for better privacy and accuracy

#20

Post by MustardTiger » Mon, May 09, 2016, 9:25 PM UTC

ocdtrekkie wrote: post #12

aluhut wrote: post #5

I would love to have Sandstorm on my RasPi but they don't support ARM. It sounds like a great home intranet.

Just notes on this for the curious: 1. Sandstorm doesn't support ARM currently because Sandstorm apps run native Linux binaries, and every app would have to be compiled for each architecture. 2. I honestly think you'd be running pretty crippled trying to do Sandstorm on a RasPi. It's a bit smaller scale than Sandstorm seems targeted for. Each open Sandstorm grain commonly uses 100 MB of RAM or more (on top of the RAM…

>Sandstorm doesn't support ARM currently because Sandstorm apps run native Linux binaries, and every app would have to be compiled for each architecture.

That's true of any linux distro providing binary packages. They all support arm anyways, it is trivially simple to compile packages. Even small projects like openbsd compile tens of thousands of packages for a dozen arches.