Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-host analytics for better privacy and accuracy

blog.filippo.io

Post reply on HN
Print view View original

41–50 of 93 posts

Re: Self-host analytics for better privacy and accuracy

#41

Post by moehm »

nacs wrote: post #38
toomuchtodo wrote: post #34

Earlier quoted context omitted.

Put Rabbit or Kafka in front of MySQL then?

They have a Redis-based solution to sit in front of mySQL (seems to be a "Pro" feature though): https://piwik.org/faq/how-to/faq_19738/

It's not a Pro feature as in you have to pay for it. It's developed by the devs at piwik.pro, but you can download it for free.

Re: Self-host analytics for better privacy and accuracy

#42

Post by pixelmonkey »

Piwik is a great project, but it tends not to work well for handling sites with millions of events per day. Your MySQL table starts to bust at the seams pretty quickly.

For big sites, you'll want that event data in GiB's of plain raw logs that you can bulk load into tools like BigQuery or Redshift for analysis.

My team has built/delivered a SaaS web content analytics platform for the past few years called Parse.ly. We instrument pageview events (like Google Analytics) automatically, and we also instrument time-on-page using heartbeat events. We collect 50 billion monthly page events for over 600 top-traffic sites, and display it all in a real-time dashboard.

To emphasize that our customers own the data they send us, we recently launched a Raw Data Pipeline product:

http://parse.ly/data-pipeline

Basically, we host a secure S3 bucket and Kinesis stream for customers, and deliver their raw (enriched) event data there. From there, they typically load it into their own BigQuery or Redshift instance, or they analyze pieces of it directly with Python/R/Excel/etc.

Our customers tell us this strikes the right balance among data ownership, query flexibility, and hassle-free infrastructure.

Re: Self-host analytics for better privacy and accuracy

#43

Post by angry-hacker »

dragonne wrote: post #33
tirus wrote: post #16

For those of us that prefer not to be tracked by self-hosted Sandstorm apps, 2 new ublock rules (they're very rough, but I am no ublock wizard): ||sandcats.io/embed.js$script view.gif?page=$image Note that you would have to change the filters if the scripts and tracking pixel are renamed of course but this should catch a majority of the push button installs.

Why do you object to self-hosted analytics? I understand blocking centralized trackers (I do so myself), but self-hosted doesn't seem problematic in the same way GA being present on half the pages on the Internet is. It also strikes me as an unwinnable battle for all but the largest sites.

Because OP is against all kind of tracking? And because he can...

Re: Self-host analytics for better privacy and accuracy

#44

Post by blacksmith_tb »

__jal wrote: post #37

I've done my own analytics since before there were hosted services. Still use an old-school, proprietary tool called Sawmill[1]. One very nice aspect (out of many) about it is that it handles hundreds of other log formats out of the box, so it can report sensibly on switches, email, firewall logs... just about anything with very little effort. No connection, other than being a long-time happy customer. [1] http://saw…

I haven't used Sawmill in ages, but it wasn't bad, once upon a time. There's also GoAccess, which is a snazzy ncurses logfile analyzer:

https://www.goaccess.io/

Re: Self-host analytics for better privacy and accuracy

#46

Post by alexatkeplar »

educar wrote: post #35
shermozle wrote: post #30

Earlier quoted context omitted.

Cos your Redis will never go down? Or need to be upgraded? Patched?

How does snowplow solve this problem? By writing to disk?

Snowplow is implemented as a unidirectional data pipeline:

    tracking -> collection -> enrichment -> storage
Between each step there is typically some kind of persistent queue (mostly S3/Kinesis), and data won't be lost if a downstream component is not operational. Examples:

* If your event collector is unavailable, raw events will be cached in the tracker in localStorage, SQLite or similar

* If your Redshift database is read-only for maintenance, enriched events will be held back until Redshift is writeable again

Re: Self-host analytics for better privacy and accuracy

#47

Post by ocdtrekkie »

Ileca wrote: post #45

"Self-hosting analytics for better privacy and accuracy." Then why using google fonts which is listed by Disconnect.me (used by Firefox) as a tracking domain? Isn't that paradoxical?

I don't think* embedding Google Fonts allows Google much (any?) data collection beyond, presumably, that the font was requested, though it's definitely an unnecessary dependency.

Given Google Fonts allows you to download all of the fonts, with license info provided, in a variety of formats, there's almost no good reason not to embed them directly in your own site.

Right now, Sandstorm apps will currently work with Google Fonts, but the Sandstorm team intends to sandbox the client side better in the near-ish future, so I've already been nudging Sandstorm apps to make sure not to use such things when I see it.

Note that while the author's blog uses Google Fonts, the Piwik Sandstorm package does not. (I checked.)

*I don't know.

Re: Self-host analytics for better privacy and accuracy

#48

Post by 0942v8653 »

ocdtrekkie wrote: post #47
Ileca wrote: post #45

"Self-hosting analytics for better privacy and accuracy." Then why using google fonts which is listed by Disconnect.me (used by Firefox) as a tracking domain? Isn't that paradoxical?

I don't think* embedding Google Fonts allows Google much (any?) data collection beyond, presumably, that the font was requested, though it's definitely an unnecessary dependency. Given Google Fonts allows you to download all of the fonts, with license info provided, in a variety of formats, there's almost no good reason not to embed them directly in your own site. Right now, Sandstorm apps will currently work with Go…

Google Fonts does see the page you are on, in the `Referer` header. According to mitmproxy:

    host:              fonts.googleapis.com
    Connection:        keep-alive
    Proxy-Connection:  keep-alive
    Accept:            text/css,*/*;q=0.1
    User-Agent:        Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13E238 Safari/601.1
    Accept-Language:   en-us
    Referer:           https://blog.filippo.io/self-host-analytics/
    Accept-Encoding:   gzip, deflate
(I agree with the rest of your comment; the best thing to do is host the fonts yourself.)

Re: Self-host analytics for better privacy and accuracy

#49

Post by curiousgal »

Ileca wrote: post #45

"Self-hosting analytics for better privacy and accuracy." Then why using google fonts which is listed by Disconnect.me (used by Firefox) as a tracking domain? Isn't that paradoxical?

He probably didn't know that given it's not the most intuitive thing to notice.

Re: Self-host analytics for better privacy and accuracy

#50

Post by FiloSottile »

Ileca wrote: post #45

"Self-hosting analytics for better privacy and accuracy." Then why using google fonts which is listed by Disconnect.me (used by Firefox) as a tracking domain? Isn't that paradoxical?

Oh, hey, good point. Let me fix that.

EDIT: Done. Not exactly straightforward to download Google Fonts but there are great helpers around. Got rid of CDNjs as well, since CloudFlare has HTTP/2 now. No 3rd parties left except GA, which will go in a day or two.

Post reply on HN