Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Self-host analytics for better privacy and accuracy

blog.filippo.io

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Self-host analytics for better privacy and accuracy

#21

Post by SwellJoe »

ljoshua wrote: post #7
d0ugie wrote: post #2

To mitigate security loss (Piwik is complex), run Piwik and serve its gif on another machine.

Can you elaborate a bit more?

I'd assume the prior poster is suggesting that running a third party tool on your primary web server(s) would increase the surface area for attacks by some (possibly not small) amount. e.g. if piwik is compromised, an attacker would have some sort of user access on your web server(s), which is generally a bad thing. I suppose some people also use the same database user for all web applications, which would potentially be disastrous.

There are mitigations one can implement without going to that length (run piwik under a different user than any of your other web applications, using suexec, use a different database and user, etc.). At the least, putting Piwik in a container or VM makes sense, if any data on your web server(s) is critical or sensitive.

I suspect for very large deployments this would go without saying. But, for users with only one web server, it might seem reasonable to drop it into the same virtual host and run it all as the same user (and it's probably safe enough to do so for many users, as long as they stay on top of updates). But, any web application you run adds surface area for attackers. Might as well isolate them as well as your skills and resources allow.

Re: Self-host analytics for better privacy and accuracy

#23

Post by yummyfajitas »

mwexler wrote: post #13

There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like Snowplow ( http://snowplowanalytics.com/ ), but it's currently hard-coded to AWS. Hosting your own analytics data can be great, but there are lots of ways to get better accuracy and control over your data without having to host everything. Still, if you can, it'…

Snowplow is only loosely hard coded to AWS. I'm using it and breaking it free is only a few hundred lines of code.

For example, rerouting Snowplow's Kinesis collector into Kafka is 114 lines, and that includes logging, metrics, etc - I basically just had to extend the AbstractSink object in their scala collector. Reading from Kafka is another couple of hundred lines, similarly writing to files.

Re: Self-host analytics for better privacy and accuracy

#25

Post by alexatkeplar »

yummyfajitas wrote: post #23
mwexler wrote: post #13

There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like Snowplow ( http://snowplowanalytics.com/ ), but it's currently hard-coded to AWS. Hosting your own analytics data can be great, but there are lots of ways to get better accuracy and control over your data without having to host everything. Still, if you can, it'…

Snowplow is only loosely hard coded to AWS. I'm using it and breaking it free is only a few hundred lines of code. For example, rerouting Snowplow's Kinesis collector into Kafka is 114 lines, and that includes logging, metrics, etc - I basically just had to extend the AbstractSink object in their scala collector. Reading from Kafka is another couple of hundred lines, similarly writing to files.

Thanks for sharing yummyfajitas - expect official Kafka support for Snowplow a little later this year [1] [2]; it's been long awaited! (Snowplow co-founder)

[1] https://github.com/snowplow/snowplow/milestones/Kafka%20%231 [2] https://github.com/snowplow/snowplow/milestones/Kafka%20%232

Re: Self-host analytics for better privacy and accuracy

#26

Post by dsp1234 »

shermozle wrote: post #14

Piwik has the problem that it writes directly to MySQL as the activity happens. If your database is down, you lose data. If you have a spike of traffic above what your DB can handle in writes, you lose data. Snowplow doesn't have this problem.

They also have a problem that their password hashing method is still md5[0].

[0] - https://developer.piwik.org/api-reference/Piwik/Auth

Re: Self-host analytics for better privacy and accuracy

#27

Post by yummyfajitas »

alexatkeplar wrote: post #25
yummyfajitas wrote: post #23

Earlier quoted context omitted.

Snowplow is only loosely hard coded to AWS. I'm using it and breaking it free is only a few hundred lines of code. For example, rerouting Snowplow's Kinesis collector into Kafka is 114 lines, and that includes logging, metrics, etc - I basically just had to extend the AbstractSink object in their scala collector. Reading from Kafka is another couple of hundred lines, similarly writing to files.

Thanks for sharing yummyfajitas - expect official Kafka support for Snowplow a little later this year [1] [2]; it's been long awaited! (Snowplow co-founder) [1] https://github.com/snowplow/snowplow/milestones/Kafka%20%231 [2] https://github.com/snowplow/snowplow/milestones/Kafka%20%232

Nice. If you build some sort of native maxmind or other geotargeting into the scala collector, that would also be cool.

(Not that it was difficult to roll my own - so far snowplow is perfect for my needs - but obviously I'd rather use an official one.)

Re: Self-host analytics for better privacy and accuracy

#28

Post by alexatkeplar »

yummyfajitas wrote: post #27
alexatkeplar wrote: post #25

Earlier quoted context omitted.

Thanks for sharing yummyfajitas - expect official Kafka support for Snowplow a little later this year [1] [2]; it's been long awaited! (Snowplow co-founder) [1] https://github.com/snowplow/snowplow/milestones/Kafka%20%231 [2] https://github.com/snowplow/snowplow/milestones/Kafka%20%232

Nice. If you build some sort of native maxmind or other geotargeting into the scala collector, that would also be cool. (Not that it was difficult to roll my own - so far snowplow is perfect for my needs - but obviously I'd rather use an official one.)

We do all enrichments like MaxMind, weather, arbitrary JavaScript etc downstream of collection, in our enrichment phase - the list of configurable enrichments is here: https://github.com/snowplow/snowplow/wiki/Configurable-enric...

Re: Self-host analytics for better privacy and accuracy

#29

Post by zoomzoom »

mwexler wrote: post #13

There are many, many self-hosting analytics tools, from your own big data pipes to tools like the aforementioned piwik, as well as Open Web Analytics. I like Snowplow ( http://snowplowanalytics.com/ ), but it's currently hard-coded to AWS. Hosting your own analytics data can be great, but there are lots of ways to get better accuracy and control over your data without having to host everything. Still, if you can, it'…

Also one thing to add here is that the client libraries are a lot of the work and you can use the snowplow js/ios/python etc.. no matter what server-side setup you use. I like to think of snowplow as pushing the open-source analytics standard and then hopefully an ecosystem of server-side products grows around that, led by their own product.

We're doing something as dumb as using logs from cloud storage and parsing those with a ~100 line python script into a DB. S3/GCS deal with the collector uptime and as long as you aren't time-sensitive it is a great solution.

The biggest issue with self-hosted analytics is visualizing/sharing the results with non-tech parts of the team. Piwik has some advantages here because it's closer to Google Analytics or Mixpanel than to a DB of event rows...

Re: Self-host analytics for better privacy and accuracy

#30

Post by shermozle »

moehm wrote: post #17
shermozle wrote: post #14

Piwik has the problem that it writes directly to MySQL as the activity happens. If your database is down, you lose data. If you have a spike of traffic above what your DB can handle in writes, you lose data. Snowplow doesn't have this problem.

You can query your writes in Redis, so it won't be lost if your database goes down. https://piwik.org/faq/how-to/faq_19738/

Cos your Redis will never go down? Or need to be upgraded? Patched?
Post reply on HN