Roll Your Own Analytics • Hacker News Board

Re: Roll Your Own Analytics

#51

Post by derekdahmer » Thu, Mar 14, 2019, 5:26 PM UTC

lukethomas wrote: post #34

At the past 3 places I've worked, we've setup Snowplow Analytics ( https://snowplowanalytics.com/ ) and would strongly recommend it over GA, Segment, and other third-party systems. If you're looking for a first-party system, Snowplow is an amazing setup.

I looked into Snowplow but was intimidated by their Trackers/Collectors/Enrich/Storage/Data Modeling/Analytics flow.

How hard is it to setup? Does it come with a UI to easily view, sort and filter these events into graphs like Mixpanel or Amplitude?

Re: Roll Your Own Analytics

#52

Post by gorkemcetin » Thu, Mar 14, 2019, 5:33 PM UTC

If you want to decommission Google all the way, you can have a look at Countly (https://github.com/countly/countly-server). It is also available on Digital Marketplace so it is quite straightforward to deploy on a cheap instance. It has an API so you can query your data, too.

Re: Roll Your Own Analytics

#53

Post by aaaaaaaaaaab » Thu, Mar 14, 2019, 5:39 PM UTC

soared wrote: post #49

aaaaaaaaaaab wrote: post #12

Earlier quoted context omitted.

>You should track its usage with your own analytics. I appreciate the joke, but don't give them ideas. Since one can't normally track interactions with the scrollbar, I wouldn't be surprised if they were to implement their own "scrollbar" from DIVs to be able to track if the user had dragged it. Just put back the built-in scrollbar. Period.

You don't track the scrollbar itself, just scroll depth by quartile https://www.simoahava.com/analytics/scroll-depth-trigger-goo...

This can’t differentiate between scrolling via the scrollbar and other means (keyboard, trackpad, mouse wheel).

Re: Roll Your Own Analytics

#54

Post by fuball63 » Thu, Mar 14, 2019, 5:42 PM UTC

Can anyone post a good guide as to what analytics I should be collecting? I want to implement a system of my own like this, but would like a quick primer on the subject.

Re: Roll Your Own Analytics

#55

Post by mejakethomas » Thu, Mar 14, 2019, 5:42 PM UTC

(Data engineer here)

Nice article! I did something very similar to this for my blog but used Snowplow's javascript tracker (https://github.com/snowplow/snowplow-javascript-tracker), a cloudfront distribution with s3 log forwarding, a couple lambda functions (with s3 "put" triggers), S3 as the post-processed storage layer, and AWS athena as the query layer. The system costs under $1 per month, is very scalable, and is producing amazingly good/structured data with mid-level latency. I've written about it here:

https://bostata.com/post/client-side-instrumentation-for-und...

By using the snowplow javascript tracker, you get a ton of functionality out of the box when it comes to respecting "do not track", structured event formatting, additional browser contexts, etc. If you want to see how the blog site is functionally instrumented, filter network requests by "stm" (sent time) and you'll see what's being collected.

I've found (after setting similar systems for 15+ companies of varying scale) that where a system like this breaks down is when you want to warehouse event data and tie it to other critical business metrics (stripe, salesforce, database tables that underpin the application, etc). Another point it starts to break down is when you need low-latency data access. At that point it makes more and more sense to run data into a stream (kinesis/kafka/etc) and have "low latency" (couple hundred ms or less) and "high latency" (minutes/hours/etc) points of centralization.

Using multi-az/replicated stream-based infrastructure (like snowplow's scala stuff) has been completely transformational to numerous companies I've set it up at. A single source of truth when it comes to both low-latency and med/high-latency client side event data is absolutely massive. Secondly, being able to tie many sources of data together (via warehousing into redshift or snowflake) is eye-opening every single time. I've recently been running ~300k+ requests/minute through snowplow's stream-based infrastructure and it's rock-solid.

Again, nice post! It's awesome to see people doing similar things. :)

Re: Roll Your Own Analytics

#56

Post by mejakethomas » Thu, Mar 14, 2019, 5:47 PM UTC

fuball63 wrote: post #54

Can anyone post a good guide as to what analytics I should be collecting? I want to implement a system of my own like this, but would like a quick primer on the subject.

I've found the documentation here to be very comprehensive if you want to start learning why/what/how:

https://github.com/snowplow/snowplow/wiki/javascript-tracker https://github.com/snowplow/snowplow/wiki/canonical-event-mo... https://developer.matomo.org/api-reference/tracking-api

Re: Roll Your Own Analytics

#57

Post by mejakethomas » Thu, Mar 14, 2019, 5:53 PM UTC

derekdahmer wrote: post #51

lukethomas wrote: post #34

At the past 3 places I've worked, we've setup Snowplow Analytics ( https://snowplowanalytics.com/ ) and would strongly recommend it over GA, Segment, and other third-party systems. If you're looking for a first-party system, Snowplow is an amazing setup.

I looked into Snowplow but was intimidated by their Trackers/Collectors/Enrich/Storage/Data Modeling/Analytics flow. How hard is it to setup? Does it come with a UI to easily view, sort and filter these events into graphs like Mixpanel or Amplitude?

Snowplow is awesome - it doesn't come with a UI but here's a sample of what data is included:

https://github.com/snowplow/snowplow/wiki/canonical-event-mo...

A pretty common move is to drop this data into redshift/snowflake and query it with Mode/Looker/Tableau/whatever. Athena is a viable option as well, until you get into higher data volumes and don't want to pay for each scan.

Context: I'm a tech lead (data engineering) @ a public company, have set this system up 15+ times @ numerous other companies, and could not live without it at this point. Current co's snowplow systems process 250M+ events per day peaking @ 300k+ reqs/min, on very cost-efficient infra.

Re: Roll Your Own Analytics

#58

Post by mejakethomas » Thu, Mar 14, 2019, 5:54 PM UTC

mxuribe wrote: post #50

This is quite timely! I'm in the middle of transitioning away from google analytics for my apps/sites (to de-google-ify as much as possible). However, my plan was to implement self-hosted matomo/piwik. In fact, i was considering only implementing the server log-reading feature (and not the javascript tracker script) primarily to lessen webpage bloat but also because my analytics needs are quite basic and minimal. (I'…

Agreed!

Re: Roll Your Own Analytics

#59

Post by lostapathy » Thu, Mar 14, 2019, 5:57 PM UTC

fbelzile wrote: post #30

folkrav wrote: post #26

Earlier quoted context omitted.

Also mentioned in the article: > Avoid ad-blockers - My goal with analytics is to learn how people use my site so I can improve it and serve them better. I'm not using ad-tech so there's there's no point in getting blocked by 25% of visitors with an ad-blocker. That means doing 1st-party analytics, without using a 3rd-party tracking snippet—even self-hosted!* > *Some ad-blockers already block self-hosted Matamo/Piwik…

I'm okay with this. If a user agent is attempting to block my tracking code (piwik.js), it's likely that the user doesn't want to be tracked. Part of respecting user privacy is accepting the fact my tracking scripts will be blocked by most privacy extensions. I feel Matomo does privacy correctly. By default it continues to use the well known piwik.js filename that extensions block and also respects the DNT (do not tr…

> I'm okay with this. If a user agent is attempting to block my tracking code (piwik.js), it's likely that the user doesn't want to be tracked.

What users DO want to be tracked????

Re: Roll Your Own Analytics

#60

Post by taurath » Thu, Mar 14, 2019, 5:58 PM UTC

It seems like losing sessions because people closed their browser or unchecked exit events is potentially the weekest point of the setup. Still, very nice and user friendly!