Ask HN: Good tech talks on how analytics systems are implemented?

Post by psankar » Wed, Jun 12, 2019, 10:01 AM UTC

usgroup wrote: post #2

IMO, your requirements are too basic to need a serious system. Either log interaction to a file or a database, parse the output and query it with SQL to produce your basic metrics, or just write to Google Analytics. When this starts creaking at the seams it'll mean that you either have bigger analysis and/or scalability requirements and it'll much clearer what you need to look for.

Post by psankar » Wed, Jun 12, 2019, 10:02 AM UTC

acidbaseextract wrote: post #4

Go with off the shelf. You'll get something far better that you can build yourself, and if you need something custom, you'll have a much clearer idea what your analysis is missing. Writing to Google Analytics, Amplitude, Mixpanel (all of which have free tiers) or equivalent all should handle your case well.

Post by karanke » Wed, Jun 12, 2019, 10:13 AM UTC

psankar wrote: post #9

karanke wrote: post #6

Note: I build and maintain such systems for a living. There's a lot of context that's missing from your post, some questions that can help us guide you in the right direction: 1) Can your website call out to external services, or are you limited to operating behind a company network? 2) Is this more of an ad-hoc analysis or do you want to invest in a framework to be able to track such metrics systematically over time…

1) The website cannot call to external services, which is the primary reason why we thought about implementing it from scratch. 2) We want to invest in building a good framework to track such metrics systematically over time 3) We have some non-web API clients too. Adblock is not a problem. 4) Accuracy is better. Speed is not that critical and could even be a few minutes delayed. 5) The data will be kept for a few mo…

Post by sfifs » Wed, Jun 12, 2019, 10:32 AM UTC

Post by lelima » Wed, Jun 12, 2019, 10:42 AM UTC

psankar wrote: post #10

serial_dev wrote: post #7

I was working for a startup implementing analytics tools. In my opinion, our setup was over-engineered, but I wasn't there at the beginning, so I might be wrong. Also, requirements changed a couple of times, so this could also explain why something that looked necessary for scaling and speed, ended up being this over-engineered mess. This is how it worked: After javascript tracker fired, we got log files, passed them…

The analytics are not just limited to web clients. There would be API clients too. The deployment will be in a private enteprise vpn and so talking to external services may not be an option. I am aware of these tools like cassandra/flink/spark/kafka etc. But I am more curios about the best tools and architectural patterns that work well with each other.