Ask HN: Good tech talks on how analytics systems are implemented?

I am doing a new sub-system, for analytics, which I can design / implement from scratch. I get a bunch of unique users (say a few thousands). Now I need to track each of these users and do some analytics (Which places are they logging in from ? How long do their sessions last on average ? What are the usual links/endpoints that they visit ? etc.) in my API server. I have a few thousand active users and about two dozen parameters on which I want to track and analyze these parameters.

I have never implemented such an analytics type system. I want to learn about people who have already implemented similar systems. Are there any good tech talks, engineering blog posts, video courses, etc. that highlight the design/technology/architecture choices and their benefits.

Designing data intensive systems, M. Kleppmann

I was working for a startup implementing analytics tools. In my opinion, our setup was over-engineered, but I wasn't there at the beginning, so I might be wrong. Also, requirements changed a couple of times, so this could also explain why something that looked necessary for scaling and speed, ended up being this over-engineered mess. This is how it worked: After javascript tracker fired, we got log files, passed them through Kafka, then parsed the log files and performed calculations through Storm (Java). For storage, we used Cassandra. The system also had other parts, but I don't remember why they were there, tbh.

My thought process for solving your problem would be the following. First, you need to understand what's good for you and for your company might not be the same. You want the challenge, you want to implement something that could scale and you want to use exotic tools for achieving this. It's interesting and looks good in your CV. Your company might just want the results. You need to decide which is more important.

If we prioritize your companies needs over keeping you entertained, I'd follow this thought process:

Can't you just use Google Analytics? You can also connect it to BigQuery and do lots of customizations. Maybe time would be better spent learning GA. It's powerful, but most of us cannot use it well.

Second question: if for some reason, you don't want to use Google Analytics, can you use another, possibly open-source and/or self-hosted analytics solution? Only because you can design it from scratch, it doesn't mean you should.

Third: Alright, you want to implement something from scratch. For this scale, you can probably just log and store events in an SQL database, write the queries, and display it in a dashboard.

Then, if you really want to go further, there are many tools that are designed to scale well and perform analytics, "big data". By looking for talks about these tools, you will get a better understanding of how things work. There are various open-source projects you should read more about: Cassandra, Scylla, Spark, Storm, Flink, Hadoop, Kafka, Hadoop, Parquet, just to name a few.

1) The website cannot call to external services, which is the primary reason why we thought about implementing it from scratch.

2) We want to invest in building a good framework to track such metrics systematically over time

3) We have some non-web API clients too. Adblock is not a problem.

4) Accuracy is better. Speed is not that critical and could even be a few minutes delayed.

5) The data will be kept for a few months at least, if not years. The storage costs are not a big problem. This is for enterprise based solution, where the deployment may happen in a private network.