Ask HN: Good tech talks on how analytics systems are implemented?

Ahoy https://github.com/ankane/ahoy is an interesting tool that we use to replace Google Analytics in most projects now.

It covers all the basic needs, and even if you're not using Rails, I think you can draw inspiration from it!

Start by adopting https://github.com/snowplow/snowplow then grow as and where you feel restricted.

I’ve worked with a GA implementation before which I don’t recommend if you want to own your data or if you want unsampled, detailed logs. I’ve also seen a full end to end implementation that uses server log shipping to s3, log parsing and complicated ETL processes which I also don’t recommend due to the sheer effort it would take to build.

I’d say go with something like Matomo (formerly Piwik) https://matomo.org. If you wanted to build your own, I’d suggest keeping it simple. Look at Matomo’s architecture and replicate https://github.com/matomo-org/matomo.

I have an open-source project that collects the customer events via SDKs and stores it in a data warehouse.

It's a distributed system, the mobile and web SDKs batch the user events on their devices and push it to our API in JSON format. The API servers enrich & sanitize the data, validate the schema, convert it to a serialized AVRO binary, and push it to a commit-log system such as Kinesis or Kafka (It's pluggable).

We have another project that fetches data from Kafka & Kinesis in small batches, converts the data into columnar format and stores it in an S3 bucket / Google Cloud Storage. Then, we integrate their preferred data-warehouse into their distributed filesystem. That way they have all their raw data in their infrastructure for other systems such as fraud detection, recommendation, etc. but they have SQL access to their data as well.

That being said, this architecture is for >100M events per month. If your data is not that much, you can actually ingest your data into an RDBMS and it just works fine. We support Postgresql at Rakam and you need is the API server and a Postgresql instance in that case. Our open-source version supports Postgresql so you can look into the source code from here: https://github.com/rakam-io/rakam Would love to get some contribution. :)

For the analysis part, all these metrics can be created using just SQL, the modern data-warehouse solutions (BigQuery and Snowflake) also support javascript and it's relatively easy to build funnel & retention queries that way. It requires more work but now you have more control & flexibility over your data.

Couple bits (good overall): "1) Design the reports you want. Pay special attention to interactive elements like filters and drilldowns. List all dimensions and metrics you need. Think about privacy." I think what you're getting at here is figure out what information you want to get and then work backwards to figure out if you have the data. A couple minor changes I'd make: A) don't just figure out a a report, figure out what actions you'd want to see. If something gets above or below a threshold, who should be doing what? (Reports for the sake of reports is generally bad) B) Are you trying to build things that will push for operational, tactical, or strategic change? The manifestations of those are often very different. Operational bits are often dashboards / KPIs, whereas with strategic changes we often would want to present something more akin to a story. C) Privacy - Think of GDPR / PII _now_. Look at each metric / dimension and understand the data classification of it.

2 and 3 are tied to each other. You could have visualizatoin drive storage or visa versa. Just understand the tradeoffs.

I'd suggest for who's done it before / talks, etc. there are a ton out there. There's those chats from the FAANGs and various groups in the valley (Lyft, etc.). Tons of blog posts there. Vendors have (largely predisposed towards them) builds. Finally the talks/slides at datacouncil and strata often contain lots of more .. "pointed" information. The high level bible that lots of folks would say look at is Kleppmann's "Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems".