31–40 of 87 posts
I have seen so many developers over-engineer this exact problem. Forget about Kafka, Kinesis, Redshift, Airflow, Storm, Spark, Cassandra etc. You don't need them, not even close. Unless you want to add a bunch of expensive distributed systems and operational overhead for fun/resume building, they're going to waste your time and hurt your stability.
1) Design the reports you want. Pay special attention to interactive elements like filters and drilldowns. List all dimensions and metrics you need. Think about privacy. 2) Find your visualisation tool of choice. This is more important than any architecture choice for the tracking because this makes your data useable. [1] 3) Select your main data storage that is compatible with your visualisation tool, data size, bud…
Couple bits (good overall): "1) Design the reports you want. Pay special attention to interactive elements like filters and drilldowns. List all dimensions and metrics you need. Think about privacy." I think what you're getting at here is figure out what information you want to get and then work backwards to figure out if you have the data. A couple minor changes I'd make: A) don't just figure out a a report, figure…
Start by adopting https://github.com/snowplow/snowplow then grow as and where you feel restricted.
Seriously, just put everything in Postgres. You have so little data, you shouldn't even be thinking about an "analytics system". I have seen so many developers over-engineer this exact problem. Forget about Kafka, Kinesis, Redshift, Airflow, Storm, Spark, Cassandra etc. You don't need them, not even close. Unless you want to add a bunch of expensive distributed systems and operational overhead for fun/resume building…
Designing data intensive systems, M. Kleppmann
Thanks. But it is a bit theoretical. I was looking more on the lines of people's practical experience, with architectural choices, tools that they used, etc.
Start by adopting https://github.com/snowplow/snowplow then grow as and where you feel restricted.
You'll be a lot better off spending your mental energy thinking about the outcomes you want to achieve (user engagement, upselling, growth, etc) and the types of analysis you'll need to understand what changes you need to make to produce those outcomes. Protip: this is actually really hard, and people underestimate it by orders of magnitude. A blog post by Roger Peng (with indirect commentary from John Tukey) ... https://simplystatistics.org/2019/04/17/tukey-design-thinkin...
One other immediate tip is to start thinking about correlating your telemetry with user surveys - again, strongly focusing on outcomes and the controllable aspects of those outcomes.
Don't let the data lead the discuisson; decide on the question you're asking, and the implications of all of the possible answers to that question (clearly yes, clearly no, mixed, etc) before you ask it.
Then engineer the lightest weight system possible to ingest, process, store, analyze, and visualize that data.
For me, that would just be:
1. Log data in whatever logging tool you like. Persist the raw stuff forever in a cheap data lake. 2. Batch at some fixed interval into a staging area of a relational DB. 3. Transform it with stored procedures for now (while you figure out what the right transforms are) into a flat fact table. 4. Visualize in Superset or PowerBI or even plain old Excel.
Once you've got the patterns of analysis at least fundamentally right you can consider stream processing (Flink or Kafka Streams are fine) to replace 2 and 3.
Structure your code so you crunch your historical data once, store in redis, and then new data gets shoved in the redis cache as your time dimensions on your metrics progress based on business logic.
Until your data is at enterprise volume, you really don't need an OLAP system.
Seriously, just put everything in Postgres. You have so little data, you shouldn't even be thinking about an "analytics system". I have seen so many developers over-engineer this exact problem. Forget about Kafka, Kinesis, Redshift, Airflow, Storm, Spark, Cassandra etc. You don't need them, not even close. Unless you want to add a bunch of expensive distributed systems and operational overhead for fun/resume building…
Also, assuming that SQL is being used for storing analytics events, would you not cache events in a queue and then flush them to DB in a batch?