81–87 of 87 posts
Earlier quoted context omitted.
I worked in a 15m/year revenue product for 3 years. Our Analytic system was screw by cookie messages and now GDPR. Marketeers wanted to serve Analytics through Google Tag Manager, which helped customers to block our analytics launcher, meaning 0 data for most of the visits.
Relying solely on client-side logs gets less and less reliable each year.
Earlier quoted context omitted.
>>I'm more curios about the best tools and architectural patterns that work well with each other. Well you can go with: Fancy: Hdfs(distributed file system) as storage - oozie as workflow scheduler for your load(python/hive/scala/spark) - Tableau for visualization (your business ussers will love it. Mid range: SQL Server as storage - Informatica for your workload - power BI /SSRS for visuals Open/low budget: PostgreS…
Tableau has a terrible UI and looks like it hasn't been updated since 1995. Go for Looker instead.
Earlier quoted context omitted.
Pruning the file is done the same way as logrotate, or even with logrotate. You don't track state. When the loader starts up, you spool through the whole file and use ON DUPLICATE KEY IGNORE in your INSERT statement.
The nice thing about the queue is that you can do fanout (eg multiple SQS subscribers to the SNS topic or similar) and have multiple parallel ingestors or change out the backend consumer/db injector without the producers knowing anything about it. You also get HA for free; your solution depends on local node disk persistence.
We have reached a point where people think that local node disk persistence is an exotic property.
Earlier quoted context omitted.
The nice thing about the queue is that you can do fanout (eg multiple SQS subscribers to the SNS topic or similar) and have multiple parallel ingestors or change out the backend consumer/db injector without the producers knowing anything about it. You also get HA for free; your solution depends on local node disk persistence.
> your solution depends on local node disk persistence. We have reached a point where people think that local node disk persistence is an exotic property.
If we’re gonna sign our lives over to Amazon and Google anyway, we might as well get some reliability out of the exchange.
My go-to v0 solution is JSON (simple, with no nested objects or lists) written to s3 (partitioned by date, see Hive date partitioning) and AWS Athena (serverless Presto) to do SQL queries on those JSONs. You can build the system in less than an hour, you don't have to manage any VMs and it's relatively easy to extend to a more serious solution (e.g. if you need major scale or Spark-like analytic jobs). Relational dat…
Start by adopting https://github.com/snowplow/snowplow then grow as and where you feel restricted.
I would be suspicious of most tech talks on this. If someone is giving a tech talk on their analytics systems, they are either working at enormous scale (Facebook, Google), selling something (Splunk), or over engineering their system (many startups). I second advice elsewhere in this thread. Log it into PostgreSQL. If you start overloading that, look into sampling your data before you look into a fancier system. Make…
That's actually what we do at Rakam. Postgresql fits in many analytics workloads with partitioned tables, parallel queries, and BRIN indexes. The only limitation is that since it's not horizontally scalable, your data must fit in one server. `it just works` up to ~10M events per month. The SDKs provide ways to send the event data with the following format: rakam.logEvent('pageview', {url: ' https://test.com'} ) The e…