Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Good tech talks on how analytics systems are implemented?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 87 posts

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#51

Post by btown »

madhadron wrote: post #48

I would be suspicious of most tech talks on this. If someone is giving a tech talk on their analytics systems, they are either working at enormous scale (Facebook, Google), selling something (Splunk), or over engineering their system (many startups). I second advice elsewhere in this thread. Log it into PostgreSQL. If you start overloading that, look into sampling your data before you look into a fancier system. Make…

This also makes a lot of things really easy. Want to join against your products table to see what product categories are most popular for certain customer segments? It’s a single query or Tableau drag-and-drop away. You don’t know what you’ll need to access fast to answer business questions, so use a system designed for flexibility until you can’t.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#52

Post by okjustgo »

madhadron wrote: post #48

I would be suspicious of most tech talks on this. If someone is giving a tech talk on their analytics systems, they are either working at enormous scale (Facebook, Google), selling something (Splunk), or over engineering their system (many startups). I second advice elsewhere in this thread. Log it into PostgreSQL. If you start overloading that, look into sampling your data before you look into a fancier system. Make…

Can you elaborate on the log then load advice? Specifically the problems it solves or issues it prevents?

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#55

Post by psds2 »

okjustgo wrote: post #52
madhadron wrote: post #48

I would be suspicious of most tech talks on this. If someone is giving a tech talk on their analytics systems, they are either working at enormous scale (Facebook, Google), selling something (Splunk), or over engineering their system (many startups). I second advice elsewhere in this thread. Log it into PostgreSQL. If you start overloading that, look into sampling your data before you look into a fancier system. Make…

Can you elaborate on the log then load advice? Specifically the problems it solves or issues it prevents?

If you mess up the database you can just reread the log files. It also helps manage backpressure during activity spikes where your db can't keep up.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#56

Post by reilly3000 »

Some questions for you:

- Who will be viewing these reports when they are done? Who do you want to have a view of the data eventually?

- How fresh do you need the data to be? Is 24 hours, 4 hours, or 4 seconds okay to wait?

- Do you need to be alerted of anomalies in the data?

- How long do you intend to store the raw data? Aggregated data?

- Does your data need to contain anything that could personally identify a user in order to make a useful analysis? Do you serve customers in the EU?

I'll check back later today and see if I can provide any insights based on your response.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#57

Post by C1sc0cat »

madhadron wrote: post #48

I would be suspicious of most tech talks on this. If someone is giving a tech talk on their analytics systems, they are either working at enormous scale (Facebook, Google), selling something (Splunk), or over engineering their system (many startups). I second advice elsewhere in this thread. Log it into PostgreSQL. If you start overloading that, look into sampling your data before you look into a fancier system. Make…

Dumping out to a text file quickly and have some aysnronous queue insert that into your database Is one solution but you have to watch uniqifiers, reconciliation and handling failed inserts so you can fix and reinject any failed records.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#58

Post by gorkemcetin »

Definitely not a "good talk/blog post/video course" type of thing, however if you are interested on how we built Countly from ground up, together with the technology stack behind, you can check our source code here:

https://github.com/countly/countly-server

While we have used MongoDB, Nodejs, Linux as underlying platforms, there are several options out there you may check.

Note that some (if not most) of the effort would go into SDK development, and to tell you the truth, SDK development is no easy task as it requires knowledge of how different platforms work.

The point is (and take it as a warning): you will never be satisfied with what you have - after you are done with vital data, then there is custom events, raw data, user profiles, online users, and then you will start eating your own dog food as it becomes your part-time job.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#59

Post by madhadron »

awinder wrote: post #49
madhadron wrote: post #47

Earlier quoted context omitted.

Alternately, log to disk and have another process ingest and write to the database. Appending to files is an amazing persistent queue.

How does your writer process deal with failures / tracking state of what it has written / prune the file when it doesn't need old data anymore? You don't have to use SQS but this problem has tons of available options, I wouldn't resort to rolling your own (if anything just pick up something that works off a leveldb/sqlite/etc file and has already implemented all this boring stuff for you)

Pruning the file is done the same way as logrotate, or even with logrotate. You don't track state. When the loader starts up, you spool through the whole file and use ON DUPLICATE KEY IGNORE in your INSERT statement.
Post reply on HN