Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Good tech talks on how analytics systems are implemented?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 87 posts

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#41

Post by jrumbut »

psankar wrote: post #9
karanke wrote: post #6

Note: I build and maintain such systems for a living. There's a lot of context that's missing from your post, some questions that can help us guide you in the right direction: 1) Can your website call out to external services, or are you limited to operating behind a company network? 2) Is this more of an ad-hoc analysis or do you want to invest in a framework to be able to track such metrics systematically over time…

1) The website cannot call to external services, which is the primary reason why we thought about implementing it from scratch. 2) We want to invest in building a good framework to track such metrics systematically over time 3) We have some non-web API clients too. Adblock is not a problem. 4) Accuracy is better. Speed is not that critical and could even be a few minutes delayed. 5) The data will be kept for a few mo…

I would say what you need to consider more than anything is making sure you have the right data, and that the data can be combined.

This is the hard part of analytics for an app that is more back office oriented, understanding what will be needed to get truly accurate and useful information to support the reports people will want in 2 years, 5 years, whatever time frame is long enough for things to really change in your environment.

Try to think in an adversarial way, what question could someone come up with that I can't answer. The user who has seen the most errors? Usage trends by department?

One place this might lead is wanting to put a way to link request/error logs back to an application level user account (in a way that respects security and privacy), this can become great debugging info too.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#42

Post by dougk16 »

manishsharan wrote: post #40
sfkdjf9j3j wrote: post #31

Seriously, just put everything in Postgres. You have so little data, you shouldn't even be thinking about an "analytics system". I have seen so many developers over-engineer this exact problem. Forget about Kafka, Kinesis, Redshift, Airflow, Storm, Spark, Cassandra etc. You don't need them, not even close. Unless you want to add a bunch of expensive distributed systems and operational overhead for fun/resume building…

wouldn't writing events from analytics sub-system into a SQL database put load on the DB even when it is not warranted ? My point is that if you are using a SQL db and most of the database is for business transactions, wouldn't logging events which are not mission critical , unnecessarily consume db resources ? Also, assuming that SQL is being used for storing analytics events, would you not cache events in a queue a…

Typically you have a separate database just for analytics, exactly for the reason you described. You don't want to hold up actual business logic.

And yes typically you also have a job queue for the same reason. So e.g. API calls don't take longer due to database event writes.

If your needs are similar to the original questioner (still starting out, few thousand users), I'd say even the separate database isn't necessary, just the job queue. Heck even the job queue is probably optional at that scale, but most hosting platforms make job queues easy enough to integrate so I say why not.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#43

Post by gfodor »

manishsharan wrote: post #40
sfkdjf9j3j wrote: post #31

Seriously, just put everything in Postgres. You have so little data, you shouldn't even be thinking about an "analytics system". I have seen so many developers over-engineer this exact problem. Forget about Kafka, Kinesis, Redshift, Airflow, Storm, Spark, Cassandra etc. You don't need them, not even close. Unless you want to add a bunch of expensive distributed systems and operational overhead for fun/resume building…

wouldn't writing events from analytics sub-system into a SQL database put load on the DB even when it is not warranted ? My point is that if you are using a SQL db and most of the database is for business transactions, wouldn't logging events which are not mission critical , unnecessarily consume db resources ? Also, assuming that SQL is being used for storing analytics events, would you not cache events in a queue a…

We put unwarranted load on computers all the time, and nobody notices. They're good at that. Now, if you think the load we are talking about will actually be so much that humans will be affected by it, then it makes sense to reconsider. But at 1000 DAU, you could probably run the database (analytics logging included) on a Linux server from the 1990s and its CPU and disk would not be the transaction bottleneck. (With proper indexing ofc.)

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#45

Post by ironchef »

srrr wrote: post #32
ironchef wrote: post #30

Earlier quoted context omitted.

Couple bits (good overall): "1) Design the reports you want. Pay special attention to interactive elements like filters and drilldowns. List all dimensions and metrics you need. Think about privacy." I think what you're getting at here is figure out what information you want to get and then work backwards to figure out if you have the data. A couple minor changes I'd make: A) don't just figure out a a report, figure…

Good additions. For report design I found the book "Information Dashboard Design" by Stephen Few valuable. It talks about actionable data and has many examples.

I hadn't heard of that book before. Thank you.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#46

Post by lilbobbytables »

psankar wrote: post #10
serial_dev wrote: post #7

I was working for a startup implementing analytics tools. In my opinion, our setup was over-engineered, but I wasn't there at the beginning, so I might be wrong. Also, requirements changed a couple of times, so this could also explain why something that looked necessary for scaling and speed, ended up being this over-engineered mess. This is how it worked: After javascript tracker fired, we got log files, passed them…

The analytics are not just limited to web clients. There would be API clients too. The deployment will be in a private enteprise vpn and so talking to external services may not be an option. I am aware of these tools like cassandra/flink/spark/kafka etc. But I am more curios about the best tools and architectural patterns that work well with each other.

There has to be something somewhat out of the box for this, no? It seems insane to have to build this for any given project.

I'd imagine you could use Segment (or source(s) that plugs into it) to accomplish much of this.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#47

Post by madhadron »

sneak wrote: post #21
sfifs wrote: post #18

If it's a only few thousand enterprise users, I'd actually say log to a bog standard relational database from server side like MySQL or Postgres. Think through table schemas for everything you're going to log and make sure primary keys and nomenclature for everything talk to each other. Virtually any analytics platform or software talks to standard databases. Record as much as you can because analytics use cases typi…

Suggestion: send the events to an elastic queue (like SQS) first, and have a pair of ingestion processes do the actual INSERTs when the db is available. Then you can take the db offline for maintenance and upgrades and not lose data.

Alternately, log to disk and have another process ingest and write to the database. Appending to files is an amazing persistent queue.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#48

Post by madhadron »

I would be suspicious of most tech talks on this. If someone is giving a tech talk on their analytics systems, they are either working at enormous scale (Facebook, Google), selling something (Splunk), or over engineering their system (many startups).

I second advice elsewhere in this thread. Log it into PostgreSQL. If you start overloading that, look into sampling your data before you look into a fancier system. Make sure you have identifiers in each row for each entity a row is part of: user, session, web request. If you're not building replicated PostgreSQL (which you probably won't need for this), log to files first, and build another little process that tails the files and loads the rows into PostgreSQL. That log then load advice is hard learned experience from working at Splunk.

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#49

Post by awinder »

madhadron wrote: post #47
sneak wrote: post #21

Earlier quoted context omitted.

Suggestion: send the events to an elastic queue (like SQS) first, and have a pair of ingestion processes do the actual INSERTs when the db is available. Then you can take the db offline for maintenance and upgrades and not lose data.

Alternately, log to disk and have another process ingest and write to the database. Appending to files is an amazing persistent queue.

How does your writer process deal with failures / tracking state of what it has written / prune the file when it doesn't need old data anymore? You don't have to use SQS but this problem has tons of available options, I wouldn't resort to rolling your own (if anything just pick up something that works off a leveldb/sqlite/etc file and has already implemented all this boring stuff for you)

Re: Ask HN: Good tech talks on how analytics systems are implemented?

#50

Post by petercooper »

If your analytics are merely a 'nice to have' but losing a day or two of results would be acceptable in a crisis, I'd log everything to Redis and then run a daily report to drag aggregated values into another database system. I would clogging your main database system up with analytics related queries on a day to day basis, for sure.
Post reply on HN