Ask HN: Good tech talks on how analytics systems are implemented?

That's actually what we do at Rakam. Postgresql fits in many analytics workloads with partitioned tables, parallel queries, and BRIN indexes. The only limitation is that since it's not horizontally scalable, your data must fit in one server. `it just works` up to ~10M events per month.

The SDKs provide ways to send the event data with the following format:

rakam.logEvent('pageview', {url: 'https://test.com'})

The event types and attributes are all dynamic. The API server automatically infers the attribute types from the JSON blob and creates the tables which correspond to event types and the columns which correspond to event attributes and inserts the data into that table. It also enriches the events with visitor information such as user agent, location, referrer, etc. The users just run the following SQL query:

SELECT url, count(*) from pageview where _city = 'New York' group by 1

All the project is open-source: https://github.com/rakam-io/rakam Would love to get some contribution!

This sounds like a classic case of build vs. buy. If analytics are not your core product, inventing a new solution is going to cost you more than buying an existing analytics solution. There are dozens, a few of which have even been in the news the last few days due to acquisitions.

I'm not going to endorse any of them over the others, but I will say you'll be better off using a 3rd party than coding this yourself.

I helped build an analytics platform that served more than millions of events. Nothing about it is really difficult until you scale heavily.

The tl;dr - It's not worth your time / energy to build from scratch at this scale. Leveraging an existing standard like the analytics.js spec [1] makes web analytics very easy and quick to get started. With this approach you just have to add one JS snippet to your website and never need to update it in your code. If you are interested in the internals, you might enjoy digging deeper into the spec to understand why it was designed this way.

Two services that implement this spec are Segment [2] and MetaRouter [3] [full disclosure: I helped build the product that became MetaRouter at a previous job]. They have different target audiences and pricing models but both are worth a look.

You can think of these types of services as a meta analytics service that routes your events to destination analytics services and data stores of your choice. The great thing about using the standard is you can benefit from all of the many integrations that have already been created with various analytics services, databases, data warehouses, etc [4]. These destination catalogs can also help you decide what services to explore and try next as you need more advanced features.

To get started with a meta analytics service, in the management dashboard, just add your API keys and config values for each service. For a simple service like Google Analytics this is literally just one simple key to copy and paste.

As far as adding custom even monitoring to your site, within the analytics.js spec, first, you mainly want to be concerned with the Track call [5] which is a way to say for an arbitrary event e.g., ProductAddedToCart, I would like to attach this JSON object of properties e.g., a product name and price.

And finally, user info like name, email, IP, etc are handled by Identify [6]. You can add custom fields too (traits on an identify are ~= properties to a track event but less transient).

Going with an existing standard and SaaS-based approach will save a ton of time and engineering effort.

[1]: https://segment.com/docs/spec/

[2]: https://segment.com/

[3]: https://www.metarouter.io/

[4]: https://segment.com/docs/destinations/

[5]: https://segment.com/docs/spec/track/#example

[6]: https://segment.com/docs/spec/identify/#example

The nice thing about the queue is that you can do fanout (eg multiple SQS subscribers to the SNS topic or similar) and have multiple parallel ingestors or change out the backend consumer/db injector without the producers knowing anything about it.

You also get HA for free; your solution depends on local node disk persistence.

For such a small scale, you can use a simple event tracking schema from your client-side and server-side code and have a simple stream processor to join these events and then save it to a simple event table in a SQL database. The DB tech you choose should be something suitable for OLAP workloads. For your scale, PostgreSQL or MySQL would just work fine. When your data grows you can look at more distributed systems like Vertica or Memsql or Clickhouse etc.

In this architecture, most of your brain cycles will go into designing the queries for generating aggregates at regular intervals from the raw events table and storing in various aggregate tables. You must be familiar with facts and dimensions tables as understood in data warehouse context.