Live data from Hacker News

Show HN: Trifle – Open-source analytics that stores answers, not events

trifle.io

11–20 of 22 posts

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#11

How is this different fro Prometheus and its associated time series db? that looks like exactly how prom works, give or take. (and some of the negatives around stats that come with it, e.g. p95 calculation)

you’re right that there are some similarities here. you can do histogram same way in Trifle just as in Prometheus. and the p95 issue is in both. sum and counts can only get you that far. you can get normal approximation of percentiles, but its a compromise you make.

where the differences are clearer is who is it aimed for. Prometheus is a server you need to operate while Trifle pushes data into a database you already own. and then there’s a part that Prometheus scrapes for your data and you need a pushgateway to be able to push to it. one is build for infrastructure monitoring and the other is a library you use to push your increments.

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#12
post #4
post #3

A billion events is only ~4GB on disk. With disk so cheap (even now), why throw that away immediately instead of at least waiting 30 days, 1 year, etc. until you know you have your analysis right?

That's a fair point, and honestly disk cost rarely is the blocker these days. ~4GB assumes a few bytes per event, in our case it is closer to 100GB a day. That would be stored as raw text. Being able to rebuild that would require us to have some kind of pipeline that can do that effectively and that kind of defeats the point of Trifle. That said, it really depends on the use case. For us the last 24h-48h holds the mo…

Mentioning below in case it helps someone looking into such estimates:

For a website analytics product that I was building, where every page-view/event stores many values such as browser, device, os, page-params, referrer.. and other typical stuff, it was coming at about 200GB/Billion events, after DB-level compression... the database obviously was columnar.

At these levels of storage needs, things start to cost real deal. With expected monthly intake of just around 10 Billion events, that ramps up to 2TB added every month.

Because of this, I had to move to S3-scaled/backed, nvme-cached architecture, but, that made query times fluctuate to some noticeable extent.

So, that ~4GB-only is for teeny-tiny data per metric sample.. probably 3-4 numeric values at max.

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#14

How is this different fro Prometheus and its associated time series db? that looks like exactly how prom works, give or take. (and some of the negatives around stats that come with it, e.g. p95 calculation)

Exactly. That is the first question the web page should answer.

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#15
post #14

How is this different fro Prometheus and its associated time series db? that looks like exactly how prom works, give or take. (and some of the negatives around stats that come with it, e.g. p95 calculation)

Exactly. That is the first question the web page should answer.

That's a good callout. I had a basic comparison stuffed away on a different page, but you're right that it belongs on the landing page. so I've moved it there as well. I'll work on more in-depth comparisons against specific tools. Thanks!

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#16
What is the stack that is required to setup your opensource repo? and what's the machine config you recommend.

btw, I liked the pricing page. I am planning to setup a similar pricing page my project deepsql.ai (dba agent for postgres and mysql).

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#17

What is the stack that is required to setup your opensource repo? and what's the machine config you recommend. btw, I liked the pricing page. I am planning to setup a similar pricing page my project deepsql.ai (dba agent for postgres and mysql).

The library has no stack, it's just a set of plugins you can either use directly (Ruby, Elixir, Go) or with a framework (Rails, Sinatra, Phoenix, Ash, etc). They write into a database you already run. Setup is adding the plugin and pointing it at your DB. Redis, Postgres, MongoDB, MySQL or SQLite. As for when you would feel some load from it, it really depends on your volume. I would say somewhere around 100k+ events a day is where the load starts being noticeable. Until then I wouldn't worry much about extra load.

If you want to self-host the App, it's pretty lightweight. While it's an Elixir app and you could compile and run it yourself, it comes packaged as a Docker image and it has a Kubernetes/Helm deployment documented. You can either point it to your own Postgres or it will launch its own so it can hold some of its own data like users, dashboards, monitors, etc. For 2-3 users one instance is plenty and you can easily run that on 1 CPU and 2GB RAM. Double that if you want redundancy.

Good luck with deepsql!

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#19

You mention OTEL stuff in passing (Jaeger, etc). Did you ever consider using an open standard like OpenTelemetry? I admit it's challenging to get it going but standards can be helpful.

tbh it's partly timing and partly layer. the origins of Trifle go back to 2015 and when I revived it in 2021 OTel was still quite new. it wasn't really an "instead of" decision back then.

but I would also say they sit at different layers. while OTel is a standard for emitting telemetry and data still ends up being stored in some backend (jaeger for traces, prometheus for metrics), Trifle keeps data in the database you already have. which was kinda my point from the beginning, you don't need a dedicated database/stack to do basic analytics. I believe the audiences differ a bit as well. OTel grew towards infra/observability, while Trifle mostly ends up tracking product/business counters. for now I don't see these as competition.

back in 2021 I also started working on Trifle::Traces. which is a really (really!) simple tracing library for Ruby to wrap your code in different tracing blocks - I really disliked the idea of having puts statements everywhere and then patching stuff together in raw logs. at the end you get nicely structured output that is then completely up to you how you store it. we ended up putting searchable metadata in MongoDB and actual data in S3. then we built a small internal UI around it. ngl it sounds awfully like OTel tracing and jaeger.

so yea, that's a fair question. if OTel had been where it is today back then, maybe I would have fewer repos. I believe we often get lost thinking that we need these big tools to do basic things. to me, this is the gap that Trifle fills.

Re: Show HN: Trifle – Open-source analytics that stores answers, not events

#20
post #19

You mention OTEL stuff in passing (Jaeger, etc). Did you ever consider using an open standard like OpenTelemetry? I admit it's challenging to get it going but standards can be helpful.

tbh it's partly timing and partly layer. the origins of Trifle go back to 2015 and when I revived it in 2021 OTel was still quite new. it wasn't really an "instead of" decision back then. but I would also say they sit at different layers. while OTel is a standard for emitting telemetry and data still ends up being stored in some backend (jaeger for traces, prometheus for metrics), Trifle keeps data in the database yo…

Thanks for the response - that makes a lot of sense. I wonder how hard it would be to make a compatibility layer in the future.
Post reply on HN