OpenData Timeseries: Prometheus-compatible metrics on object storage
1–10 of 18 posts
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#2The "why should I care" is about 3/4 of the way down the page:
> None of these numbers are exact, but the structural gap is clear: a handful of nodes costing roughly $560/month versus $10,000-20,000/month for a managed service at the same scale. As we explained earlier, it’s practical to operate OpenData Timeseries yourself and fully realize these massive cost savings since it isn’t a traditional distributed database that manages partitioned and replicated state.
It doesn't look 100% turn-key, but those are compelling numbers.
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#3The "why should I care" is about 3/4 of the way down the page: > None of these numbers are exact, but the structural gap is clear: a handful of nodes costing roughly $560/month versus $10,000-20,000/month for a managed service at the same scale. As we explained earlier, it’s practical to operate OpenData Timeseries yourself and fully realize these massive cost savings since it isn’t a traditional distributed database…
Good point, a tl;dr is probably worthwhile.
It's definitely not quite turn key just yet but we've been dogfooding it in production against a moderate metrics use case (~30k samples/s) and have it hooked up to grafana (you just configure a prometheus source and point to your deployed URL). We run it on a single node with no replicas ;)
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#4Wow this is so, so much cheaper than alternatives
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#5The "why should I care" is about 3/4 of the way down the page: > None of these numbers are exact, but the structural gap is clear: a handful of nodes costing roughly $560/month versus $10,000-20,000/month for a managed service at the same scale. As we explained earlier, it’s practical to operate OpenData Timeseries yourself and fully realize these massive cost savings since it isn’t a traditional distributed database…
Good call out, updated the intro with a summary of the cost benefit. Thanks for the feedback!
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#6Comparing self-hosted prices with managed solutions isn't exactly apples to apples.
But if you do compare, VictoriaMetrics cloud for 3Mil active series and twice higher ingestion rate (100K samples/s or 30s scrape interval) will cost you ~$1k/month + storage costs.
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#7Comparing self-hosted prices with managed solutions isn't exactly apples to apples. But if you do compare, VictoriaMetrics cloud for 3Mil active series and twice higher ingestion rate (100K samples/s or 30s scrape interval) will cost you ~$1k/month + storage costs. See https://victoriametrics.cloud/#estimate-cost
Agreed. VictoriaMetrics is indeed a very compelling offering. The disk-less approach is significantly simpler to operate, which I think is the biggest difference. Running opendata's version yourself has fewer moving pieces, and standard operations become trivial because no single service retains permanent state.
It's a meaningful change in calculation of running yourself vs paying someone to do it for you IMO.
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#8Comparing self-hosted prices with managed solutions isn't exactly apples to apples. But if you do compare, VictoriaMetrics cloud for 3Mil active series and twice higher ingestion rate (100K samples/s or 30s scrape interval) will cost you ~$1k/month + storage costs. See https://victoriametrics.cloud/#estimate-cost
[deleted]
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#9Comparing self-hosted prices with managed solutions isn't exactly apples to apples. But if you do compare, VictoriaMetrics cloud for 3Mil active series and twice higher ingestion rate (100K samples/s or 30s scrape interval) will cost you ~$1k/month + storage costs. See https://victoriametrics.cloud/#estimate-cost
Agreed. VictoriaMetrics is indeed a very compelling offering. The disk-less approach is significantly simpler to operate, which I think is the biggest difference. Running opendata's version yourself has fewer moving pieces, and standard operations become trivial because no single service retains permanent state. It's a meaningful change in calculation of running yourself vs paying someone to do it for you IMO.
anecdotally I've heard confirmations of the challenge of running VictoriaMetrics clusters at scale. they're way better than Cortex/Thanos and they've built a pretty awesome product but still are a pretty significant operational burden.
Re: OpenData Timeseries: Prometheus-compatible metrics on object storage
#10Interesting solution! According to the provided numbers at "query latency" chapter, the query over cold data, which selects samples for 497 time series over 6 hours time range takes 15 seconds if the queried data isn't available in the cache. This means that typical queries over historical data will take eternity to execute ;(