Earlier quoted context omitted.
Mimir [1] is what we use where I work. We are very happy with it, and we have very long retention. Previously, our Prometheus setup was extremely slow if you went past today, but Mimir partitions the data to make it extremely fast to query even long time periods. We also used Thanos for a while, but Mimir apparently worked better. [1] https://grafana.com/oss/mimir/
Yeah. Would be interested to see how VictoriaMetrics compares to Mimir, not just Prometheus. To be fair many projects in Prometheus "long term store" space come and gone - Thanos, Cortex, M3
The Rise of Open Source Time Series Databases
61–70 of 71 posts
Re: The Rise of Open Source Time Series Databases
#62Can anyone comment why these are superior to ClickHouse? I really like the ClickBench which compares the various products performance (and open source).
Re: The Rise of Open Source Time Series Databases
#63Re: The Rise of Open Source Time Series Databases
#64Earlier quoted context omitted.
The naive relational time series table looks like: propertyid, timestamp, value Then add a covering index so your reads only ever hit the index. This works completely fine for low billions of rows. After that suggest using clickhouse. It is less general but at large enough scale you need to make some tradeoffs. Completely fine to start with a relational DB in many cases though.
Thanks for this. Can we keep going? The article mentioned stock prices, so let's use your schema: LLY.NYSE, 1726377148, 924.38 SHOP.NYSE, 1726377216, 72.45 SHOP.NYSE, 1726377245, 72.41 LLY.NYSE, 1726377284, 924.39 LLY.NYSE, 1726377310, 924.36 Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concer…
The key is the covering index, you don't want to hit the actual table at all. You should re-organize the index periodically as well in order to ensure optimal query times.
This is suboptimal in many ways as the records are tiny yet there is per row overhead. However, it works fine to low billions of rows in general. I think it would work fine in SQLite as well. Redis is a very different animal however so a different strategy would be needed.
Migrating the same schema to ClickHouse when you need it (I have no affiliation with them), will give you 50X lower data footprint, increase your ingest rates by 10X and your query times by about 2X. However, if you don't really need that, don't bother adding another component in your infrastructure (imo). Also, ClickHouse is much more of a purpose built Ferrari. I.e. you can't expect it to do all kinds of "off schema" type queries (i.e. give me all the stock values on Jan 10, @ 10:32:02.232am). In a relational DB you can add indexes and get results reasonably well. In ClickHouse schema == query pattern essentially. Naturally, you should know what you are doing with both models but in ClickHouse, don't treat it like a "magic data box" that will can handle anything you throw at it (beyond a certain scale at least).
Re: The Rise of Open Source Time Series Databases
#65Earlier quoted context omitted.
Thanks for this. Can we keep going? The article mentioned stock prices, so let's use your schema: LLY.NYSE, 1726377148, 924.38 SHOP.NYSE, 1726377216, 72.45 SHOP.NYSE, 1726377245, 72.41 LLY.NYSE, 1726377284, 924.39 LLY.NYSE, 1726377310, 924.36 Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concer…
> That said, I'm still significantly confused why the hell you can only store numeric values. It just seems like a very arbitrary constraint. "In mathematics, a time series is a series of data points indexed (or listed or graphed) in time order." A a numeric datapoint is by far the largest use case, timeseries databases optimize for that. Most allow you to have several labels ofcourse, i.e. you could store country bo…
However, strings aren't as common as usually you are a value of some sort.
Re: The Rise of Open Source Time Series Databases
#66Earlier quoted context omitted.
Thanks for this. Can we keep going? The article mentioned stock prices, so let's use your schema: LLY.NYSE, 1726377148, 924.38 SHOP.NYSE, 1726377216, 72.45 SHOP.NYSE, 1726377245, 72.41 LLY.NYSE, 1726377284, 924.39 LLY.NYSE, 1726377310, 924.36 Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concer…
Non-TSDB databases can do an adequate job — up to a certain point. However, getting good performance comes down to query access paths. For enormous amounts of data, you want your data to be sequential (a few batch reads are generally faster than lots of random ones) and sorted. Databases like Postgres don't store rows in index order, and their indexes are geared towards small fetches as opposed to big ranges of seque…
Re: The Rise of Open Source Time Series Databases
#67Earlier quoted context omitted.
Thanks for this. Can we keep going? The article mentioned stock prices, so let's use your schema: LLY.NYSE, 1726377148, 924.38 SHOP.NYSE, 1726377216, 72.45 SHOP.NYSE, 1726377245, 72.41 LLY.NYSE, 1726377284, 924.39 LLY.NYSE, 1726377310, 924.36 Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concer…
The constraint is not inherent to TSDB generally. Influxdb supports string values for example, as mentioned in the article. You also have log databases, like Loki, that are designed like a TSDB, except they only store strings. My guess is that constraining to numbers greatly simplifies the implementation, especially so for the query language and aggregation functions.
Re: The Rise of Open Source Time Series Databases
#68Can anyone comment why these are superior to ClickHouse? I really like the ClickBench which compares the various products performance (and open source).
Re: The Rise of Open Source Time Series Databases
#69Earlier quoted context omitted.
Thanks for this. Can we keep going? The article mentioned stock prices, so let's use your schema: LLY.NYSE, 1726377148, 924.38 SHOP.NYSE, 1726377216, 72.45 SHOP.NYSE, 1726377245, 72.41 LLY.NYSE, 1726377284, 924.39 LLY.NYSE, 1726377310, 924.36 Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concer…
Non-TSDB databases can do an adequate job — up to a certain point. However, getting good performance comes down to query access paths. For enormous amounts of data, you want your data to be sequential (a few batch reads are generally faster than lots of random ones) and sorted. Databases like Postgres don't store rows in index order, and their indexes are geared towards small fetches as opposed to big ranges of seque…
Re: The Rise of Open Source Time Series Databases
#70My biggest beef with VictoriaMetrics is I can't delete and re-ingest a range of data. i.e. if I discover yesterdays data was messed up due to some issue in downstream data processing, I can't delete yesterdays data and re-ingest the cleaned up data. (Or even just delete yesterdays bad data) The only workaround I've found so far is to dump out the whole timeseries which could go back months/years, delete the timeserie…
I will look up the query and get back to you