Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise of Open Source Time Series Databases

victoriametrics.com

Post reply on HN
Print view View original

41–50 of 71 posts

Re: The Rise of Open Source Time Series Databases

#41

Post by ekianjo »

d0mine wrote: post #16
jnordwick wrote: post #9

If you are writing an article about time series databases, and you don't mention KDB - straight to jail. It is the grandfather of time series and predates influx by about a decade. It is still the fastest out there too. It is used by about every major financial and trading institution in the US and Europe. Everybody thinks TSDB are something new-ish, but they've been around since the days of APL. All you youngins dis…

kdb is not open-source, is it? https://news.ycombinator.com/item?id=19973847

correct, it is not

Re: The Rise of Open Source Time Series Databases

#42

Post by erbdex »

Graphite[1] has interesting plugin ecosystem like Grafana[2] & has been around since 2009. Architecture of Open Source Applications book[3] has done a nice overview of the architecture. I really like how it reuses linux filesystem as data store, making archiving very simple.

[1]: https://graphiteapp.org/ [2]: https://grafana.com/ [3]: https://aosabook.org/en/v1/graphite.html

Re: The Rise of Open Source Time Series Databases

#43

Post by peteforde »

osigurdson wrote: post #33
peteforde wrote: post #31

If the author ends up reading this... as someone who is ignorant but curious about time series databases, it was frustrating to see an example of what relational data looks like, but then no example of what time series data looks like. It might take a bit more context or setup, but this article is an opportunity to educate folks who aren't deep in the same trench. Edit: in fairness, this article doesn't describe itse…

The naive relational time series table looks like: propertyid, timestamp, value Then add a covering index so your reads only ever hit the index. This works completely fine for low billions of rows. After that suggest using clickhouse. It is less general but at large enough scale you need to make some tradeoffs. Completely fine to start with a relational DB in many cases though.

Thanks for this. Can we keep going?

The article mentioned stock prices, so let's use your schema:

LLY.NYSE, 1726377148, 924.38

SHOP.NYSE, 1726377216, 72.45

SHOP.NYSE, 1726377245, 72.41

LLY.NYSE, 1726377284, 924.39

LLY.NYSE, 1726377310, 924.36

Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concerned, time only flows in one direction. Periodicity is determined by the client application but could be throttled due to a bottleneck, resulting in fewer writes.

I can imagine how storing in blocks based on time ranges and the fact that time number goes up makes this very easy to index.

With even this simple example, it's clear what this type of datastore could be useful for, even though I'm not convinced you couldn't do pretty exciting things in Redis or sqlite3.

That said, I'm still significantly confused why the hell you can only store numeric values. It just seems like a very arbitrary constraint.

Re: The Rise of Open Source Time Series Databases

#44

Post by beagle3 »

Mongoose wrote: post #17

Had me until claiming that InfluxDB was the first mainstream TSDB in 2013 . OpenTSDB (2010)? Graphite (2008)? RRDtool (1999)? Maybe Influx took off in a way these prior projects didn't, but people have been storing time series data for decades.

Also K (1993) and A+ (1988) although the former only became public in 1998 or so, and the latter in 2003 - they were only available inside Morgan Stanley in the beginning IIRC.

Re: The Rise of Open Source Time Series Databases

#45

Post by dijit »

osigurdson wrote: post #35
Mongoose wrote: post #17

Had me until claiming that InfluxDB was the first mainstream TSDB in 2013 . OpenTSDB (2010)? Graphite (2008)? RRDtool (1999)? Maybe Influx took off in a way these prior projects didn't, but people have been storing time series data for decades.

InfluxDB always seemed like it was run by children who are good at raising VC money.

Agreed, after trying to buy a commercial license from them I was left… wanting to avoid the company entirely.

Re: The Rise of Open Source Time Series Databases

#46

Post by Too »

peteforde wrote: post #43
osigurdson wrote: post #33

Earlier quoted context omitted.

The naive relational time series table looks like: propertyid, timestamp, value Then add a covering index so your reads only ever hit the index. This works completely fine for low billions of rows. After that suggest using clickhouse. It is less general but at large enough scale you need to make some tradeoffs. Completely fine to start with a relational DB in many cases though.

Thanks for this. Can we keep going? The article mentioned stock prices, so let's use your schema: LLY.NYSE, 1726377148, 924.38 SHOP.NYSE, 1726377216, 72.45 SHOP.NYSE, 1726377245, 72.41 LLY.NYSE, 1726377284, 924.39 LLY.NYSE, 1726377310, 924.36 Okay, so you're appending values that capture a moment in time for a given index, with the understanding that you're almost never going to revise a tuple. So far as we're concer…

The constraint is not inherent to TSDB generally. Influxdb supports string values for example, as mentioned in the article. You also have log databases, like Loki, that are designed like a TSDB, except they only store strings.

My guess is that constraining to numbers greatly simplifies the implementation, especially so for the query language and aggregation functions.

Re: The Rise of Open Source Time Series Databases

#47

Post by Too »

starkparker wrote: post #5

This is unsurprisingly a VictoriaMetrics post that frames the history in a narrow way to talk favorably about VictoriaMetrics.

Conveniently forgetting to mention any of the other big Open Source players in the field. Cortex, Graphite, Mimir, Thanos, M3, TimescaleDB.

Re: The Rise of Open Source Time Series Databases

#48

Post by Too »

PeterZaitsev wrote: post #23
atombender wrote: post #14

Earlier quoted context omitted.

Mimir [1] is what we use where I work. We are very happy with it, and we have very long retention. Previously, our Prometheus setup was extremely slow if you went past today, but Mimir partitions the data to make it extremely fast to query even long time periods. We also used Thanos for a while, but Mimir apparently worked better. [1] https://grafana.com/oss/mimir/

Yeah. Would be interested to see how VictoriaMetrics compares to Mimir, not just Prometheus. To be fair many projects in Prometheus "long term store" space come and gone - Thanos, Cortex, M3

Can you elaborate a bit more on the come and gone part?

Re: The Rise of Open Source Time Series Databases

#50

Post by saswatamcode »

PeterZaitsev wrote: post #23
atombender wrote: post #14

Earlier quoted context omitted.

Mimir [1] is what we use where I work. We are very happy with it, and we have very long retention. Previously, our Prometheus setup was extremely slow if you went past today, but Mimir partitions the data to make it extremely fast to query even long time periods. We also used Thanos for a while, but Mimir apparently worked better. [1] https://grafana.com/oss/mimir/

Yeah. Would be interested to see how VictoriaMetrics compares to Mimir, not just Prometheus. To be fair many projects in Prometheus "long term store" space come and gone - Thanos, Cortex, M3

Hi! None of these are "gone" in any way, and are actually used in various different enterprises at global scale.
Post reply on HN