Earlier quoted context omitted.
> aggregation you'd want from numeric, labelled metrics. We use it exactly for this, works great.
Do you use the new metrics store? I just learned they have one now in the new versions.
No we don't use metrics store.
Why Not to Build a Time-Series Database
121–128 of 128 posts
Re: Why Not to Build a Time-Series Database
#122Earlier quoted context omitted.
You don't have "missing rows". For time series A you have a time point at 12:01 AM, a datapoint at 12:02 AM, and another datapoint two weeks later at 5:04 PM. For time series B the times are different. You need some notion of whatever state the physical system was in at any given time.
Yes, I understand this as the "last value recorded" concept in my comment. KDB+ supports this with "asof" joins. Others can just do it by scanning a wider time frame or the entire table. Null gaps in a columnstore can be skipped over basically instantaneously and usually are just zone map/index lookups. Again I question how common this query is and whether it's really worth limiting yourself to a special TSDB because…
In our case, "last value" isn't good enough. We do interpolation, and use compression algorithms for which interpolation minimizes reconstruction error.
Re: Why Not to Build a Time-Series Database
#123Earlier quoted context omitted.
A solved problem at what scale? A kid with pencil and paper can solve the problem when it's small enough.
I don't have personal knowledge, but my understanding is that timeseries data in the oil industry is insane scale. Think 10s or 100s of wells in a field, sensors at regular distances in every well each taking multiple readings (temperature, flow etc) on a sub-second cadence. And a big oil company will of course have many fields. It's in the same league as timeseries data in financial markets, which I have more direct…
What the article is talking about is an order of magnitude larger. Think millions of data points per minute coming out of a medium sized data center. It gets harder when you look at the cardinality of the data (explained better by the article), which corresponds to the sized of the indexes.
Re: Why Not to Build a Time-Series Database
#124Earlier quoted context omitted.
Do you use the new metrics store? I just learned they have one now in the new versions.
No we don't use metrics store.
That doesn't really sound like a cost effective way of doing metrics... The amount of data you license from splunk is likely quite expensive compared to a dedicated metrics system, right?
Re: Why Not to Build a Time-Series Database
#125Earlier quoted context omitted.
'metric_name text' is actually a tag-value list. Many TSDB's allows you to match data by tag. Each tag should be represented by a column in your example. Single table design will be prone to high read/write amplification due to data alignment. Usually, you need to read many series at once so your query will turn into full table scan. Or it will read a lot of unneeded data which happened to be located near the data yo…
> 'metric_name text' is actually a tag-value list. Many TSDB's allows you to match data by tag. Each tag should be represented by a column in your example. For the life of me, I can't figure out why this would be a good idea. I feel like I must not understand what you're saying: If I've got a million disks that I want to draw usage graphs for, why I would put each one in a separate column? What's the business use-cas…
Modern TSDB is expected to support tags. This means that every series will have a unique set of tag-value pairs associated with it. E.g. 'host=Foo OS=CentOS arch=amd64 ssdmodel=Intel545 ...'. And in the query you could pinpoint relevant series by this tags thus the tags should be searchable. For instance, I may want to see how specific SSD model performs on specific OS or specific app. If the set of tags is stored as a json in one field such queries wouldn't work efficiently.
About that 1M writes thing. You have two options. 1) Organize data by metric name first, or 2) by timestamp. In case of 2) the updates will be linear but reads will have huge amplification. In case of 1) updates will be random, but reads will be fast.
Re: Why Not to Build a Time-Series Database
#126Earlier quoted context omitted.
I appreciate your links to further reading, and I'm trying to read the Aurora paper right now but after reading the abstract and the intro (I'm in progress right now), I can't find a case that is uniquely fit/perfect for data historians... I know this is already asking a lot, but would you mind giving me one go-to- use case that really made you think "this is what purpose-built data historian-style databases are good…
So Aurora isn't a historian, but is a complex event processing system. It's an entirely different beast that solves very specific problems around high-speed queries that could theoretically require scanning through all data stored historically for queries. I'm not a huge fan of historians (I've spent too much of my career working with them), but I can definitely tell you where they make sense. The scenario is this: I…
Thanks for taking the time to give such a detailed answer and explanation on historian systems.
Re: Why Not to Build a Time-Series Database
#127It's pretty ridiculous that "Time-Series Database" has come to mean ingesting massive amounts of streaming data. They've been around a long time and have many use cases. They're a great way to store data efficiently, accessing specific data if you know the time range you are looking for is very fast and simple, and you can roll your own in a few dozen lines of C if that's what you want to do. If that's all you need,…
That may be a perfectly good solution if you have a very static infrastructure and narrow use case. As a thought exercise, for the most trivial solution, you could create a single append only flat file. This may work well for writes, but what happens when you want to read the datapoints for only a single series in time order? This would result in an expensive scan over the whole file. An improvement could be to creat…
To be clear, my post was to state that there are many use cases for time-series databases and bemoan the fact that most current development centers around a specific use case. That is in fact what I wrote.
I have a hard time believing that "a dynamic containerised infrastructure which produce a unique number of timeseries over very short intervals" is the superset of all time-series use cases, but perhaps it is so.
Re: Why Not to Build a Time-Series Database
#128"time-series database" is some of the most overhyped nonsense since noSQL. Time-series is just data with time as a primary component. It comes in all shapes and volume, but if you have a lot of data and are running heavy OLAP queries than we already have an entire class of capable databases. Use any modern distributed relational column-oriented database, set primary key to metric id + timestamp, and you'll be able to…
This works for a small / medium size company but certainly does not scale for bigger companies. There are several problems that you gonna run at scale.
That's true for pretty much any technology.