51–60 of 132 posts
I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.
I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.
Do you use a database for your data? We store our flight test data as individual DATAQ files with a naming scheme. This is manageable for our very simple civil aircraft so far.
EDIT: I would like to know how other flight test people visualize their test data. Seems like you can only make so many envelope charts and strip charts (they work and are effective, but man, sometimes I'd like something different)
Prometheus uses a file per timeseries, with two levels of delta encoding to keep the data small. This is our second major storage iteration, and seems to be doing pretty well with a single server able to handle over 2M timeseries. See http://prometheus.io/docs/introduction/faq/#why-does-prometh...
You might think this could be better solved by a simple write-ahead-log (WAL), but the problem is that samples can come in at completely irregular intervals, and we still need to append data from the WAL to the appropriate time series files once chunks for a specific series are full. That would create frequent holes of already persisted data in the WAL, and it would have to be regularly compacted/rewritten. Which is in principle equivalent to the regular checkpointing approach again.
EDIT: There's some more information about Prometheus's storage here: http://prometheus.io/docs/operating/storage/ - but we really should write a white paper or blog post about it soon.
Prometheus uses a file per timeseries, with two levels of delta encoding to keep the data small. This is our second major storage iteration, and seems to be doing pretty well with a single server able to handle over 2M timeseries. See http://prometheus.io/docs/introduction/faq/#why-does-prometh...
The subject of Time Series has lately been on my mind as well, see my blog posts on accuracy of Graphite vs RRD, as well as InfluxDB storage: http://grisha.org/ I am leaning towards none of the above being the best solution and am in the process of writing my own (too early to announce yet).
Like I mentioned here [0], whisper isn't the only storage option for graphite. Another user [1] mentioned blueflood. Have you evaluated any of these cassandra-based options? [0] https://news.ycombinator.com/item?id=9808035 [1] https://news.ycombinator.com/item?id=9808662
Prometheus uses a file per timeseries, with two levels of delta encoding to keep the data small. This is our second major storage iteration, and seems to be doing pretty well with a single server able to handle over 2M timeseries. See http://prometheus.io/docs/introduction/faq/#why-does-prometh...
I don't understand why Prometheus uses leveldb to begin with though, why do you need it vs using a const offset of an interval? From reading it seems it's used for an index, I'm guessing because intervals can be variable?
As a simple example, we have one LevelDB index which has single label=value pairs as the keys and as the LevelDB value, the identifiers of the time series which have those label=value dimensions. If you now query for e.g. all time series with labels foo="biz" AND bar="baz", we will do two lookups in that index: one for the key foo="biz", and one for bar="baz". We will now have two sets of time series identifiers which we intersect (AND-style matching) to arrive at the set of time series you're interested in querying. Only then do we actually start loading any actual time series data (not from LevelDB this time).
I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…
I've hit the same problem and I would like to move back to a SQL data store. However none of the nice dashboards / visualizations support postgres or any SQL database (for now)... My question (to everyone): what do you use as replacement for kibana or grafana?
As dashboards for Prometheus, you can currently use PromDash (http://prometheus.io/docs/visualization/promdash/), Console HTML templates (http://prometheus.io/docs/visualization/consoles/), or Grafana (http://prometheus.io/docs/visualization/grafana/).
Durable long-term storage is still outstanding. Although replication into OpenTSDB and InfluxDB is experimentally there.
Why does every TSD seem so overly engineered and for all the wrong reasons? Why not just use a time decaying ring buffer (multiple buffers could be used), one statistic one file (or more depending on the decay) and offset by a set interval if you have irregular intervals 'smooth' it to fit O(1) for most things. My other issue is (from a glance) looking at some TSD they ignore most research done on how to effectively…
But if you'd actually RTFA, you'd know that Whisper is hardly the end-all of TSD, that cache to batch is not so trivial as you make it seem.
I do believe there's sort of optimal solution that isn't very complex and not far away from many solutions out there. But most definitely more complex than the TSD's in existance today.
You say over-engineered, but if you'd take a look at what's out there, they're all super simple. Usually being a simple layer over some pre-existing storage engine.
I'd be surprised if in any of the currently popular TSD's there more than a month or two of fulltime work put in just the storage engine. Even though there are so many, no one has time to make them as optimized as say popular SQL databases are.
I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…
Earlier quoted context omitted.
> 20 million rows > 30 seconds Ugh, that's terrible. grepping the file or reading and parsing a csv file is probably faster.
It was pretty awful watching the vps choke to death when I tried implementing any feature using full set prices. Pegged at 99% cpu usage with the go garbage collector frantically trying to not let the process crash... that was not an environment I wanted to take to production.
There's your problem. If you're at the "make it fast" part of "make it work, make it right, make it fast", you should almost certainly be on dedicated hardware.