Time series database Graphite seems to be falling into disfavor • Hacker News Board

Re: Time series database Graphite seems to be falling into disfavor

#71

Post by michaelcampbell » Mon, Nov 09, 2015, 3:40 PM UTC

meowface wrote: post #33

glial wrote: post #27

Earlier quoted context omitted.

Not sure what kind of queries you want to do, but Elasticsearch has a fairly extensive Query DSL that'll let you do all sorts of aggregations: https://www.elastic.co/guide/en/elasticsearch/reference/curr... There is a Python implementation that makes creating complex queries pretty easy: https://github.com/elastic/elasticsearch-dsl-py I agree with the criticisms of Kibana, but I have had no problems querying Elastics…

I'm not sure I'd consider that a DSL. It's unwieldy to write a full JSON object for every one-off query I want to do. Using one of their examples, this: { "query": { "filtered": { "query": { "bool": { "must": [{"match": {"title": "python"}}], "must_not": [{"match": {"description": "beta"}}] } }, "filter": {"term": {"category": "search"}} } }, "aggs" : { "per_tag": { "terms": {"field": "tags"}, "aggs": { "max_lines":…

https://github.com/NLPchina/elasticsearch-sql/

does exactly that with a reasonably good subset of SQL and ES query language mixed in.

It operates in 2 modes; in one it runs the query, in another it spits back out what the equivalent ES JSON query is. We use this as a quick prototyping tool and modify the ES query as needed, as most of us here still "think" in SQL for a lot of things.

Re: Time series database Graphite seems to be falling into disfavor

#72

Post by cbsmith » Mon, Nov 09, 2015, 4:00 PM UTC

obfuscurity_ wrote: post #26

cbsmith wrote: post #18

Live by the sword, die by the sword. It is kind of amazing how hard it is for existing time series database systems to track the changing needs of the marketplace.

You mean for open source time series database systems run entirely by volunteers in their spare (personal) time? I know, right? What jerks.

What? I'm not saying they are jerks. I'm pointing out that it is harder for existing projects to track the changes in the market place. It is hard for a new open source project to launch and build up enough of a community to be viable.

Re: Time series database Graphite seems to be falling into disfavor

#73

Post by cbsmith » Mon, Nov 09, 2015, 4:04 PM UTC

chaotic-good wrote: post #42

cbsmith wrote: post #17

Earlier quoted context omitted.

I highly recommend looking at Keogh's work, particularly the iSAX and iSAX2 stuff: http://www.cs.ucr.edu/~eamonn/SAX.htm While it is used for machine learning, it actually makes a lot of sense to follow a similar approach for more general time series applications.

SAX requires different type of storage. You can store SAXified time series in ElasticSearch, or Solr but time-series database doesn't fit for this. Time-series databases should be able to generate SAX representation.

> You can store SAXified time series in ElasticSearch, or Solr but time-series database doesn't fit for this.

That's making some presumptions about the time-series database use case. SAX is convenient for storing and retrieving the data as well as identifying trends or recurring behaviour. What more do you need?

Re: Time series database Graphite seems to be falling into disfavor

#74

Post by chaotic-good » Mon, Nov 09, 2015, 5:38 PM UTC

cbsmith wrote: post #73

chaotic-good wrote: post #42

Earlier quoted context omitted.

SAX requires different type of storage. You can store SAXified time series in ElasticSearch, or Solr but time-series database doesn't fit for this. Time-series databases should be able to generate SAX representation.

> You can store SAXified time series in ElasticSearch, or Solr but time-series database doesn't fit for this. That's making some presumptions about the time-series database use case. SAX is convenient for storing and retrieving the data as well as identifying trends or recurring behaviour. What more do you need?

You can't retrieve original time-series data from SAX storage because of normalization. To query time-series data by content (approximate 1-NN, motif discovery, etc) you need inverted index.

Re: Time series database Graphite seems to be falling into disfavor

#75

Post by jabl » Mon, Nov 09, 2015, 6:03 PM UTC

devdas wrote: post #50

IIAOPSW wrote: post #3

Here I was thinking I would read some informative article on graphite, graphene and related carbon derivatives. Dear programmers: pick better names.

Naming is a hard problem in computer science :)

Like they say, the two fundamentally difficult problems in CS are cache invalidation, naming things, and off by one errors.

Re: Time series database Graphite seems to be falling into disfavor

#76

Post by dmourati » Mon, Nov 09, 2015, 6:06 PM UTC

duaneb wrote: post #46

dmourati wrote: post #15

Obligatory response from Graphite contributor and author Jason Dixon who is shouted out in the vividcortex intro. http://obfuscurity.com/2015/11/Everybody-Loves-Graphite Personal response: I've used Graphite, OpenTSDB, Ganglia, Cacti, and a bunch more solutions. Recently, work transitioned from OpenTSDB to a hosted solution from a startup called Wavefront. https://www.wavefront.com/ This has been a smash hit. Scale m…

A closed silo seems like a distinct step backwards compared to a stack you can maintain yourself.

Read a little more into the second part of your sentence, especially the word "maintain."

Re: Time series database Graphite seems to be falling into disfavor

#77

Post by discordianfish » Mon, Nov 09, 2015, 6:45 PM UTC

jrv wrote: post #54

discordianfish wrote: post #28

Earlier quoted context omitted.

Graphite's data model doesn't support multidimensional metrics which is in my opinion it's biggest shortcoming. To support multidimensional data, a suitable data model is needed. AFAIK influxdb stores all label dimensions and the value for each data point. No matter how high your cardinality is, the storage requirement is the same. If you need to have, let say, a dimension 'client_ip' in a metrics http_response_time,…

Though I'm a Prometheus author, I need to defend InfluxDB here :) Since 0.9.0, they support tags (vs. fields), which store indexed dimensions similarly to how Prometheus does it. So this argument doesn't count anymore. Still, there are many other differences in scope and functionality between the two systems.

So influxdb is now ready to be used as prometheus long time storage?

Re: Time series database Graphite seems to be falling into disfavor

#78

Post by emilburzo » Tue, Nov 10, 2015, 3:44 PM UTC

gh02t wrote: post #67

svckr wrote: post #41

Earlier quoted context omitted.

I find it confusing that these solutions seem to conflate time series storage/querying with visualization. Is there a reason for this? I recall that Kibana is a plugin for Elastic Search but don't know about the others. Isn't it possible to connect a hypothetical standalone visualization frontend to any TSDB?

Grafana is sort of that. It's a visualization tool that can use a bunch of different TSDBs as the backend. I use it with InfluxDB to track and plot sensor data from my home IoT setup. Basically all sensors (primarily temp because it controls my thermostat) are connected to an isolated WLAN and broadcast do their readings via MQTT. A process on the server collects them and stores them in Influx, which Grafana can use…

> sensor data from my home IoT setup

Have you documented your setup anywhere?

I'm DIY-ing something similar and I'm always curious how others have approached things.

Re: Time series database Graphite seems to be falling into disfavor

#79

Post by cbsmith » Tue, Nov 10, 2015, 4:56 PM UTC

chaotic-good wrote: post #74

cbsmith wrote: post #73

Earlier quoted context omitted.

> You can store SAXified time series in ElasticSearch, or Solr but time-series database doesn't fit for this. That's making some presumptions about the time-series database use case. SAX is convenient for storing and retrieving the data as well as identifying trends or recurring behaviour. What more do you need?

You can't retrieve original time-series data from SAX storage because of normalization. To query time-series data by content (approximate 1-NN, motif discovery, etc) you need inverted index.

The normalized data is the index. It can still be pointing to the raw data. The nice thing is that the index organizes the data in a way that makes it easily (lossless) compressible.

Re: Time series database Graphite seems to be falling into disfavor

#80

Post by chaotic-good » Wed, Nov 11, 2015, 10:35 AM UTC

cbsmith wrote: post #79

chaotic-good wrote: post #74

Earlier quoted context omitted.

You can't retrieve original time-series data from SAX storage because of normalization. To query time-series data by content (approximate 1-NN, motif discovery, etc) you need inverted index.

The normalized data is the index. It can still be pointing to the raw data. The nice thing is that the index organizes the data in a way that makes it easily (lossless) compressible.

> It can still be pointing to the raw data.

Yep. Each SAX word should be mapped to the list of seriesid:timestamp pairs. This list is often referred as postings list in information retrieval. The resulting data-structure is an inverted index. SAX and iSAX papers describes inverted index variant (really bad one) based on folder structure but one can use convenient IR tools for this.