Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time series database Graphite seems to be falling into disfavor

vividcortex.com

Post reply on HN
Print view View original

21–30 of 83 posts

Re: Time series database Graphite seems to be falling into disfavor

#21

Post by gtrubetskoy »

cbsmith wrote: post #17
gtrubetskoy wrote: post #4

I've been in search of a good time-series database solution for some time now, and have pretty much given up on it and am in the process of rolling my own: https://github.com/grisha/timeriver My issues with the present state of TS isn't the volume. I was looking for using TS outside of the DevOps world. Everyday things like your heart rate over time, price of gas at the nearest station, number of people in line at yo…

I highly recommend looking at Keogh's work, particularly the iSAX and iSAX2 stuff: http://www.cs.ucr.edu/~eamonn/SAX.htm While it is used for machine learning, it actually makes a lot of sense to follow a similar approach for more general time series applications.

I'm familiar with SAX, it's remarkably powerful at discerning patterns and identifying similarity between series, good stuff, thanks for the link.

Re: Time series database Graphite seems to be falling into disfavor

#22

Post by dang »

jsudhams wrote: post #9

Can some one change heading so it is understood as software and not actual Graphite

Ok, we changed the title to use representative language from the article.

Btw, this article was heavily flagged. It's not really legit to flag a story just because people don't like the title. Plenty of good stories have problematic titles. Depriving others of a chance to read the content, especially when there's a good discussion going on in the thread, is a bad use of flagging power.

Re: Time series database Graphite seems to be falling into disfavor

#23

Post by meowface »

Karunamon wrote: post #13
lumpypua wrote: post #12

What does a modern metrics stack look like? There are so many words... Graphfana, Kibana, StatsD, Graphite, etc. Which bits should I choose and attach together?

I've got to advise against Kibana/Elasticsearch until the company gets more mature. I was 100% on board with them up until the redesign. Not so much for the design (oh look it's white instead of black, who cares), but the way they've handled it subsequently. The 3.x -> 4.x transition for Kibana left a product that was missing really basic features (like, the ability to set graph colors for one - a bug/feature request…

After many years of using Splunk at my job, switching to ELK for personal project use was quite a disappointment for me. Of course, ELK is free and Splunk is very expensive, but I was still surprised at the gap.

This might be derailing the thread a bit, but is there any log management platform like ELK or Splunk that has an expressive and versatile query language like Splunk's? My biggest issue with ELK is that analytics is mostly expected to be done through Kibana's GUI, while with Splunk you can craft terse queries to do almost any sort of transformation and visualization imaginable. I don't like how ELK is so GUI-oriented.

Re: Time series database Graphite seems to be falling into disfavor

#24

Post by frik »

Karunamon wrote: post #13
lumpypua wrote: post #12

What does a modern metrics stack look like? There are so many words... Graphfana, Kibana, StatsD, Graphite, etc. Which bits should I choose and attach together?

I've got to advise against Kibana/Elasticsearch until the company gets more mature. I was 100% on board with them up until the redesign. Not so much for the design (oh look it's white instead of black, who cares), but the way they've handled it subsequently. The 3.x -> 4.x transition for Kibana left a product that was missing really basic features (like, the ability to set graph colors for one - a bug/feature request…

Kibana is in incredible heavy weight in comparision to other dashboards like e.g. Grafana. Kibana loads like 4MB+ of JS and data. It is designed for intranet usage where your Elastic Search cluster is near you.

Re: Time series database Graphite seems to be falling into disfavor

#25

Post by dalys »

gtrubetskoy wrote: post #4

I've been in search of a good time-series database solution for some time now, and have pretty much given up on it and am in the process of rolling my own: https://github.com/grisha/timeriver My issues with the present state of TS isn't the volume. I was looking for using TS outside of the DevOps world. Everyday things like your heart rate over time, price of gas at the nearest station, number of people in line at yo…

Thanks for your thoughts Gregory!

We're also in the process now of finding a good storage solution for time series data at my dayjob. We're not storing any server metrics, but more personal user health data and related metrics. So we want the flexibility of being able to write new metrics on your personal timeline without altering a schema. Most reads would be querying single users timeline to fetch their data and deliver through APIs for rendering in clients. Also of course to do analysis on all users timelines, find correlations etc. but those queries are less frequent and not so time critical. I'm starting working on a prototype now with MongoDB. Other DBs that have come up has been InfluxDB, Riak TS, Amazon DynamoDB and probably some others I don't remember. Haven't actually thought that much about Postgres, but thanks for your links to your blog posts I will read up on how Postgres might work. Most other non-time series data would still be in our MySQL setup.

Re: Time series database Graphite seems to be falling into disfavor

#26

Post by obfuscurity_ »

cbsmith wrote: post #18

Live by the sword, die by the sword. It is kind of amazing how hard it is for existing time series database systems to track the changing needs of the marketplace.

You mean for open source time series database systems run entirely by volunteers in their spare (personal) time? I know, right? What jerks.

Re: Time series database Graphite seems to be falling into disfavor

#27

Post by glial »

meowface wrote: post #23
Karunamon wrote: post #13

Earlier quoted context omitted.

I've got to advise against Kibana/Elasticsearch until the company gets more mature. I was 100% on board with them up until the redesign. Not so much for the design (oh look it's white instead of black, who cares), but the way they've handled it subsequently. The 3.x -> 4.x transition for Kibana left a product that was missing really basic features (like, the ability to set graph colors for one - a bug/feature request…

After many years of using Splunk at my job, switching to ELK for personal project use was quite a disappointment for me. Of course, ELK is free and Splunk is very expensive, but I was still surprised at the gap. This might be derailing the thread a bit, but is there any log management platform like ELK or Splunk that has an expressive and versatile query language like Splunk's? My biggest issue with ELK is that analy…

Not sure what kind of queries you want to do, but Elasticsearch has a fairly extensive Query DSL that'll let you do all sorts of aggregations:

https://www.elastic.co/guide/en/elasticsearch/reference/curr...

There is a Python implementation that makes creating complex queries pretty easy:

https://github.com/elastic/elasticsearch-dsl-py

I agree with the criticisms of Kibana, but I have had no problems querying Elasticsearch directly. It also supports scripted queries if the built-in aggregations aren't enough.

Of course, then you have to build your own visualizations with the results...

Re: Time series database Graphite seems to be falling into disfavor

#28

Post by discordianfish »

late2part wrote: post #8
discordianfish wrote: post #5

It looks like they (deliberately?) didn't mention the open source projects that were specifically build to address the shortcomings of graphite, like prometheus or opentsdb.

The author most likely didn't mention them because they suffer from the same limiting assumptions that graphite does. When you're dealing with multidimensional changing metrics, opentsdb has the same issues graphite does.

Graphite's data model doesn't support multidimensional metrics which is in my opinion it's biggest shortcoming. To support multidimensional data, a suitable data model is needed. AFAIK influxdb stores all label dimensions and the value for each data point. No matter how high your cardinality is, the storage requirement is the same. If you need to have, let say, a dimension 'client_ip' in a metrics http_response_time, that's a reasonable model. I'd argue though that outside analytics/data warehousing such high cardinality is rarely needed. That's why prometheus (can't speak for opentsdb) stores the dimensions identifying a metric (metric key/label pairs) once. For all data points it just needs to store the value which makes reading and writing less expensive. Now however VividCortex stores it's metrics, it has to choose from similar trade offs. Sure, they host it for you - still, there are open source project out there addressing exactly those issues.

Re: Time series database Graphite seems to be falling into disfavor

#29

Post by realharo »

Karunamon wrote: post #13
lumpypua wrote: post #12

What does a modern metrics stack look like? There are so many words... Graphfana, Kibana, StatsD, Graphite, etc. Which bits should I choose and attach together?

I've got to advise against Kibana/Elasticsearch until the company gets more mature. I was 100% on board with them up until the redesign. Not so much for the design (oh look it's white instead of black, who cares), but the way they've handled it subsequently. The 3.x -> 4.x transition for Kibana left a product that was missing really basic features (like, the ability to set graph colors for one - a bug/feature request…

There are also some serious unresolved issues in the current versions of the ELK stack bits, such as https://github.com/elastic/kibana/issues/5170, https://github.com/elastic/kibana/issues/1961, https://github.com/elastic/logstash/issues/3440. Even with all that though, it's still an incredibly useful tool for log analysis. It's a much different scenario than what one would use statsd, graphite, etc. for.

Re: Time series database Graphite seems to be falling into disfavor

#30

Post by fapjacks »

dmourati wrote: post #15

Obligatory response from Graphite contributor and author Jason Dixon who is shouted out in the vividcortex intro. http://obfuscurity.com/2015/11/Everybody-Loves-Graphite Personal response: I've used Graphite, OpenTSDB, Ganglia, Cacti, and a bunch more solutions. Recently, work transitioned from OpenTSDB to a hosted solution from a startup called Wavefront. https://www.wavefront.com/ This has been a smash hit. Scale m…

Wavefront looks like it's probably closed source. Is this true?
Post reply on HN