Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prometheus 2.0

prometheus.io

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Prometheus 2.0

#31

Post by jrv »

chimeracoder wrote: post #30
bbrazil wrote: post #25

Earlier quoted context omitted.

> I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems. And from the link: > Logging can be useful for some purposes. However, it’s rare that they’re the only tool for monitoring your code. And it’s even rarer that they’re the best tool. Metrics are a tool that take a different approach to logs, once you get beyond small system…

> Metrics are a tool that take a different approach to logs, once you get beyond small systems you need both. I talked about this earlier in the year: Quite the opposite - the "some purposes" I'm talking about are precisely the small scale. As scale grows, the use case of logs and metrics converges, and metrics become a strictly better tool.

The question is about tracking and storing individual events (logs) with arbitrary per-item detail vs. dimensionally limited aggregations (time series / metrics). In either case, I think we agree that the data should be recorded in a structured way, and when I say "logs" I just mean a record of individual items, not of sampled/aggregated metrics.

Given that, you need both logs (individual events) and metrics. Logs give you crucial insight into individual interesting events such as single requests that bring your service down, but logs are orders of magnitude more expensive than metrics in tracking, storage, and processing. So that's why you use metrics for a much wider scope and for longer time periods.

Re: Prometheus 2.0

#32

Post by SuperQue »

will-not-smith wrote: post #3
SkyRocknRoll wrote: post #2

Does this storage engine onpar with influxdb ?

Nope, not on par. While there is no direct comparison to InfluxDB in the article, look at the disk usage and (more importantly) disk I/O utilization, which is indistinguishable from zero on the graph, whereas before it was at 20-30% with the same load. So not on par. Much, much better. (Of course, Prometheus and InfluxDB are not perfect substitutes for one another, so there's much more to look at than just storage en…

We just upgraded one of GitLab's Prometheus servers (we have HA redundant servers).

We saw a better than 10x reduction in CPU use.

https://imgur.com/3aJtYu9

This server doesn't take any query load, but it does take the same ingestion load as the others.

(The long down time in the middle is waiting for rm -rf on the old data dir.)

Re: Prometheus 2.0

#33

Post by scotch_drinker »

chimeracoder wrote: post #30
bbrazil wrote: post #25

Earlier quoted context omitted.

> I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems. And from the link: > Logging can be useful for some purposes. However, it’s rare that they’re the only tool for monitoring your code. And it’s even rarer that they’re the best tool. Metrics are a tool that take a different approach to logs, once you get beyond small system…

> Metrics are a tool that take a different approach to logs, once you get beyond small systems you need both. I talked about this earlier in the year: Quite the opposite - the "some purposes" I'm talking about are precisely the small scale. As scale grows, the use case of logs and metrics converges, and metrics become a strictly better tool.

Not for high cardinality events like what happened to a particular user during a single session. Metrics will never help with that type of problem.

Re: Prometheus 2.0

#34

Post by nopzor »

Really psyched to see this drop, the new engines performance looks amazing. And performance was already pretty stellar to start with!

Looks like Prometheus is also now the most popular tsdb on github!

The grafana team is working quite heavily on our Prometheus integrations, with some major recent improvements to the query editor in 4.6. We have many more plans to work even more tightly with the project.

Kudos to the Prometheus team!

Re: Prometheus 2.0

#35

Post by chimeracoder »

scotch_drinker wrote: post #33
chimeracoder wrote: post #30

Earlier quoted context omitted.

> Metrics are a tool that take a different approach to logs, once you get beyond small systems you need both. I talked about this earlier in the year: Quite the opposite - the "some purposes" I'm talking about are precisely the small scale. As scale grows, the use case of logs and metrics converges, and metrics become a strictly better tool.

Not for high cardinality events like what happened to a particular user during a single session. Metrics will never help with that type of problem.

> Not for high cardinality events like what happened to a particular user during a single session. Metrics will never help with that type of problem.

No, and as I explain in both that article and the video, logs aren't the best solution for that use case either.

Re: Prometheus 2.0

#36

Post by pauldix »

SkyRocknRoll wrote: post #2

Does this storage engine onpar with influxdb ?

InfluxDB creator here. I'd say it's on par and in some cases a bit better. They can do some optimizations that we're unable to do. We have to worry about writing historical data rather than just append only workloads. We also have to support more data types than float64.

The design of the new Prometheus engine and InfluxDB's Time Structured Merge Tree engine are very similar. They're both heavily inspired by LSM Trees.

Our goal is to add more support for Prometheus standards like the metrics format and even PromQL over time. We recently added remote read/write support, which will be in the 1.4 release next week. https://www.influxdata.com/blog/influxdb-now-supports-promet...

I'd like to see Influx become something that is complementary to Prometheus or at least solidly in the Prometheus ecosystem, like players like Weaveworks, etc.

Re: Prometheus 2.0

#37

Post by pauldix »

SEJeff wrote: post #15
SkyRocknRoll wrote: post #2

Does this storage engine onpar with influxdb ?

FWIW, when I met Paul Dix a few months ago, he had nothing but good to say about the new Prometheus. They released this blog post shortly after our meeting: https://www.influxdata.com/blog/influxdb-now-supports-promet... I wouldn't see them as competitors so much as complimentary. Influx is really going for the long term tsdb storage, whereas prometheus is more shorter term with higher resolutions and alerting.

Yup, I'm definitely a fan of their work. The new engine is a great design and all the work on the metrics standard is great. Having that as a standard format for service implementors to expose performance metrics is invaluable and we're going to support that as much as we can.

Re: Prometheus 2.0

#38

Post by [deleted] »

jrv wrote: post #27
bbrazil wrote: post #23

Earlier quoted context omitted.

Prometheus is not intended as durable long term storage, it's fundamentally limited to the size of a machine. You should also design your monitoring be able to tolerate completely losing the data of a Prometheus. We recommend using another system for long term data, see https://prometheus.io/docs/operating/integrations/#remote-en... for some examples.

The problem is (as you know) that single machines are in practice still too reliable and Prometheus is still too good at storing data for long times that many people have come to rely on it despite warnings :)

[deleted]

Re: Prometheus 2.0

#39

Post by pfranz »

Literally yesterday I did an install of 1.8.2 after seeing 2.x still in beta. Time to upgrade.

Congrats to the Prometheus team.

Re: Prometheus 2.0

#40

Post by wanghq »

Does anyone need to view metrics older than 2 weeks? What's your solution? I feel it's odd if you rely on a different software, e.g. for the most recent data, use Prometheus, and for older data, use something else. What if I want to compare the same metric in the last 4 weeks?
Post reply on HN