21–30 of 43 posts
No good migration story for existing data, that'll hurt us quite a lot. :(
If you need your Prometheus data to survive, you're doing it wrong.
How Prometheus compares to Splunk?
Splunk is a event logging system, compared to Prometheus which is metrics based. You need both types of systems to be able to properly observe your systems, they're complementary.
While this is the traditional way of looking at them, I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems.
I've written and spoken about this approach before: https://medium.com/@chimeracoder/dont-read-your-logs-13586c7... and https://vimeo.com/221049715
Earlier quoted context omitted.
If you need your Prometheus data to survive, you're doing it wrong.
What are we doing wrong? We use Prometheus for many things, for example I'd like to know how specific latencies have changed over time. Why should I store these numbers somewhere else?
We recommend using another system for long term data, see https://prometheus.io/docs/operating/integrations/#remote-en... for some examples.
What kind of data? I have no clue. How? Not the faintest.
Anyway, you might want to add a portion to describe what it is for us scrubs that haven't a clue. Possibly a demo page?
Earlier quoted context omitted.
Splunk is a event logging system, compared to Prometheus which is metrics based. You need both types of systems to be able to properly observe your systems, they're complementary.
> Splunk is a event logging system, compared to Prometheus which is metrics based. You need both types of systems to be able to properly observe your systems, they're complementary. While this is the traditional way of looking at them, I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems. I've written and spoken about this appr…
And from the link:
> Logging can be useful for some purposes. However, it’s rare that they’re the only tool for monitoring your code. And it’s even rarer that they’re the best tool.
Metrics are a tool that take a different approach to logs, once you get beyond small systems you need both. I talked about this earlier in the year: https://www.youtube.com/watch?v=hCBGyLRJ1qo
I was curious what this is, so I clicked on it. I spent around 30 seconds scanning the homepage and this is my observation: Uses data to make insights. What kind of data? I have no clue. How? Not the faintest. Anyway, you might want to add a portion to describe what it is for us scrubs that haven't a clue. Possibly a demo page?
A little vague but I'm pretty sure I know what these guys do based on that.
Earlier quoted context omitted.
What are we doing wrong? We use Prometheus for many things, for example I'd like to know how specific latencies have changed over time. Why should I store these numbers somewhere else?
Prometheus is not intended as durable long term storage, it's fundamentally limited to the size of a machine. You should also design your monitoring be able to tolerate completely losing the data of a Prometheus. We recommend using another system for long term data, see https://prometheus.io/docs/operating/integrations/#remote-en... for some examples.
I was curious what this is, so I clicked on it. I spent around 30 seconds scanning the homepage and this is my observation: Uses data to make insights. What kind of data? I have no clue. How? Not the faintest. Anyway, you might want to add a portion to describe what it is for us scrubs that haven't a clue. Possibly a demo page?
Earlier quoted context omitted.
> Splunk is a event logging system, compared to Prometheus which is metrics based. You need both types of systems to be able to properly observe your systems, they're complementary. While this is the traditional way of looking at them, I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems. I've written and spoken about this appr…
> I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems. And from the link: > Logging can be useful for some purposes. However, it’s rare that they’re the only tool for monitoring your code. And it’s even rarer that they’re the best tool. Metrics are a tool that take a different approach to logs, once you get beyond small system…
[edit]
So I watched a few minutes into this, Brian, and it seems to me that either an expert system, or some form of rudimentary AI, that observes the monitoring system can be the driver of an intelligent alerting system. In other words, it seems 'alerts' are, in the final analysis, the higher value proposition.
And I fully agree with you: it really is a waste of talent to have engineers glued to screens watching graphs.
Earlier quoted context omitted.
> Splunk is a event logging system, compared to Prometheus which is metrics based. You need both types of systems to be able to properly observe your systems, they're complementary. While this is the traditional way of looking at them, I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems. I've written and spoken about this appr…
> I strongly disagree that metrics and logs are different toolsets, or that you would need both of them in order to properly observe your systems. And from the link: > Logging can be useful for some purposes. However, it’s rare that they’re the only tool for monitoring your code. And it’s even rarer that they’re the best tool. Metrics are a tool that take a different approach to logs, once you get beyond small system…
Quite the opposite - the "some purposes" I'm talking about are precisely the small scale. As scale grows, the use case of logs and metrics converges, and metrics become a strictly better tool.