Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring demystified: A guide for logging, tracing, metrics

techbeacon.com

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Monitoring demystified: A guide for logging, tracing, metrics

#81

Post by gdcohen »

KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

Gavin from Zebrium here. We've found that if only you somehow knew what you were monitoring for in logs, they can be a great source of detecting (and then describing) the long tail of unknown/unknowns (failure modes with unknown symptoms and causes). Our approach is to be able to find these patterns in near real-time using ML. This blog by our CTO explains the tech with some good examples: https://www.zebrium.com/blog/is-autonomous-monitoring-the-an....

Re: Monitoring demystified: A guide for logging, tracing, metrics

#82

Post by sagichmal »

twic wrote: post #33
KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming Yeah nah, but, okay, nah yeah. Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app. Unless you're v…

You're optimizing for the wrong thing. The hard part about this space isn't extracting value from data, it's physically shipping the data through the infrastructure and into the relevant systems. Metrics are so great compared to logs precisely because they're precalculated (read: highly compressed) before leaving the originating service.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#83

Post by sagichmal »

secondcoming wrote: post #28

We log extensively. Here are some of my thoughts it - at least in C++, the requirement to be able to log from pretty much anywhere can lead to messy code that either passes a reference to your logger to all classes that might possibly need it, or you've got an extern global somewhere. Yuck. - logging can enable laziness. Being able to log that something weird happened can be considered a sufficient substitute for pro…

> if your logger doesn't support cycling log files it's useless. If something goes wrong you can easily fill a disk.

Few applications should be logging to disk directly. Services running under systemd or any modern orchestration platform should log to stdout/stderr and let the system manage the stream.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#84

Post by Bombthecat »

dkersten wrote: post #22
buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

I’d add to no. 1 by saying include a correlation id or request id so you have a way to filter the logs into a single linear stream related to the same action.

Also adding a context tag can very powerful, especially in times of Microservices and some kind of event driven stuff like monthly payments.

Imagine: user registers, the post gets and id, and context registering. Then he adds a credit card. (a new id, context credit payments) after 14 days the bill goes out, same id, same context.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#85

Post by detaro »

sagichmal wrote: post #82
twic wrote: post #33

Earlier quoted context omitted.

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming Yeah nah, but, okay, nah yeah. Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app. Unless you're v…

You're optimizing for the wrong thing. The hard part about this space isn't extracting value from data, it's physically shipping the data through the infrastructure and into the relevant systems. Metrics are so great compared to logs precisely because they're precalculated (read: highly compressed) before leaving the originating service.

That does sound like you are assuming that everyone always talks about large-scale environments with dozens or more machines?

Re: Monitoring demystified: A guide for logging, tracing, metrics

#86

Post by KaiserPro »

twic wrote: post #33
KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming Yeah nah, but, okay, nah yeah. Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app. Unless you're v…

> Unless you're very careful, it's also easy to end up double-aggregating,

Oh no, never do anything fancy on the client end. yeah thats total trash. Any client that does any kind of aggregating is a massive pain in the arse.

Counters are good enough for 90% of everything you want. You can turn counters into hits per second easily. Plus they are more resistant to time based averaging. If you do your stats correctly, you can even has resetting counters create nice smooth graphs (non negative derivatives are a god send)

> Dropwizard

Yes, this is a library that argues strongly against the use of metrics. From what I recall 1 node of casasndra will output close to 50,000 metrics by default. That is too much.

When a team I worked with were migrating away from splunk to graphite/grafana, they shat out something close to a million metrics. 99.8% were totally useless.

> You need to build and bodge a terrifying pile of stuff to get post-calculated metrics going.

Yes! I think thats my main objection. Its so bloody expensive to do post-hock metrics. you can buy in splunk, but thats horrifically expensive. Or you can use an open source version and loose 4 person years before you even get a graph.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#87

Post by mmkos »

viraptor wrote: post #35
secondcoming wrote: post #28

We log extensively. Here are some of my thoughts it - at least in C++, the requirement to be able to log from pretty much anywhere can lead to messy code that either passes a reference to your logger to all classes that might possibly need it, or you've got an extern global somewhere. Yuck. - logging can enable laziness. Being able to log that something weird happened can be considered a sufficient substitute for pro…

I'd disagree with 2 and 4. 2. Given a large enough system you will encounter situations where the only action you can take is to log "this really shouldn't happen" and try to roll back as cleanly as possible. This may be due to either complexity or a bug manifesting in a layer completely different than where it occurred (I've seen a null reference crash on "if(foo) foo->bar();" in the past) 4. I believe loggers shoul…

> I've seen a null reference crash on "if(foo) foo->bar();" in the past

References can't be null. Regardless, that's a valid check for a null pointer and I don't think what you wrote is at all possible (unless maybe in some multithreaded scenario?).

Re: Monitoring demystified: A guide for logging, tracing, metrics

#88

Post by twic »

therealdrag0 wrote: post #75
adamzegelin wrote: post #69

Earlier quoted context omitted.

> if you're using the Dropwizard Metrics library, for example, you've already lost. Can you go into a bit more detail here? Curious to know where Dropwizzard goes wrong. I prefer to use the Prometheus client libraries where possible. Prometheus' data model is "richer" -- metric families and labels, rather than just named metrics. Adapting from Dropwizzard to Prometheus is a pain, and never results in the data feeling…

I think they just mean the host is aggregating, so any further aggregation is compounded slant time the data. Like StatsD’s default is shipping metrics every 10s, so if you graph it and your graph rolls up those data points into 10 minute data points (cuz you’re viewing a week at once), then you’re averaging an average. Or averaging a p95. People often miss that this is happening, and it can drastically change the na…

Yes, exactly this. It's the fact that you're doing aggregation in two places. Since you're always going to be aggregating on the backend, aggregating in the app is bad news.

It may be interesting to think about the class of aggregate metrics that you can safely aggregate. Totals can be summed. Counts can be summed. Maxima can be maxed. Minima can be minned. Histograms can be summed (but histograms are lossy). A pair of aggregatable metrics can be aggregated pairwise; a pair of a total and a count lets you find an average.

Medians and quantiles, though, can't be combined, and those are what we want most of the time.

Someone who loves functional programming can tell us if metrics in this class are monoids or what.

There is an unjustly obscure beast called a t-digest which is a bit like an adaptive histogram; it provides a way to aggregate numbers such that you can extract medians and quantiles, and the aggregates can be combined:

https://github.com/tdunning/t-digest

Re: Monitoring demystified: A guide for logging, tracing, metrics

#89

Post by sagichmal »

detaro wrote: post #85
sagichmal wrote: post #82

Earlier quoted context omitted.

You're optimizing for the wrong thing. The hard part about this space isn't extracting value from data, it's physically shipping the data through the infrastructure and into the relevant systems. Metrics are so great compared to logs precisely because they're precalculated (read: highly compressed) before leaving the originating service.

That does sound like you are assuming that everyone always talks about large-scale environments with dozens or more machines?

Everything we're talking about here is trivial until you reach a minimum scale. "Dozens of machines" is still trivial, to be honest.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#90

Post by detaro »

sagichmal wrote: post #89
detaro wrote: post #85

Earlier quoted context omitted.

That does sound like you are assuming that everyone always talks about large-scale environments with dozens or more machines?

Everything we're talking about here is trivial until you reach a minimum scale. "Dozens of machines" is still trivial, to be honest.

Not for many people running such environments. So you either see with very minimal setups without tooling to help (which is survivable at small scales, but inefficient and mindnumbing), or towers of complexity that followed widely shared advice by people always assuming massive scale.
Post reply on HN