Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring demystified: A guide for logging, tracing, metrics

techbeacon.com

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Monitoring demystified: A guide for logging, tracing, metrics

#21

Post by viraptor »

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki)

I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

Re: Monitoring demystified: A guide for logging, tracing, metrics

#22

Post by dkersten »

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

I’d add to no. 1 by saying include a correlation id or request id so you have a way to filter the logs into a single linear stream related to the same action.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#23

Post by viraptor »

neolog wrote: post #13
KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming The other side is that I don't know what metrics I'll want until later. When do you think it's better to pull metrics from structured logs vs generating metrics in app?

You can go the https://www.honeycomb.io/ way and make structured logs your metrics. It will cost you a lot in storage, but simplifies a lot. Just throw properly structured logs into storage as long as you query them efficiently (which honeycomb provides)

Re: Monitoring demystified: A guide for logging, tracing, metrics

#24

Post by ysoft »

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

I don't think loki is production ready. Needs more work. It's going to be great with grafana though

Re: Monitoring demystified: A guide for logging, tracing, metrics

#25

Post by ignoramous »

KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> If you are searching through logs to _find_ a problem, its far too late.

To be fair, this is addressed in the article which links to Netflix's blog on the topic and how they do so effectively at their scale: https://netflixtechblog.com/lessons-from-building-observabil...

Re: Monitoring demystified: A guide for logging, tracing, metrics

#26

Post by rudasn »

viraptor wrote: post #21
buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

Twice I fell into the trap of datadog, having paid more than $300 for a single month in each case.

The simplicity of it, dashboards, notebooks, logs etc, is what makes it so appealing though.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#27

Post by count »

rudasn wrote: post #26
viraptor wrote: post #21

Earlier quoted context omitted.

> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

Twice I fell into the trap of datadog, having paid more than $300 for a single month in each case. The simplicity of it, dashboards, notebooks, logs etc, is what makes it so appealing though.

If it saves you more than an hour or two, $300 seems super worth it for a system making you money...

Re: Monitoring demystified: A guide for logging, tracing, metrics

#28

Post by secondcoming »

We log extensively. Here are some of my thoughts it

- at least in C++, the requirement to be able to log from pretty much anywhere can lead to messy code that either passes a reference to your logger to all classes that might possibly need it, or you've got an extern global somewhere. Yuck.

- logging can enable laziness. Being able to log that something weird happened can be considered a sufficient substitute for proper testing.

- logs are only as useful as the info they contain. This can mean state needs to be passed around all over the place just so that it can all be eventually logged on one line (it saves your data team from having to do a 'join')

- if your logger doesn't support cycling log files it's useless. If something goes wrong you can easily fill a disk.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#30

Post by KaiserPro »

neolog wrote: post #13
KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming The other side is that I don't know what metrics I'll want until later. When do you think it's better to pull metrics from structured logs vs generating metrics in app?

Aha! that is the eternal question.

TL;DR:

almost never. structured logs are expensive in terms of infra, management and query time. Storing logs just in case is much more expensive at any kind of scale compared to metrics alone.

Long answer:

A lot of it depends on what the service/program is meant to be doing.

If we take a proxying webs service router for example listening on example.com/* We would want metrics to tell us how well its doing for its specific job, and any upstream services.

So for each service URL we'd want at least a hit count for 2xx, 3xx, specific 4xx and 5xx return codes. We'd also want the time taken to process that request.

We'd also probably want to know the total number of active connection to back end, and total clients connected. Memory and CPU usage would also be a given.

From that we could easily ascertain the health of upstream services, the performance, and total load (which is useful for autoscaling of either the service router, or the upstream apps)

I think it requires sitting down with a peice of paper and imagining your service/app breaking, and then working back to see how that would look. Once you've done that, you can figure out some counters to keep track of those thins.

Post reply on HN