Monitoring demystified: A guide for logging, tracing, metrics • Hacker News Board

Re: Monitoring demystified: A guide for logging, tracing, metrics

#61

Post by sciurus » Fri, Jul 31, 2020, 5:17 PM UTC

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

I agree with 2. I have a presentation at https://www.polibyte.com/2019/02/10/pytennessee-presentation... which goes into how to get started with Prometheus. Not as in "how to set it up", but more about what to instrument and why, how to name things, etc. Despite the title, there's very little in it that's specific to Python.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#62

Post by aloknnikhil » Fri, Jul 31, 2020, 5:29 PM UTC

kasey_junk wrote: post #43

aloknnikhil wrote: post #41

Earlier quoted context omitted.

No. The original Kafka paper does talk about logs in the observability sense as a premise to solve the aggregation problem. https://cs.uwaterloo.ca/~ssalihog/courses/papers/netdb11-fin... > There is a large amount of “log” data generated at any sizable internet company. This data typically includes (1) user activity events corresponding to logins, pageviews, clicks, “likes”, sharing, comments, and search queries; (2)…

A quote from the LinkedIn blog post linked in the article: “But before we get too far let me clarify something that is a bit confusing. Every programmer is familiar with another definition of logging—the unstructured error messages or trace info an application might write out to a local file using syslog or log4j. For clarity I will call this "application logging". The application log is a degenerative form of the lo…

Fair enough. But I don't think quoting this for logging in the tracing sense is wrong here. He does acknowledge that trace logs are a degenerative form of logs from the perspective of log processing. The only difference being in the semantics of human readable text v/s binary logs.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#63

Post by _i4zn » Fri, Jul 31, 2020, 5:46 PM UTC

bostonsre wrote: post #49

_i4zn wrote: post #46

Earlier quoted context omitted.

> None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you). I was thinking "God, this is exactly why I hate Datadog" as I was reading your description and got a great laugh when I reached the end. Their billing is absolutely byzantine. I don't know that I've ever seen a company that had such a stark difference between gr…

I'm a customer that uses most of their tools (no network performance monitoring since it's less useful than a service mesh and no logging because we need longer history than most and cost would be prohibitive). Is it really that expensive when compared to other vendors? Thought their newer logging tool was a lot cheaper than splunk and their apm tool for distributed tracing is also pretty cheap when compared with som…

It's not the cost per se, though I do think they're pretty high for some features. It's the pricing models and the associated patterns.

For instance, you have to pay Datadog per host you install the agent on. In addition to the per host cost, you have to pay per container you run on that host (past a very small baseline per host), and the per contain cost turns out to be nearly as high as the per host cost if you have reasonable density. Why am I paying Datadog per container I run? Aside from a not particularly useful dashboard, why does a process namespace and some cgroup metrics nearly double my bill? They are literally just processes on a server. Because Datadog wants you to run more hosts, so you install more agents.

Every feature they add also seems to be charged separately, but is not behind any sort of feature gate. This means new features just show up for my developers, and they have no clue if it costs money to use them. I can't just disable or cap, for example, their custom metrics per user, per project, or at all. So when my developers see a useful feature and start using it, all of a sudden I have an extra $10k on my monthly bill. Even more fun are features that show up and are initially free but then start charging.

This is such a pain that we've had to tell dev teams not to use Datadog features outside of a curate list. Every product has some rough edges, but with Datadog the patterns are all setup such that you end up paying them thousands of extra dollars. Again, great product, but not a business I would be interested in associating with again given the choice.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#64

Post by gnufx » Fri, Jul 31, 2020, 6:20 PM UTC

I never see an important system management principle brought up: If you get a user complaint (for some value of "user") and not an alert, you should fix the monitoring system so that you don't get another occurrence of it or related problems. Obviously that's within reason, depending on the circumstances; the effort might not be worth it.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#65

Post by gnufx » Fri, Jul 31, 2020, 6:33 PM UTC

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

Why should you have to use Prometheus? There are plenty of options, and good reasons why you might want to push data rather than pull. Measurements should minimize perturbation of the system being measured, and the (computer) system generating data is likely best placed to determine when and how, when that matters -- e.g. in HPC, where jitter is important.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#66

Post by gnufx » Fri, Jul 31, 2020, 6:38 PM UTC

secondcoming wrote: post #28

We log extensively. Here are some of my thoughts it - at least in C++, the requirement to be able to log from pretty much anywhere can lead to messy code that either passes a reference to your logger to all classes that might possibly need it, or you've got an extern global somewhere. Yuck. - logging can enable laziness. Being able to log that something weird happened can be considered a sufficient substitute for pro…

> the requirement to be able to log from pretty much anywhere can lead to messy code

Ah, Milewski's example of insight from the supposedly useless mathematical stuff: https://bartoszmilewski.com/2014/12/23/kleisli-categories/ (and the corresponding lecture video).

Re: Monitoring demystified: A guide for logging, tracing, metrics

#67

Post by chandraonline » Fri, Jul 31, 2020, 6:43 PM UTC

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

I would add that if you add a traceId to #1 and use something like https://www.jaegertracing.io/. You get even more.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#68

Post by code_slinger » Fri, Jul 31, 2020, 6:49 PM UTC

_i4zn wrote: post #46

viraptor wrote: post #21

Earlier quoted context omitted.

> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

> None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you). I was thinking "God, this is exactly why I hate Datadog" as I was reading your description and got a great laugh when I reached the end. Their billing is absolutely byzantine. I don't know that I've ever seen a company that had such a stark difference between gr…

> Their product is really the best turn-key option out there, but I'm always hesitant to use its features without double checking it's not going to add 50% to my bill.

You might want to check out New Relic One, especially with the new pricing model. I think they even added a Prometheus integration recently?

Re: Monitoring demystified: A guide for logging, tracing, metrics

#69

Post by adamzegelin » Fri, Jul 31, 2020, 7:07 PM UTC

twic wrote: post #33

KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming Yeah nah, but, okay, nah yeah. Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app. Unless you're v…

> if you're using the Dropwizard Metrics library, for example, you've already lost.

Can you go into a bit more detail here? Curious to know where Dropwizzard goes wrong.

I prefer to use the Prometheus client libraries where possible. Prometheus' data model is "richer" -- metric families and labels, rather than just named metrics. Adapting from Dropwizzard to Prometheus is a pain, and never results in the data feeling "native" to Prometheus.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#70

Post by dionian » Fri, Jul 31, 2020, 7:14 PM UTC

KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

sometimes log volume would be too high (high TX count)

although in most of the systems in my career it has not been the case.