Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring demystified: A guide for logging, tracing, metrics

techbeacon.com

Post reply on HN
Print view View original

71–80 of 93 posts

Re: Monitoring demystified: A guide for logging, tracing, metrics

#71

Post by Godel_unicode »

rudasn wrote: post #52
smoe wrote: post #37

Earlier quoted context omitted.

It depends where you are in the world. When I was working in Switzerland, most SaaS pricing were no-brainers for us. But since I work in Latin America for small companies with local costumers, all the different services and tools you might want to use, with prices targeted at "western" customers, much more quickly add up to the equivalent of having multiple people on staff full time. Still it is often not worth to ro…

> It depends where you are in the world. Exactly this. We operate in Eastern Europe with local clients, offering on-prem SaaS. If I added all my clients' servers on datadog it would very easily eat through our profit margins. > Still it is often not worth to roll your own I tried hard not to, but at at the end, after spending 1 week trying to setup netdata and failed, I decided not to spend another week trying to set…

Tangent: what's the difference between on-prem SaaS and just "software"?

Re: Monitoring demystified: A guide for logging, tracing, metrics

#72

Post by dig1 »

The Art of Monitoring [1], covers most of these stuff in a unified manner.

You are introduced to some basics (push vs. pull monitoring), then proceeded with simple system metrics collection (cpu, memory) via collectd, then goes to logs ingestion and ends up extracting application-specific metrics from jvm and python applications.

I highly recommend it, even for seasoned professionals.

[1] https://artofmonitoring.com/

Re: Monitoring demystified: A guide for logging, tracing, metrics

#73

Post by smoe »

Godel_unicode wrote: post #71
rudasn wrote: post #52

Earlier quoted context omitted.

> It depends where you are in the world. Exactly this. We operate in Eastern Europe with local clients, offering on-prem SaaS. If I added all my clients' servers on datadog it would very easily eat through our profit margins. > Still it is often not worth to roll your own I tried hard not to, but at at the end, after spending 1 week trying to setup netdata and failed, I decided not to spend another week trying to set…

Tangent: what's the difference between on-prem SaaS and just "software"?

I guess it refers to them servicing (updating, reacting to downtime, etc.) the software, while it being deployed on premise? In contrast to the clients IT department doing so.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#74

Post by therealdrag0 »

twic wrote: post #33
KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming Yeah nah, but, okay, nah yeah. Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app. Unless you're v…

The problem is post calculating is so slow. At least from my naive viewpoint. I can load dozens of graphs in datadog in seconds, can change tags or time frame and takes literally a second to load. Our Splunk dashboards can take over a minute to load, and reload for any change is more waiting.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#75

Post by therealdrag0 »

adamzegelin wrote: post #69
twic wrote: post #33

Earlier quoted context omitted.

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming Yeah nah, but, okay, nah yeah. Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app. Unless you're v…

> if you're using the Dropwizard Metrics library, for example, you've already lost. Can you go into a bit more detail here? Curious to know where Dropwizzard goes wrong. I prefer to use the Prometheus client libraries where possible. Prometheus' data model is "richer" -- metric families and labels, rather than just named metrics. Adapting from Dropwizzard to Prometheus is a pain, and never results in the data feeling…

I think they just mean the host is aggregating, so any further aggregation is compounded slant time the data. Like StatsD’s default is shipping metrics every 10s, so if you graph it and your graph rolls up those data points into 10 minute data points (cuz you’re viewing a week at once), then you’re averaging an average. Or averaging a p95. People often miss that this is happening, and it can drastically change the narrative.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#76

Post by RabbitmqGuy »

therealdrag0 wrote: post #74
twic wrote: post #33

Earlier quoted context omitted.

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming Yeah nah, but, okay, nah yeah. Generating metrics in the app is much more intrusive, and requires that you figure out the metrics you need ahead of time. It adds dependencies, sockets, and threads to your app. Unless you're v…

The problem is post calculating is so slow. At least from my naive viewpoint. I can load dozens of graphs in datadog in seconds, can change tags or time frame and takes literally a second to load. Our Splunk dashboards can take over a minute to load, and reload for any change is more waiting.

Splunk taking minutes to load dashboard is not a problem imposed by post-calculation, it's more of a problem of lack of schema.

Most post calculation works on free text logs and thus has to regex it's way to a solution. But it doesn't have to be that way; that's why the original poster talked about a lack of tooling in the post-calculation world

Re: Monitoring demystified: A guide for logging, tracing, metrics

#77

Post by rudasn »

Godel_unicode wrote: post #71
rudasn wrote: post #52

Earlier quoted context omitted.

> It depends where you are in the world. Exactly this. We operate in Eastern Europe with local clients, offering on-prem SaaS. If I added all my clients' servers on datadog it would very easily eat through our profit margins. > Still it is often not worth to roll your own I tried hard not to, but at at the end, after spending 1 week trying to setup netdata and failed, I decided not to spend another week trying to set…

Tangent: what's the difference between on-prem SaaS and just "software"?

We provide both the hardware and the software. Our clients are pretty small, with no IT staff, and no technical expertise.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#78

Post by rudasn »

smoe wrote: post #73
Godel_unicode wrote: post #71

Earlier quoted context omitted.

Tangent: what's the difference between on-prem SaaS and just "software"?

I guess it refers to them servicing (updating, reacting to downtime, etc.) the software, while it being deployed on premise? In contrast to the clients IT department doing so.

Yes that's right.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#79

Post by rob-olmos »

chasers wrote: post #59
KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Only if you're using Elastic.

I was going to try out Elastic APM for self-hosted APM option. Would that be the same case of a waste of time, energy, money? TIA for any insights!

Re: Monitoring demystified: A guide for logging, tracing, metrics

#80

Post by gdcohen »

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

Gavin from Zebrium here. Completely concur with #1. We are big advocates of writing good logs and not having to worry about structured vs unstructured (and even if you structure your logs, you'll still probably have to deal with unstructured logs in third party components).

Our approach to deal with logs is to use ML to structure them after the fact (and we can deal with changing log structures). You can read about it in a couple of our blogs like: https://www.zebrium.com/blog/using-ml-to-auto-learn-changing... and https://www.zebrium.com/blog/please-dont-make-me-structure-l....

Post reply on HN