Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Monitoring demystified: A guide for logging, tracing, metrics

techbeacon.com

Post reply on HN
Print view View original

11–20 of 93 posts

Re: Monitoring demystified: A guide for logging, tracing, metrics

#11

Post by msolujic »

KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

Agreed for most what was said there. Still, I find that people mostly use SLA as only thing important to track for alerting and incidents arousal. There is a lot of said about importance of defining solid SLI - Service Level Indicators which are aligned to SLO - Service Level Objectives SLAs are usually given to external user of SaaS, not very useful for SRE team.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#12

Post by kostarelo »

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

> Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number), and that the message will help you debug.

if you include all these information and the logs are not structured, you won't get much information out of them.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#13

Post by neolog »

KaiserPro wrote: post #3

A few things I have learnt along the way: Logs are great, but only once you've identified the problem. If you are searching through logs to _find_ a problem, its far too late. Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming (example: dont use access logs to collect 4xx/5xx a…

> Processing/streaming logs to get metrics is a terrible waste of time, energy and money. Spend that producing high quality metrics directly from the apps you are looking after/writing/decomming

The other side is that I don't know what metrics I'll want until later.

When do you think it's better to pull metrics from structured logs vs generating metrics in app?

Re: Monitoring demystified: A guide for logging, tracing, metrics

#14

Post by dig1 »

say_it_as_it_is wrote: post #4

Is there an open source solution for processing streams of structured and unstructured logs and routing then onward? I see solutions for moving logs to elastic or Kafka but nothing for evaluating the log.

Riemann [1]. You can create custom endpoint for accepting almost any kind of messages, logs or data.

[1] http://riemann.io/

Re: Monitoring demystified: A guide for logging, tracing, metrics

#15

Post by ysoft »

say_it_as_it_is wrote: post #4

Is there an open source solution for processing streams of structured and unstructured logs and routing then onward? I see solutions for moving logs to elastic or Kafka but nothing for evaluating the log.

I haven't found anything, we are moving to hosted Humio really soon. It uses kafka

Re: Monitoring demystified: A guide for logging, tracing, metrics

#16

Post by onefuncman »

waihtis wrote: post #5

> Logging is critical to detecting attacks and intrusions. Yes, but not universally - and just collecting logs will not take you far. Logging everything and trying to approach security via the ’collect all data’ is both expensive and inaccurate, and one of the major inefficiencies in modern cyber.

This is done efficiently at scale by both Cylance and Crowdstrike, but is certainly only one part of a defense in depth strategy.

There are viable products around human threat hunting which would be impossible without a 'collect all the data' component.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#17

Post by FrontAid »

Recently, I was searching for a service which offers those functionalities on a very basic level. I tried several options and was really disappointed with all of them. The only one that I found to be usable was https://logdna.com/. I've now been using it for a couple of weeks and it works OK. It offers logging, alerts, metrics/dashboards, and some other things. And all that for a reasonable pricing.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#18

Post by deepGem »

buro9 wrote: post #2

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number…

From the post "The key, he says, is using the right transaction identifiers so that calls can be traced across components, services, and queues".

I think this is a key feature not many people implement especially in today's world of over blown micro services, having a transaction id from the time the request hits the reverse-proxy till the database write is so helpful in debugging, saves a ton of time.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#19

Post by onefuncman »

say_it_as_it_is wrote: post #4

Is there an open source solution for processing streams of structured and unstructured logs and routing then onward? I see solutions for moving logs to elastic or Kafka but nothing for evaluating the log.

The "OG" in the space is collectd, which is still my favorite choice if you are responsible at the operating system level: https://collectd.org/wiki/index.php/Chains

https://github.com/elastic/logstash was one of the first modern approaches. I started using it less the more often I ran into JRuby related bugs.

https://github.com/trivago/gollum is my pick from the golang ecosystem.

There are many more variants depending on how much complexity you are trying to apply. If you need to apply machine learning models, for example, you're probably going to end up with something similar to Apache Storm, though I don't know if it's operational story has improved enough to consider it over other alternatives, I lost track years ago between Apache Spark and the half dozen other stream processing projects.

Re: Monitoring demystified: A guide for logging, tracing, metrics

#20

Post by waihtis »

onefuncman wrote: post #16
waihtis wrote: post #5

> Logging is critical to detecting attacks and intrusions. Yes, but not universally - and just collecting logs will not take you far. Logging everything and trying to approach security via the ’collect all data’ is both expensive and inaccurate, and one of the major inefficiencies in modern cyber.

This is done efficiently at scale by both Cylance and Crowdstrike, but is certainly only one part of a defense in depth strategy. There are viable products around human threat hunting which would be impossible without a 'collect all the data' component.

You are correct, and this is the key part - what % of organizations have money, skills and people to build a robust enough capability around threat hunting, for example?

I’ve been super lucky to meet various orgs and their security in all geographies and many industries and my gut feeling is 1 out of 10 teams.

Post reply on HN