Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging practices I follow

16elt.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Logging practices I follow

#51

Post by cpach »

partdavid wrote: post #47
david38 wrote: post #41

Lots of logs can be replaced by metrics. People go crazy with logs.

Very much (e.g. timing data, counts like success and error counts and operations should be metrics). If not metrics, then traces. If not metrics or traces, then business events (e.g. like alerts, audit records). Almost everything that people put in logs actually belongs somewhere else, in my opinion. Which is evidenced by so much of log processing being about turning logs back into whatever it was they were supposed…

Excuse an old cave man, but in what way does traces and metrics replace logs?

Re: Logging practices I follow

#52

Post by adra »

david38 wrote: post #41

Lots of logs can be replaced by metrics. People go crazy with logs.

Because by and large, a bunch of technologies have very spotty support for metrics, and almost always involve third party systems. Logs are dumb simple. I'm not saying metrics aren't valuable (quite the opposite), but getting started in metrics usually involves some level of institutional investment.

Re: Logging practices I follow

#53

Post by guhidalg »

ryeguy_24 wrote: post #20

Question...why no mention of "change" logs? I'm curious as to why I don't see change logs mentioned often as an important overlay to general system logging. I liken errors and debugs all related to heart beat and breathing rate but without information like "climbing stairs" or "changed medication", it may be hard to understand context or understand why new errors are being seen. The first question I would expect to a…

At $FORMER_EMPLOYER, we had a company-wide service that tracked changes of all kinds: source code changes, deployment changes, config changes, etc... It was useful because some changes are never reflected in YOUR logs, but they are reflected in someone else's logs. The systems that tried to do log change detection were all bad because the ML-driven clustering systems didn't produce interpretable information.

Re: Logging practices I follow

#54

Post by adra »

cpach wrote: post #51
partdavid wrote: post #47

Earlier quoted context omitted.

Very much (e.g. timing data, counts like success and error counts and operations should be metrics). If not metrics, then traces. If not metrics or traces, then business events (e.g. like alerts, audit records). Almost everything that people put in logs actually belongs somewhere else, in my opinion. Which is evidenced by so much of log processing being about turning logs back into whatever it was they were supposed…

Excuse an old cave man, but in what way does traces and metrics replace logs?

"Operation xyz completed in 15.445 seconds." This can be expressed as a metric (or a trace) so that the "operation_xyz_completed" is a metric and 15.445 seconds is evaluated as a metric data point. The result is an easily chart to graph the average, p99, whatever of the operation to gauge if this is normal or exceptional. It's dead simple to alert on metrics as well often, so it helps to unlock alerting. Log alerts are valid but often more limiting without a bunch of parsing or being really naive.

Re: Logging practices I follow

#55

Post by ilyt »

tobylvp wrote: post #27

I think all of this sounds fine in theory, but the reality is that most logged information will not be needed - ever. Exactly what information is needed when can be difficult to predict. So, if a developer feels something might be important, they should probably log it. Within reason, I think it is better to have it and not need it than to need it and not have it. It seems the author is putting a heavy emphasis on tr…

Same applies to metrics, if it moves, log it, especially with semi-recent TSDBs allowing to store a metric ton of metric in very litte space (well, aside from Mimir from Grafana that managed to fail that lesson...)

> It seems the author is putting a heavy emphasis on trying to create readable logs. Finding the signal in the noise. I am biased, but I think this is a failure of the tools used to read the logs rather than the logs themselves. This is why I wrote LogViewPlus (https://www.logviewplus.com/).

Well, it's failure on many levels. "informational" logs, like say your traditional access.log are mostly used for metrics/analytics but also server as context to any warning or error that app returns during processing of the request. But at same time you kinda want them to have be encoded in something more structured than "a piece of formatted text" (say, a JSON line), while at the same this approach reduces glancability of logs to near zero.

On other side having hundreds lines of code just to decode logs into something searchable is also pretty bad and most importantly very fragile to code changes.

"Just do everything in machine format then send it to collector" like Jaeger (with bonus being ability to do distributed tracing) is a solution but very obese one and needs every app supporting distributed tracing

Re: Logging practices I follow

#56

Post by ilyt »

masklinn wrote: post #45
charles_f wrote: post #8

One additional thing I like in structured logs is having some form of context level information be included eith your logs, so that you already know things like tenant id, user id, request id, and basic parameters of the request without having to rewrite all that everytime you get an exception. Unrelated: I live in the Pacific north west and I clicked on this expecting to find a list like "don't log old growth for ti…

Is there a structured form of tracing? Because I feel like this contextual information should easily be part of a trace.

opentelemetry + Jaeger all-in-one binary is probably easiest way to start experimenting. Results are pretty useful but it is more work than simple logging

Re: Logging practices I follow

#57

Post by ilyt »

adra wrote: post #54
cpach wrote: post #51

Earlier quoted context omitted.

Excuse an old cave man, but in what way does traces and metrics replace logs?

"Operation xyz completed in 15.445 seconds." This can be expressed as a metric (or a trace) so that the "operation_xyz_completed" is a metric and 15.445 seconds is evaluated as a metric data point. The result is an easily chart to graph the average, p99, whatever of the operation to gauge if this is normal or exceptional. It's dead simple to alert on metrics as well often, so it helps to unlock alerting. Log alerts a…

Well, you usually want metric and trace of it, at the very least if it fails

Re: Logging practices I follow

#58

Post by ilyt »

marginalia_nu wrote: post #24
darkwater wrote: post #17

Earlier quoted context omitted.

If your infra is not on-prem, yes it will cost you more money as you are generating more and more and bigger logs.

You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less. I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at w…

> You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data.

Now count that for queryable data source so running a database of some sort (Elasticsearch probably for logs) 24/7 at fast enough speeds that it is ops-useful

Metrics are significantly cheaper tho, at least if you use some dedicated TSDB with good storage engine like Victoriametrics or influxdb.

Re: Logging practices I follow

#59

Post by adra »

marginalia_nu wrote: post #6
chaghalibaghali wrote: post #3

Earlier quoted context omitted.

There are still going to be time and effort costs involved in scaling that infrastructure as your log volume increases

You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design. If you're at the scale where this actually matters and you're serving millions or…

Except you know when you.actually want to do something valuable with all those logs. You _should_ be creating logs (signals) to be valuable in some way (diagnostics, alerting, canaries statistics), etc. If you're just dumping logs into opaque blobs that are never looked at them sure write them to blobs to your heart's content and have fun hunting and pecking for reasons you're users are already screaming at you. That strategy is fine, but the limitations are clear. It's reactive.

Re: Logging practices I follow

#60

Post by adra »

ryeguy_24 wrote: post #20

Question...why no mention of "change" logs? I'm curious as to why I don't see change logs mentioned often as an important overlay to general system logging. I liken errors and debugs all related to heart beat and breathing rate but without information like "climbing stairs" or "changed medication", it may be hard to understand context or understand why new errors are being seen. The first question I would expect to a…

Because these are probably better expressed as metrics. A good observation platform will try to weave all of these concepts together to form that picture you're describing. Some observation providers call these events or annotations.

Theres a difficult cognitive dissonance for looking at a logged event and know that it's unusual or not. Some signals are in fact positively correlated with a failure, and some times it's just noise. Good tooling hopefully makes the distinction of these two options as easy as possible.

Post reply on HN