Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log by time, not by count

johnscolaro.xyz

Post reply on HN
Print view View original

81–90 of 109 posts

Re: Log by time, not by count

#81

Post by haiku2077 »

delusional wrote: post #43
KronisLV wrote: post #31

Earlier quoted context omitted.

My colleagues love to log as little as possible and most of the projects I’ve seen still treat logs as files instead of event streams that could have some search and filtering and categorization and automated alerting. It’s kind of unfortunate, because for example there’d be pushback against logging branches in code etc., except for trace logs (that others wouldn’t add) that are also off most of the time when problem…

I've had colleagues try this. It rarely works. Logging every if end up introducing a huge amount of overhead, both in terms of processing power, but especially in terms of storage. You almost always end up having to filter based on some sort of log level that you then turn off by default in production. The problem with that is that you're now required to reproduce the issue after turning on the logging, and if you al…

Something I've done in the past is send some logs to BigQuery for cheap mass storage and others to Grafana for fast querying and use in live dashboards. Basically a filter rule in our logging agent to send different events to different destinations. I think with some more hacking I could get both datasources into the same Grafana frontend...

Re: Log by time, not by count

#82

Post by pphysch »

liveoneggs wrote: post #79
haiku2077 wrote: post #20

Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…

{ "timestamp": "2025-07-21 11:31:52", "event": "user-login", ... }, { "event": 7521, "event-type": "error-code-2025" }, { "message": Traceback (most recent call last): File " ", line 6, in lumberjack bright_side_of_life() ~~~~~~~~~~~~~~~~~~~^^ File " ", line 10, in bright_side_of_life return t[5] ~^^^ IndexError: tuple index out of range }, { "timestamp": "1753112562", "event": "user-click", ... },

logfmt > json IMO

For Python users, there's a "logfmter" package which is enormously more straightforward than the popular "structlog" one.

Re: Log by time, not by count

#83

Post by kixelated »

AdieuToLogic wrote: post #2

This post falls into a common trap; conflating logging with metrics. Log interesting things, where interesting is defined as context outside what the "happy path" execution performs. Collect and make available system metrics, such as invocation counts, processing time histograms, etc., to make available what the post uses log statements to disseminate same.

Absolutely.

Logs should be bursty, because they're most useful when debugging rare issues. If you have identical log lines, then that should have been a metric instead.

Metrics should be sampled based on frequency, because they deduplicate. I'm a huge fan of logarithmically sampling metrics.

Re: Log by time, not by count

#84

Post by DougN7 »

taeric wrote: post #74

Often logging backends have a sampling rate to take care of this for you. As it is almost certainly better to setup a buffer in your logging layer and deal with this there, than to try and do this everywhere in your code.

Except that you waste CPU cycles preparing the log string and calling the log function only to have it all thrown away.

Re: Log by time, not by count

#85

Post by pvorb »

DougN7 wrote: post #84
taeric wrote: post #74

Often logging backends have a sampling rate to take care of this for you. As it is almost certainly better to setup a buffer in your logging layer and deal with this there, than to try and do this everywhere in your code.

Except that you waste CPU cycles preparing the log string and calling the log function only to have it all thrown away.

At least in the Java world it is common to let the logging framework handle parameter evaluation for you.

Re: Log by time, not by count

#86

Post by jdbernard »

pvorb wrote: post #85
DougN7 wrote: post #84

Earlier quoted context omitted.

Except that you waste CPU cycles preparing the log string and calling the log function only to have it all thrown away.

At least in the Java world it is common to let the logging framework handle parameter evaluation for you.

This feels like one of those "not obvious until you've seen it in production" requirements: any production-ready logging framework should have a mechanism to delay parameter evaluation until after the threshold/destination checks are performed. Most languages have some version of deferred execution (lazy evaluation, thunks, etc.)

Re: Log by time, not by count

#87

Post by marcosdumay »

aarmenaa wrote: post #67
stackskipton wrote: post #58

As SRE/DevOps/Ops whatever, I'm screaming. Metrics should be emitted in separate stream and never by logs outside corner cases. Logs should be used to determine WHY the system is having issues but never IS the system having issues. Log alerting is a fools errand that looks like a great idea at start but quickly becomes a sand trap that will drive future people crazy and at scale, will overwhelm systems. Why is log al…

I've noticed that for some reason developers really like using logs in place of actual metrics. We use Datadog, and multiple times now I have seen devs add additional logging to an application just so they can then create a monitor that counts those log events. I think it's a path of least resistance thing; emitting logs is very easy, and counting them is also very easy. Reporting actual metrics isn't really difficul…

Well, if you have low enough volume, you have to implement logs anyway and don't have a reason to optimize their volume.

I imagine many people learn on an environment like this and get thrown in a high volume one without chance to adapt.

Re: Log by time, not by count

#88

Post by closeparen »

jiehong wrote: post #55
haiku2077 wrote: post #20

Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…

Following this advice, I've seen service where you could say the main function was to produce logs, and the actual response to the user was only a small part of the traffic generated. What we really need is smart logging: only log the full span when an error is detected, otherwise no need for it. But it's not a very well supported case.

>when an error is detected

I wish we could all be so lucky as to only care "when an error is detected." Logging is about creating breadcrumbs that can be searched and cross-referenced to piece together what happened when no error is detected, but the behavior is nevertheless suspect.

Re: Log by time, not by count

#89

Post by marcosdumay »

chaps wrote: post #72
aarmenaa wrote: post #67

Earlier quoted context omitted.

I've noticed that for some reason developers really like using logs in place of actual metrics. We use Datadog, and multiple times now I have seen devs add additional logging to an application just so they can then create a monitor that counts those log events. I think it's a path of least resistance thing; emitting logs is very easy, and counting them is also very easy. Reporting actual metrics isn't really difficul…

Because when the application is breaking it's good to know why! Logs can be just as ephemeral as metrics -- in many cases, even more so. They're not even mutually exclusive. Where exactly does this anti-logs sentiment come from? Is it because tools like datadog can be lackluster for reading logs across bunches of hosts?

It GP is not anti-log.

It's claiming that you should output working logs, metrics, and failure logs into different streams.

Re: Log by time, not by count

#90

Post by stackskipton »

chaps wrote: post #72
aarmenaa wrote: post #67

Earlier quoted context omitted.

I've noticed that for some reason developers really like using logs in place of actual metrics. We use Datadog, and multiple times now I have seen devs add additional logging to an application just so they can then create a monitor that counts those log events. I think it's a path of least resistance thing; emitting logs is very easy, and counting them is also very easy. Reporting actual metrics isn't really difficul…

Because when the application is breaking it's good to know why! Logs can be just as ephemeral as metrics -- in many cases, even more so. They're not even mutually exclusive. Where exactly does this anti-logs sentiment come from? Is it because tools like datadog can be lackluster for reading logs across bunches of hosts?

For me, I don't use Datadog so it's not that $ParticularTool does not work with logs, it's all stuff I put in my original post, it's a ton of samples, filtering puts heavy strain on the systems and it's extremely brittle IME.

If you have good metrics, you can generally get much further not even logging aggregating outside tossing everything into STDOUT and checking on it when you have alerts.

Post reply on HN