Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log by time, not by count

johnscolaro.xyz

Post reply on HN
Print view View original

71–80 of 109 posts

Re: Log by time, not by count

#71

Post by layer8 »

ot wrote: post #69
layer8 wrote: post #61

Earlier quoted context omitted.

Your “actually log” is within one thread. Either the threads all do their separate “actually log”, or they have to synchronize their data into a single shared “actually log”.

> Either the threads all do their separate “actually log” But why? Often the purpose is just to log a "been here" signal with some additional details for diagnostics. You don't need to include an accumulation of everything that happened since the last log. All that you care about is that the log happens at most 1/period, say once per second. If you do want to also log some data that accumulates everything that happen…

Well, yeah, it depends on what you want to log.

> collect all the buffers

Which requires some sort of synchronization (or lock-free data structures), because of concurrent writes by other threads. In that situation, you can also simply use a dedicated thread to periodically flush the log buffers.

Re: Log by time, not by count

#72

Post by chaps »

aarmenaa wrote: post #67
stackskipton wrote: post #58

As SRE/DevOps/Ops whatever, I'm screaming. Metrics should be emitted in separate stream and never by logs outside corner cases. Logs should be used to determine WHY the system is having issues but never IS the system having issues. Log alerting is a fools errand that looks like a great idea at start but quickly becomes a sand trap that will drive future people crazy and at scale, will overwhelm systems. Why is log al…

I've noticed that for some reason developers really like using logs in place of actual metrics. We use Datadog, and multiple times now I have seen devs add additional logging to an application just so they can then create a monitor that counts those log events. I think it's a path of least resistance thing; emitting logs is very easy, and counting them is also very easy. Reporting actual metrics isn't really difficul…

Because when the application is breaking it's good to know why! Logs can be just as ephemeral as metrics -- in many cases, even more so. They're not even mutually exclusive.

Where exactly does this anti-logs sentiment come from? Is it because tools like datadog can be lackluster for reading logs across bunches of hosts?

Re: Log by time, not by count

#73

Post by lelandbatey »

I think "time based logging" is a bit of an anti-pattern, as that's LITERALLY a metric. The thrust of this is "sample at a consistent rate" which, yeah sample your numbers at a consistent rate. But also, probably don't use logs for this, probably use a metrics tracking system.

Re: Log by time, not by count

#74

Post by taeric »

Often logging backends have a sampling rate to take care of this for you. As it is almost certainly better to setup a buffer in your logging layer and deal with this there, than to try and do this everywhere in your code.

Re: Log by time, not by count

#75

Post by ot »

layer8 wrote: post #71
ot wrote: post #69

Earlier quoted context omitted.

> Either the threads all do their separate “actually log” But why? Often the purpose is just to log a "been here" signal with some additional details for diagnostics. You don't need to include an accumulation of everything that happened since the last log. All that you care about is that the log happens at most 1/period, say once per second. If you do want to also log some data that accumulates everything that happen…

Well, yeah, it depends on what you want to log. > collect all the buffers Which requires some sort of synchronization (or lock-free data structures), because of concurrent writes by other threads. In that situation, you can also simply use a dedicated thread to periodically flush the log buffers.

Yes, but crucially, only 1/period, not on every single "should log?" call, which is what I was referring to above. The per-thread mutexes are uncontended virtually all the time.

Re: Log by time, not by count

#76

Post by liveoneggs »

chaps wrote: post #72
aarmenaa wrote: post #67

Earlier quoted context omitted.

I've noticed that for some reason developers really like using logs in place of actual metrics. We use Datadog, and multiple times now I have seen devs add additional logging to an application just so they can then create a monitor that counts those log events. I think it's a path of least resistance thing; emitting logs is very easy, and counting them is also very easy. Reporting actual metrics isn't really difficul…

Because when the application is breaking it's good to know why! Logs can be just as ephemeral as metrics -- in many cases, even more so. They're not even mutually exclusive. Where exactly does this anti-logs sentiment come from? Is it because tools like datadog can be lackluster for reading logs across bunches of hosts?

logs are for people and trying to make them for computers is hard.

Re: Log by time, not by count

#77

Post by Espressosaurus »

chaps wrote: post #72
aarmenaa wrote: post #67

Earlier quoted context omitted.

I've noticed that for some reason developers really like using logs in place of actual metrics. We use Datadog, and multiple times now I have seen devs add additional logging to an application just so they can then create a monitor that counts those log events. I think it's a path of least resistance thing; emitting logs is very easy, and counting them is also very easy. Reporting actual metrics isn't really difficul…

Because when the application is breaking it's good to know why! Logs can be just as ephemeral as metrics -- in many cases, even more so. They're not even mutually exclusive. Where exactly does this anti-logs sentiment come from? Is it because tools like datadog can be lackluster for reading logs across bunches of hosts?

My experience is that metrics may tell you something is wrong, but logs are required to tell you what went wrong and why.

A simple fixed-length rolling buffer can get you pretty far for logging, but it isn't something you necessarily want to get off-device except when something bad has happened.

Re: Log by time, not by count

#78

Post by haiku2077 »

sofixa wrote: post #27
haiku2077 wrote: post #20

Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…

> log as JSON objects instead of lines of text Or logfmt which is easier to read for humans, lower overhead, and is still structured and supported in at least Grafana/Loki for parsing and queries.

Does logfmt allow nesting? I often inckude data structures like dicts/maps, arrays or complex objects in my JSON logs.

Re: Log by time, not by count

#79

Post by liveoneggs »

haiku2077 wrote: post #20

Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…

{ "timestamp": "2025-07-21 11:31:52", "event": "user-login", ... }, { "event": 7521, "event-type": "error-code-2025" }, { "message": Traceback (most recent call last): File "", line 6, in lumberjack bright_side_of_life() ~~~~~~~~~~~~~~~~~~~^^ File "", line 10, in bright_side_of_life return t[5] ~^^^ IndexError: tuple index out of range }, { "timestamp": "1753112562", "event": "user-click", ... },

Re: Log by time, not by count

#80

Post by haiku2077 »

liveoneggs wrote: post #79
haiku2077 wrote: post #20

Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…

{ "timestamp": "2025-07-21 11:31:52", "event": "user-login", ... }, { "event": 7521, "event-type": "error-code-2025" }, { "message": Traceback (most recent call last): File " ", line 6, in lumberjack bright_side_of_life() ~~~~~~~~~~~~~~~~~~~^^ File " ", line 10, in bright_side_of_life return t[5] ~^^^ IndexError: tuple index out of range }, { "timestamp": "1753112562", "event": "user-click", ... },

For Python I like structlog which has a feature for including exception info in a readable form: https://www.structlog.org/en/stable/exceptions.html
Post reply on HN