Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log by time, not by count

johnscolaro.xyz

Post reply on HN
Print view View original

31–40 of 109 posts

Re: Log by time, not by count

#31

Post by KronisLV »

haiku2077 wrote: post #20

Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…

My colleagues love to log as little as possible and most of the projects I’ve seen still treat logs as files instead of event streams that could have some search and filtering and categorization and automated alerting.

It’s kind of unfortunate, because for example there’d be pushback against logging branches in code etc., except for trace logs (that others wouldn’t add) that are also off most of the time when problems actually happen. It does help a lot in personal projects though, albeit the limited traffic there kinda minimizes any problems that ample logging might otherwise cause.

At least it’s possible to move in the direction of adding some APM like GlitchTip or Skywalking.

Re: Log by time, not by count

#33

Post by jabjq »

AdieuToLogic wrote: post #18
JohnScolaro wrote: post #10

Earlier quoted context omitted.

Thanks for taking the time to reply! I'm relatively new to working on this type of system (large scale, event driven) and half posted because I know there are people on HN way better than me at this, and was curious about their opinions. In the end, what's the difference between a log and a metric? Is one structured, and one unstructured? Is one a giant blob of text, and the other stored in a time series db? At the m…

> In the end, what's the difference between a log and a metric? Essentially, a log entry is the emission of state known by an individual code execution path at the point the log entry can be produced, whereas a metric is a measurement of a specific runtime execution performed by the system. For example, a log entry of: module_logger.info( f"Processed {num_events_processed_since_last_log} events." ) Emits a log entry…

You shouldn’t use f-strings with logging.

Re: Log by time, not by count

#34

Post by sjducb »

I think count based logging works really well for batch processing jobs where each item is a fairly constant amount of work. The log shows the time you started and prints another line (with the time) for each 1% of the batch. You can see the progress and guess when it will be done.

The human who is debugging an issue can see when we started, see that some processed successfully, see regular progress through the batch, then see that the 58th percentile batch hasn’t completed and that’ll be where the problem is.

The main benefit over the time based logging is that the code is much simpler, and the log output is simpler too.

There are even libraries like tqdm that do this for you in one line of code.

Re: Log by time, not by count

#35

Post by ethagnawl »

haiku2077 wrote: post #20

Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…

> if possible make log level per user, so can debug specific user issue

That's clever and I'll definitely use it in the future.

Re: Log by time, not by count

#36

Post by aflag »

AdieuToLogic wrote: post #2

This post falls into a common trap; conflating logging with metrics. Log interesting things, where interesting is defined as context outside what the "happy path" execution performs. Collect and make available system metrics, such as invocation counts, processing time histograms, etc., to make available what the post uses log statements to disseminate same.

I thought the post was nice. I've written scripts before where I'd just print by count and be overwhelmed. I should've just used time instead of changing the count number

Re: Log by time, not by count

#37

Post by Thorrez »

pjz wrote: post #16

The practical problem with logging by time is that it's not resource constrained: holding N seconds of logs, even when each line is a bounded size, takes potentially unlimited memory. Logging 'by count' used a bounded amount of memory, and is easy to implement with a fixed size array in memory.

You're talking about a different scenario than than the article. The article is about a strategy of how to generate a single log line. You're talking about a strategy of how to batch multiple log lines together.

Re: Log by time, not by count

#38

Post by RHSman2 »

Logging (at scale) is the most important thing to understand reality in your system. At scale, realities are many.

Metrics enable the ability to aggregate concepts into some kind of meaning.

Meaning can then have alerts associated to them.

You cannot create metrics on things you don’t know, which is why logging is the base.

Re: Log by time, not by count

#39

Post by RHSman2 »

RHSman2 wrote: post #38

Logging (at scale) is the most important thing to understand reality in your system. At scale, realities are many. Metrics enable the ability to aggregate concepts into some kind of meaning. Meaning can then have alerts associated to them. You cannot create metrics on things you don’t know, which is why logging is the base.

Btw: logging and the ability to observe systems was the single and most successful act I have done in my career as a Data Product Manager. I have had to fight, do tricks and ‘play the game’ so much.

I cannot stress the importance of understanding atomic movements.

The cost is high but not as high as the cost of not knowing.

Re: Log by time, not by count

#40

Post by incorrecthorse »

_kb wrote: post #29
JohnScolaro wrote: post #10

Earlier quoted context omitted.

Thanks for taking the time to reply! I'm relatively new to working on this type of system (large scale, event driven) and half posted because I know there are people on HN way better than me at this, and was curious about their opinions. In the end, what's the difference between a log and a metric? Is one structured, and one unstructured? Is one a giant blob of text, and the other stored in a time series db? At the m…

The breakdown used by OTel isn’t all that bad: https://opentelemetry.io/docs/concepts/signals/ In essence: Logs mark some event in the system. Metrics model some measurable, quantifiable state. In high volume systems both can then be observed through various sampling techniques. A key item is that sampling is good to handle separately to application logic creating those signals as it may change over time or be dynami…

I think the common confusion boils down to:

> The moment of capturing a measurement is known as a metric event

Which suspiciously reads like a log.

In practice, a metric is an aggregate of events (the "metric events") when you're not interested in the individual event but, but in the aggregate itself. For practical reasons this is not implemented with logs but with more primitive technical events emission.

This is not fundamentally incompatible notions. If you do an electrocardiogram, you might be interested in your BPM, but it is deduced by the full log of each beat. The segregation we do in computing is more practical than fundamental.

Post reply on HN