Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

61–70 of 232 posts

Re: Logging sucks

#61

Post by Veserv »

otterley wrote: post #39
m3047 wrote: post #34

I agree with this statement: "Instead of logging what your code is doing, log what happened to this request." but the impression I can't shake is that this person lacks experience, or more likely has a lot of experience doing the same thing over and over. "Bug parts" (as in "acceptable number of bug parts per candy bar") logging should include the precursors of processing metrics. I think what he calls "wide events"…

> Logging is not metrics is not auditing. I espouse a "grand theory of observability" that, like matter and energy, treats logs, metrics, and audits alike. At the end of the day, they're streams of bits, and so long as no fidelity is lost, they can be converted between each other. Audit trails are certainly carried over logs. Metrics are streams of time-series numeric data; they can be carried over log channels or em…

Saying they are all the same when no fidelity is lost is missing the point. The only distinction between logs, traces, and metrics is literally what to do when fidelity is lost.

If you have insufficient ingestion rate:

Logs are for events that can be independently sampled and be coherent. You can drop arbitrary logs to stay within ingestion rate.

Traces are for correlated sequences of events where the entire sequence needs to be retained to be useful/coherent. You can drop arbitrary whole sequences to stay within ingestion rate.

Metrics are pre-aggregated collections of events. You pre-limited your emission rate to fit your ingestion rate at the cost of upfront loss of fidelity.

If you have adequate ingestion rate, then you just emit your events bare and post-process/visualize your events however you want.

Re: Logging sucks

#62

Post by spike021 »

mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

If your codebase has the concept of a request ID, you could also feasibly use that to trace what a user has been doing with more specificity.

Re: Logging sucks

#63

Post by otterley »

Veserv wrote: post #57
cowsandmilk wrote: post #14

Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…

I do not see how logging could bottleneck you in a degraded state unless your logging is terribly inefficient. A properly designed logging system can record on the order of 100 million logs per second per core. Are you actually contemplating handling 10 million requests per second per core that are failing?

Generation and publication is just the beginning (never mind the fact that resources consumed by an application to log something are no longer available to do real work). You have to consider the scalability of each component in the logging architecture from end to end. There's ingestion, parsing, transformation, aggregation, derivation, indexing, and storage. Each one of those needs to scale to meet demand.

Re: Logging sucks

#64

Post by ivan_gammel »

otterley wrote: post #38
zkmon wrote: post #10

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue. Your logs are still acting like it's 2005. Logs are fine. The job of local logs is to record the talk of a local process. They are doing this fine. Local logs were never meant to give you a picture of…

One of the points the author is trying to make (although he doesn't make it well, and his attitude makes it hard to read) is that logs aren't just for root-causing incidents. When properly seasoned with context, logs give you useful information like who is impacted (not every incident impacts every customer the same way), correlations between component performance and inputs, and so forth. When connected to analytica…

It’s a wild violation of SRP to suggest that. Separating concerns is way more efficient. Database can handle audit trail and some key metrics much better, no special tools needed, you can join transaction log with domain tables as a bonus.

Re: Logging sucks

#65

Post by otterley »

ivan_gammel wrote: post #64
otterley wrote: post #38

Earlier quoted context omitted.

One of the points the author is trying to make (although he doesn't make it well, and his attitude makes it hard to read) is that logs aren't just for root-causing incidents. When properly seasoned with context, logs give you useful information like who is impacted (not every incident impacts every customer the same way), correlations between component performance and inputs, and so forth. When connected to analytica…

It’s a wild violation of SRP to suggest that. Separating concerns is way more efficient. Database can handle audit trail and some key metrics much better, no special tools needed, you can join transaction log with domain tables as a bonus.

Are you assuming they're all stored identically? If so, that's not necessarily the case.

Once the logs have entered the ingestion endpoint, they can take the most optimal path for their use case. Metrics can be extracted and sent off to a time-series metric database, while logs can be multiplexed to different destinations, including stored raw in cheap archival storage, or matched to schemas, indexed, stored in purpose-built search engines like OpenSearch, and stored "cooked" in Apache Iceberg+Parquet tables for rapid querying with Spark, Trino, or other analytical engines.

Have you ever taken, say, VPC flow logs, saved them in Parquet format, and queried them with DuckDB? I just experimented with this the other day and it was mind-blowingly awesome--and fast. I, for one, am glad the days of writing parsers and report generators myself are over.

Re: Logging sucks

#67

Post by mnahkies »

spike021 wrote: post #62
mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

If your codebase has the concept of a request ID, you could also feasibly use that to trace what a user has been doing with more specificity.

We do have both a span id and trace id - but I personally find this more cumbersome over filtering on a user id. YMMV if you're interested in a single trace then you'd filter for that, but I find you often also care what happened "around" a trace

Re: Logging sucks

#68

Post by ivan_gammel »

spike021 wrote: post #62
mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

If your codebase has the concept of a request ID, you could also feasibly use that to trace what a user has been doing with more specificity.

…and the same ID can be displayed to user on HTTP 500 with the support contact, making life of everyone much easier.

Re: Logging sucks

#69

Post by lll-o-lll »

otterley wrote: post #39
m3047 wrote: post #34

I agree with this statement: "Instead of logging what your code is doing, log what happened to this request." but the impression I can't shake is that this person lacks experience, or more likely has a lot of experience doing the same thing over and over. "Bug parts" (as in "acceptable number of bug parts per candy bar") logging should include the precursors of processing metrics. I think what he calls "wide events"…

> Logging is not metrics is not auditing. I espouse a "grand theory of observability" that, like matter and energy, treats logs, metrics, and audits alike. At the end of the day, they're streams of bits, and so long as no fidelity is lost, they can be converted between each other. Audit trails are certainly carried over logs. Metrics are streams of time-series numeric data; they can be carried over log channels or em…

Auditing is fundamentally different because it has different durability and consistency requirements. I can buffer my logs, but I might need to transact my audit.

Re: Logging sucks

#70

Post by exabrial »

Our logging guidance is: "Don't write comments, write logs" and that serves us pretty well. The point being, don't write code "clever code", write obvious code, and try to make it similar to everything else thats been done, regardless if you agree with it.
Post reply on HN