Logging sucks • Hacker News Board

Re: Logging sucks

#181

Post by aidenn0 » Sun, Dec 21, 2025, 11:50 PM UTC

I prefer narrow entries with an "event ID" attached; you can trivially filter to get all of your items with the same event ID, but it works better with all the tools designed for narrow entries.

So instead of one entry with e.g. 27 elements, you have maybe a dozen entries, all with the same event id and 2-4 elements (including the event ID) each. This also lets you log incrementally; if your server crashes before you log the wide entry you have zero data.

It also lets you adjust the granularity. E.g. for a web service you might have both a session and a request ID.

Re: Logging sucks

#182

Post by Dylan16807 » Sun, Dec 21, 2025, 11:59 PM UTC

0xbadcafebee wrote: post #148

bccdee wrote: post #142

Earlier quoted context omitted.

Why? "This was written badly" is a perfectly normal thing to say; "this was written badly because you didn't put in the effort of writing it yourself" doubly so.

Say they used AI to write it, it came out bad, and they published it anyway. They had the opportunity to "make it better" before publishing, but didn't. The only conclusion for this is, they just aren't good at writing. So whether AI is used or not, it'll suck either way. So there's no need to complain about the AI. It's like complaining that somebody typed a crappy letter rather than hand-wrote it. Either way the le…

> The only conclusion for this is, they just aren't good at writing.

Not true. It's likely an effort issue in that situation.

And that kind of effort issue is good to call out, because it compounds the low quality.

Re: Logging sucks

#183

Post by 46Bit » Mon, Dec 22, 2025, 12:00 AM UTC

cowsandmilk wrote: post #14

Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…

What we're doing at Cloudflare (including some of what the author works on) samples adaptively. Each log batch is bucketed based on a few fields, and in each bucket if there's lots of logs in each bucket we only keep the sqrt or log of the number of input logs. It works really well... but part of why it works well is we always have blistering rates of logs, so can cope with spikes in event rates without the sampling system itself getting overwhelmed.

Re: Logging sucks

#184

Post by alexwennerberg » Mon, Dec 22, 2025, 12:13 AM UTC

dxdm wrote: post #100

alexwennerberg wrote: post #5

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue. Your logs are still acting like it's 2005. If a user request is hitting that many things, in my view, that is a deeply broken architecture.

> If a user request is hitting that many things, in my view, that is a deeply broken architecture. Things can add up quickly. I wouldn't be surprised if some requests touch a lot of bases. Here's an example: a user wants to start renting a bike from your public bike sharing service, using the app on their phone. This could be an app developed by the bike sharing company itself, or a 3rd party app that bundles mobilit…

This was an excellent explanation of a complex business problem, which would be made far more complex by splitting these out into separate services. Every single 'if' branch you describe could either be a line of code, or a service boundary, which has all the complexity you describe, in addition to the added complexity of:

a. managing an external API+schema for each service

b. managing changes to each service, for example, smooth rollout of a change that impacts behavior across two services

c. error handling on the client side

d. error handling on the server side

e. added latency+compute because a step is crossing a network, being serialized/de-serialized on both ends

f. presuming the services use different databases, performance is now completely shot if you have a new business problem that crosses service boundaries. In practice, this will mean doing a "join" by making some API call to one service and then another API call to another service

In your description of the problem, there is nothing that I would want to split out into a separate service. And to get back to the original problem, it makes it far easier to get all the logging context for a single problem in a single place (attach a request ID to the all logs and see immediately everything that happened as part of that request)

Re: Logging sucks

#185

Post by lll-o-lll » Mon, Dec 22, 2025, 12:14 AM UTC

otterley wrote: post #76

lll-o-lll wrote: post #69

Earlier quoted context omitted.

Auditing is fundamentally different because it has different durability and consistency requirements. I can buffer my logs, but I might need to transact my audit.

For most cases, buffering audit logs on local storage is fine. What matters is that the data is available and durable somewhere in the path, not that it be transactionally durable at the final endpoint.

What are we defining as “audit” here? My experience is with regulatory requirements, and “durable” on local storage isn’t enough.

In practice, the audit isn’t really a log, it’s something more akin to database record. The point is that you can’t filter your log stream for audit requirements.

Re: Logging sucks

#186

Post by otterley » Mon, Dec 22, 2025, 12:20 AM UTC

lll-o-lll wrote: post #185

otterley wrote: post #76

Earlier quoted context omitted.

For most cases, buffering audit logs on local storage is fine. What matters is that the data is available and durable somewhere in the path, not that it be transactionally durable at the final endpoint.

What are we defining as “audit” here? My experience is with regulatory requirements, and “durable” on local storage isn’t enough. In practice, the audit isn’t really a log, it’s something more akin to database record. The point is that you can’t filter your log stream for audit requirements.

Take Linux kernel audit logs as an example. So long as they can be persisted to local storage successfully, they are considered durable. That’s been the case since the audit subsystem was first created. In fact, you can configure the kernel to panic as soon as records can no longer be recorded.

Regulators have never dictated where auditable logs must live. Their requirement is that the records in scope are accurate (which implies tamper proof) and that they are accessible. Provided those requirements are met, where the records can be found is irrelevant. It thus follows that if all logs over the union of centralized storage and endpoint storage meet the above requirements then it will satisfy the regulator.

Re: Logging sucks

#187

Post by yallpendantools » Mon, Dec 22, 2025, 12:22 AM UTC

hansvm wrote: post #171

Illniyar wrote: post #117

While I agree with some of it, I feel like there's a big gotcha here that isn't addressed. Having 1 single wide event, at the end of a request, means that if something unexpected happens in the middle (stack overflow, some bug that throws an error that bypasses your logging system, lambda times out etc...) you don't get any visibility into what happens. You also most likely lose out on a lot of logging frameworks you…

If that's an issue (visibility into middle layers) it just means your events aren't wide enough. There's no fundamental difference between log.error(data) and wide_event.attach(error, data), nor similar schemes using parameters rather that context/global-based state. There are still use cases for one or the other strategy, but I'm not a fan of this explanation in either direction.

> If that's an issue (visibility into middle layers) it just means your events aren't wide enough.

I hate this kind of No-True-Scotsman handwaves for how a certain approach is supposed to solve my problems. "If brute-force search is not solving all your problems, it just means your EC2 servers are not beefy enough."

I gotta admit, I don't quite "get" TFA's point and the one issue that jumped out at me while reading it and your comment is that sooner than later your wide events just become fat, supposedly-still-human-readable JSON dumps.

I think a machine-parseable log-line format is still better than wide events, each line hopefully correlated with a request id though in practice I find that user id + time correlation isn't that bad either.

>> [TFA] Wide Event: A single, context-rich log event emitted per request per service. Instead of 13 log lines for one request, you emit 1 line with 50+ fields containing everything you might need to debug.

I am not convinced this is supposed to help the poor soul who has to debug an incident at 2AM. Take for example a function that has to watch out for a special kind of user (`isUserFoo`) where "special kind" is defined as a metric on five user attributes. I.e.,

    lambda isUserFoo(u): u.isA && (u.isB || u.isC) && (u.isD || u.isE)

With usual logging I might find

     :  : {"level": "INFO", "requestID": "xxxaaa", "msg": "user is foo"}

Which immediately tells me that foo-ness is something I might want to pay attention to in this context.

With wide events, as I understand it, either you log the user in the wide event dump with attributes A to E (and potentially more!) or coalesce these into a boolean field `isUserFoo`. None of which tells me that foo-ness might be something that might be relevant in this context.

Multiply that with all the possible special-cases any logging unit might have to deal with. There's bar-ness which is also dependent on attributes A-E but with different logical connectives. There's baz-ness which is `isUserFoo(u) XOR (217828 < u.zCount < 3141592)`. The wide event is soooo context-rich I'm drowning.

Re: Logging sucks

#188

Post by minitech » Mon, Dec 22, 2025, 12:55 AM UTC

0xbadcafebee wrote: post #148

bccdee wrote: post #142

Earlier quoted context omitted.

Why? "This was written badly" is a perfectly normal thing to say; "this was written badly because you didn't put in the effort of writing it yourself" doubly so.

Say they used AI to write it, it came out bad, and they published it anyway. They had the opportunity to "make it better" before publishing, but didn't. The only conclusion for this is, they just aren't good at writing. So whether AI is used or not, it'll suck either way. So there's no need to complain about the AI. It's like complaining that somebody typed a crappy letter rather than hand-wrote it. Either way the le…

Compared to human bad writing, AI writing tends to suck more verbosely and in exciting new ways (e.g. by introducing factual errors).

Re: Logging sucks

#189

Post by hoppp » Mon, Dec 22, 2025, 1:12 AM UTC

sgarland wrote: post #153

hoppp wrote: post #151

"Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue." This right here is the fundamental problem because the way it's done is highly inefficient, complex and I believe only exists so cloud providers can sell their expensive offerings. A monolith is fine 90% of the time.

That, and everyone thinks they have to do things this way, so it’s a terrible cycle. So many problems would be solved if service calls were IPC instead of network calls.

Ye, even the smallest projects nowadays are either serverless or run in kubernetes, they never even think about IPC.

Modern deployment pipelines are very good at deploying services that are running in separate containers, and people hate custom shell scripts, even if they work great.

If it's bespoke and not standardized its harder to maintain that's true, but its also great at reducing operational costs because infrastructure fees change with less reliance on external services . Its a cycle alright.

Re: Logging sucks

#190

Post by hu3 » Mon, Dec 22, 2025, 1:14 AM UTC

wesammikhail wrote: post #173

hu3 wrote: post #166

Earlier quoted context omitted.

Nice. I guess you write logs on the "final" block of a global try/catch/final? Something like: try { // handle request code } catch (...) { // add exceptions to log } final { // insert logs into DB }

I used to do it like that and it worked really well but I changed the flow to where exceptions are actually part of the control flow of the app using PHP's set_exception_handler(), set_error_handler() and register_shutdown_function(). Example, lets say a user forgot to provide a password when authenticating, then I will throw a ClientSideException(400, "need password yada yada"); That exception will bubble up to the…

I love Exceptions as control flow! Thanks for the suggestion.

I too use specialized exceptions. Some have friendly messages that can be displayed to the user, like "Please fill the password". But critical exceptions display a generic error to the user ("Ops, sorry something went wrong on our side...") but log specifics to devs, like database connection errors, for example.