Live data from Hacker News

Logging sucks

loggingsucks.com

161–170 of 232 posts

Re: Logging sucks

#161

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

If your codebase has the concept of a request ID, you could also feasibly use that to trace what a user has been doing with more specificity.

If you care about this more than anything else (e.g. if you care about audits a LOT and need them perfect), you can simply code the app via action paths, rather than for modularity. It makes changes harder down the road, but for codebases that don’t change much, this can be a viable tradeoff to significantly improve tracing and logging.

Re: Logging sucks

#162
post #126

I hope registering an entire domain name for a blog post doesn't become a trend. I like linking to things that are likely to last a long time - a personal blog is one thing, but expecting people to keep paying the renewal fee every year for a single article feels less likely to me. A good alternative here is subdomains, since those don't have an additional annual fee. https://logging-sucks.boristane.com/ could work w…

I kind of agree, but the message in this particular post does border on https://simonwillison.net/2024/Jul/13/give-people-something-...

Maybe I should have written "Give people something to link to that they can expect to stick around for a very long time"

Re: Logging sucks

#163
post #88

A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…

Nick Blumhardt for a while longer than that as "structured logging". Seq and Serilog as enabling software and library in the .net ecosystem.

The article emphasizes that their recommendation is different from structured logging.

Re: Logging sucks

#164
post #126

I hope registering an entire domain name for a blog post doesn't become a trend. I like linking to things that are likely to last a long time - a personal blog is one thing, but expecting people to keep paying the renewal fee every year for a single article feels less likely to me. A good alternative here is subdomains, since those don't have an additional annual fee. https://logging-sucks.boristane.com/ could work w…

Sorry, this is not a "blog post" - it's far closer to digital marketing. A lead attractor as the author is trying to sell a service very clearly by the end of his page (no disrespect meant to either).

Re: Logging sucks

#165
post #83
post #72

Earlier quoted context omitted.

On the other hand, investing in better tracing tools unlocks a whole nother level of logging and debugging capabilities that aren't feasible with just request logs. It's kind of like you mentioned with using the user id as a "trace" in your first message but on steroids.

These tools tend to be very expensive in my experience unless you are running your own monitoring cloud. Either you end up sampling traces at low rates to save on costs, or your observability bill is more than your infrastructure bill.

I've tried HyperDX and SigNoz, they seem easy to self-host and decent enough

Re: Logging sucks

#166

While I agree with some of it, I feel like there's a big gotcha here that isn't addressed. Having 1 single wide event, at the end of a request, means that if something unexpected happens in the middle (stack overflow, some bug that throws an error that bypasses your logging system, lambda times out etc...) you don't get any visibility into what happens. You also most likely lose out on a lot of logging frameworks you…

The way I have solved for this in my own framework in PHP is by having a Logging class with the following interface interface LoggerInterface { // calls $this->system(LEVEL_ERROR, ...); public function exception(Throwable $e): void; // Typical system logs public function system(string $level, string $message, ?string $category = null, mixed ...$extra): void; // User specific logs that can be seen in the user's "my hi…

Nice. I guess you write logs on the "final" block of a global try/catch/final?

Something like:

  try {
    // handle request code
  } catch (...) {
    // add exceptions to log
  } final {
    // insert logs into DB
  }

Re: Logging sucks

#167
You get: "Show me all checkout failures for premium users in the last hour where the new checkout flow was enabled, grouped by error code."

I can do this with axiom already and I’ve never worried about how to log.

Re: Logging sucks

#168

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

> That was difficult to read, smelt very AI assisted though the message was worthwhile... It won’t be long before ad computem comments like this are frowned upon.

I felt unsure whether to include that particular comment, but landed on including because I think it's a real danger. I've got no problem with people using AI and do use it for some things myself.

However I don't think you should outsource understanding to LLMs, and also think that shifting the effort from the writer to the reader is a poor strategy (and disrespectful to the reader)

edit: in case it's unclear I'm not accusing the author of having outsourced their understanding to AI, but I think it's a real risk that people can fall into, the value is in the thinking people put into things not the mechanics of typing it out

Re: Logging sucks

#169
post #129

Earlier quoted context omitted.

Could you drop a few specific posts here that you think are good for someone (me) who hasn't read her stuff before? Looks like there's a decade of stuff on her blog and I'm not sure I want to start at the very beginning...

A few of my favourites: - Software Sprawl, The Golden Path, and Scaling Teams With Agency: https://charity.wtf/2018/12/02/software-sprawl-the-golden-pa... - introduces the idea of the "golden path", where you tell engineers at your company that if they use the approved stack of e.g. PostgreSQL + Django + Redis then the ops team will support that for them, but if they want to go off path and use something like MongoDB…

Terrific, thank you.

Re: Logging sucks

#170
post #61

Earlier quoted context omitted.

> Logging is not metrics is not auditing. I espouse a "grand theory of observability" that, like matter and energy, treats logs, metrics, and audits alike. At the end of the day, they're streams of bits, and so long as no fidelity is lost, they can be converted between each other. Audit trails are certainly carried over logs. Metrics are streams of time-series numeric data; they can be carried over log channels or em…

Saying they are all the same when no fidelity is lost is missing the point. The only distinction between logs, traces, and metrics is literally what to do when fidelity is lost. If you have insufficient ingestion rate: Logs are for events that can be independently sampled and be coherent. You can drop arbitrary logs to stay within ingestion rate. Traces are for correlated sequences of events where the entire sequence…

Good summary IMO.

> You can drop arbitrary logs to stay within ingestion rate.

Another way I've heard this framed in a production environments ingesting a firehose is: you can drop individual logging events because there will always be more.

Post reply on HN