That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…
If your codebase has the concept of a request ID, you could also feasibly use that to trace what a user has been doing with more specificity.
Logging sucks
161–170 of 232 posts
Re: Logging sucks
#162I hope registering an entire domain name for a blog post doesn't become a trend. I like linking to things that are likely to last a long time - a personal blog is one thing, but expecting people to keep paying the renewal fee every year for a single article feels less likely to me. A good alternative here is subdomains, since those don't have an additional annual fee. https://logging-sucks.boristane.com/ could work w…
I kind of agree, but the message in this particular post does border on https://simonwillison.net/2024/Jul/13/give-people-something-...
Re: Logging sucks
#163A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…
Nick Blumhardt for a while longer than that as "structured logging". Seq and Serilog as enabling software and library in the .net ecosystem.
Re: Logging sucks
#164I hope registering an entire domain name for a blog post doesn't become a trend. I like linking to things that are likely to last a long time - a personal blog is one thing, but expecting people to keep paying the renewal fee every year for a single article feels less likely to me. A good alternative here is subdomains, since those don't have an additional annual fee. https://logging-sucks.boristane.com/ could work w…
Re: Logging sucks
#165Earlier quoted context omitted.
On the other hand, investing in better tracing tools unlocks a whole nother level of logging and debugging capabilities that aren't feasible with just request logs. It's kind of like you mentioned with using the user id as a "trace" in your first message but on steroids.
These tools tend to be very expensive in my experience unless you are running your own monitoring cloud. Either you end up sampling traces at low rates to save on costs, or your observability bill is more than your infrastructure bill.
Re: Logging sucks
#166While I agree with some of it, I feel like there's a big gotcha here that isn't addressed. Having 1 single wide event, at the end of a request, means that if something unexpected happens in the middle (stack overflow, some bug that throws an error that bypasses your logging system, lambda times out etc...) you don't get any visibility into what happens. You also most likely lose out on a lot of logging frameworks you…
The way I have solved for this in my own framework in PHP is by having a Logging class with the following interface interface LoggerInterface { // calls $this->system(LEVEL_ERROR, ...); public function exception(Throwable $e): void; // Typical system logs public function system(string $level, string $message, ?string $category = null, mixed ...$extra): void; // User specific logs that can be seen in the user's "my hi…
Something like:
try {
// handle request code
} catch (...) {
// add exceptions to log
} final {
// insert logs into DB
}Re: Logging sucks
#167I can do this with axiom already and I’ve never worried about how to log.
Re: Logging sucks
#168That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…
> That was difficult to read, smelt very AI assisted though the message was worthwhile... It won’t be long before ad computem comments like this are frowned upon.
However I don't think you should outsource understanding to LLMs, and also think that shifting the effort from the writer to the reader is a poor strategy (and disrespectful to the reader)
edit: in case it's unclear I'm not accusing the author of having outsourced their understanding to AI, but I think it's a real risk that people can fall into, the value is in the thinking people put into things not the mechanics of typing it out
Re: Logging sucks
#169Earlier quoted context omitted.
Could you drop a few specific posts here that you think are good for someone (me) who hasn't read her stuff before? Looks like there's a decade of stuff on her blog and I'm not sure I want to start at the very beginning...
A few of my favourites: - Software Sprawl, The Golden Path, and Scaling Teams With Agency: https://charity.wtf/2018/12/02/software-sprawl-the-golden-pa... - introduces the idea of the "golden path", where you tell engineers at your company that if they use the approved stack of e.g. PostgreSQL + Django + Redis then the ops team will support that for them, but if they want to go off path and use something like MongoDB…
Re: Logging sucks
#170Earlier quoted context omitted.
> Logging is not metrics is not auditing. I espouse a "grand theory of observability" that, like matter and energy, treats logs, metrics, and audits alike. At the end of the day, they're streams of bits, and so long as no fidelity is lost, they can be converted between each other. Audit trails are certainly carried over logs. Metrics are streams of time-series numeric data; they can be carried over log channels or em…
Saying they are all the same when no fidelity is lost is missing the point. The only distinction between logs, traces, and metrics is literally what to do when fidelity is lost. If you have insufficient ingestion rate: Logs are for events that can be independently sampled and be coherent. You can drop arbitrary logs to stay within ingestion rate. Traces are for correlated sequences of events where the entire sequence…
> You can drop arbitrary logs to stay within ingestion rate.
Another way I've heard this framed in a production environments ingesting a firehose is: you can drop individual logging events because there will always be more.