Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

171–180 of 232 posts

Re: Logging sucks

#171

Post by hansvm »

Illniyar wrote: post #117

While I agree with some of it, I feel like there's a big gotcha here that isn't addressed. Having 1 single wide event, at the end of a request, means that if something unexpected happens in the middle (stack overflow, some bug that throws an error that bypasses your logging system, lambda times out etc...) you don't get any visibility into what happens. You also most likely lose out on a lot of logging frameworks you…

If that's an issue (visibility into middle layers) it just means your events aren't wide enough. There's no fundamental difference between log.error(data) and wide_event.attach(error, data), nor similar schemes using parameters rather that context/global-based state.

There are still use cases for one or the other strategy, but I'm not a fan of this explanation in either direction.

Re: Logging sucks

#172

Post by jspash »

simonw wrote: post #126

I hope registering an entire domain name for a blog post doesn't become a trend. I like linking to things that are likely to last a long time - a personal blog is one thing, but expecting people to keep paying the renewal fee every year for a single article feels less likely to me. A good alternative here is subdomains, since those don't have an additional annual fee. https://logging-sucks.boristane.com/ could work w…

The domain and article just a long-form advert for the author's observability sass. Oddly it's free, but you have to sign up for Cloudlfare, where the author is currently employed.

It's seems they are playing the long-long game!

All snarkiness aside, I learned a few things from the article. And I might even sign up to the sass since I already have a CF account.

Re: Logging sucks

#173

Post by wesammikhail »

hu3 wrote: post #166
wesammikhail wrote: post #125

Earlier quoted context omitted.

The way I have solved for this in my own framework in PHP is by having a Logging class with the following interface interface LoggerInterface { // calls $this->system(LEVEL_ERROR, ...); public function exception(Throwable $e): void; // Typical system logs public function system(string $level, string $message, ?string $category = null, mixed ...$extra): void; // User specific logs that can be seen in the user's "my hi…

Nice. I guess you write logs on the "final" block of a global try/catch/final? Something like: try { // handle request code } catch (...) { // add exceptions to log } final { // insert logs into DB }

I used to do it like that and it worked really well but I changed the flow to where exceptions are actually part of the control flow of the app using PHP's set_exception_handler(), set_error_handler() and register_shutdown_function().

Example, lets say a user forgot to provide a password when authenticating, then I will throw a ClientSideException(400, "need password yada yada");

That exception will bubble up to the exception_handler that logs and outputs the proper message to the screen. Similarly if ANY exception is thrown, regardless of where it originated, the same will happen.

When you embrace exceptions as control flow rather than try to avoid them, suddenly everything gets 10x easier and I end up writing much less code overall.

Re: Logging sucks

#174

Post by Trufa »

iLoveOncall wrote: post #112
bob1029 wrote: post #58

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. I worked with enterprise message bus loggers in semiconductor manufacturing context wherein we had thousands of participants on the message bus. It generated something like 300-400 megabytes per hour. Despite the insane volume we made this work really well using just grep and other basic CLI tools.…

> It generated something like 300-400 megabytes per hour. Despite the insane volume we made this work really well using just grep and other basic CLI tools. 400MB of logs an hour is nothing at all, that's why a naive grep can work. You don't even need to rotate your log files frequently in this situation.

Did you mean GB or TB otherwise you’re in the very low end of log volume

Re: Logging sucks

#175

Post by lanstin »

otterley wrote: post #22
cowsandmilk wrote: post #14

Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…

It’s an important architectural requirement for a production service to be able to scale out their log ingestion capabilities to meet demand. Besides, a little local on-disk buffering goes a long way, and is cheap to boot. It’s an antipattern to flush logs directly over the network.

And everything logging from the API to the network to the ingestion pipeline needs to be best effort - configure a capacity and ruthlessly drop msgs as needed, at all stages. Actually a nice case for UDP :)

Re: Logging sucks

#176

Post by ryeguy »

the_mitsuhiko wrote: post #25
heinrichhartman wrote: post #4

A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…

> A post on this topic feels incomplete without a shout-out to Charity Majors I concur. In fact, I strongly recommend anyone who has been working with observability tools or in the industry to read her blog, and the back story that lead to honeycomb. They were the first to recognize the value of this type of observability and have been a huge inspiration for many that came after.

Honeycomb is inspired by Facebook's Scuba (https://research.facebook.com/publications/scuba-diving-into...). The paper is from 2013, predating honeycomb. Charity worked there as well, but presumably was not part of the initial implementation given the timing.

Re: Logging sucks

#177

Post by hu3 »

cluckindan wrote: post #121
chickensong wrote: post #93

Earlier quoted context omitted.

You could have the log shipper filter events and create a separate audit stream with different behavior and destination.

Really, have sane log message types and include ”audit” as one of them. Log levels could be considered an anti-pattern.

I like this. But doesn't it make sense to categorize en Exception thrown as an erro somehow? And a new user registration as an email info?

Perhaps use tags then?

Re: Logging sucks

#178

Post by otterley »

m3047 wrote: post #170
Veserv wrote: post #61

Earlier quoted context omitted.

Saying they are all the same when no fidelity is lost is missing the point. The only distinction between logs, traces, and metrics is literally what to do when fidelity is lost. If you have insufficient ingestion rate: Logs are for events that can be independently sampled and be coherent. You can drop arbitrary logs to stay within ingestion rate. Traces are for correlated sequences of events where the entire sequence…

Good summary IMO. > You can drop arbitrary logs to stay within ingestion rate. Another way I've heard this framed in a production environments ingesting a firehose is: you can drop individual logging events because there will always be more.

It depends. Some cases like auditing require full fidelity. Others don’t. Plus, if you’re offering a logging service to a customer, the customer’s expectation is that once successfully ingested, your service doesn’t drop logs. If you’re violating that expectation, this needs to be clearly communicated to and assented by the customer.

The right way to think about logs, IMO, is less like diagnostic information and more like business records. If you change the framing of the problem, you might solve it in different way.

Re: Logging sucks

#179

Post by otterley »

lanstin wrote: post #175
otterley wrote: post #22

Earlier quoted context omitted.

It’s an important architectural requirement for a production service to be able to scale out their log ingestion capabilities to meet demand. Besides, a little local on-disk buffering goes a long way, and is cheap to boot. It’s an antipattern to flush logs directly over the network.

And everything logging from the API to the network to the ingestion pipeline needs to be best effort - configure a capacity and ruthlessly drop msgs as needed, at all stages. Actually a nice case for UDP :)

It depends. Some cases like auditing require full fidelity. Others don’t.

Plus, if you’re offering a logging service to a customer, the customer’s expectation is that once successfully ingested, your service doesn’t drop logs. If you’re violating that expectation, this needs to be clearly communicated to and assented by the customer.

Re: Logging sucks

#180

Post by manmal »

simonw wrote: post #129
dcminter wrote: post #44

Earlier quoted context omitted.

Could you drop a few specific posts here that you think are good for someone (me) who hasn't read her stuff before? Looks like there's a decade of stuff on her blog and I'm not sure I want to start at the very beginning...

A few of my favourites: - Software Sprawl, The Golden Path, and Scaling Teams With Agency: https://charity.wtf/2018/12/02/software-sprawl-the-golden-pa... - introduces the idea of the "golden path", where you tell engineers at your company that if they use the approved stack of e.g. PostgreSQL + Django + Redis then the ops team will support that for them, but if they want to go off path and use something like MongoDB…

The one on Generate AI seems a bit outdated. This was before Claude Code was released.
Post reply on HN