141–150 of 174 posts
Now all we need is the 1,000,000 other components in the multiple ecosystems to log in the same format and I won't have a perpetual headache. Good job Go though for being opinionated but rational.
[1] https://www.elastic.co/guide/en/ecs-logging/overview/current... [2] https://opentelemetry.io/docs/specs/otel/logs/data-model/
Earlier quoted context omitted.
Its flaws or merits aside, when you have no other useful context to add to the error, that's precisely what Errorf is for. func bar() error { err := baz.Transmogrify() return fmt.Errorf("transmogrify: %w", err) } func foo() error { err := bar() return fmt.Errorf("bar: %w", err) } func main() { err := foo() fmt.Printf("foo: %v", err) // foo: bar: transmogrify: not found } There's your callstack, without the cost of ca…
Not exactly -- you should only fmt.Errorf wrap errors which are non-nil. See my sibling comment: https://news.ycombinator.com/item?id=37234455
Earlier quoted context omitted.
In the last few months I've realized what I desperately need: a way to wrap an error with a call stack at the point where it enters our code base. This would probably save me on average 20-30 minutes a week. I see this all the time: main.go:141 error: could not transmogrify the thing: a144cd21c48 And then I literally grep the code base to find the error message. That works ~50% of the time, but the other 50%, I see t…
This is such an infuriating problem. I'm convinced I'm using Go wrong, because I simply can't understand how this doesn't make it a toy language. Why the $expletive am I wasting 20-30 and more minutes per week of my life looking for the source of an error!? Have you seen https://github.com/tomarrell/wrapcheck ? It's a linter than does a fairly good job of warning when an error originates from an external package but…
Ignore the word error for a moment. Think about how you program in the general case, for a hypothetical type T. What is it that you do to to your T values to ensure that you don't have the same problem?
Now do that same thing when T is of the type error. There is nothing special about errors.
Oof. We just converted all of our logging to zap[0] to get structured JSON logging for downstream parsing. Wonder how the perf stacks up. [0]: https://github.com/uber-go/zap
It looks like they've included slog in their performance benchmarks, which show zap as considerably more performant (though I don't really understand the benchmark).
What it comes down to is that zap special cases things like slice-of-int, slice-of-string, slice-of-timestamp, slog doesn't, and the benchmark includes all those special cases. I question whether your typical log statement includes slices. A more fair benchmark would be just scalar types, and zap & slog optimizations there look pretty similar.
https://github.com/uber-go/zap/blob/fd37f1f613a87773fc30f719...
https://github.com/uber-go/zap/blob/fd37f1f613a87773fc30f719...
Earlier quoted context omitted.
A better interface/API is really what I meant. The performance characteristics are probably worth the tradeoff.
It's most definitely not. Logging is crucial for monitoring services, and making logging statements many times slower will either sink your service or push developers to avoid logging and making the service impossible to debug. Most log metadata will be attached by libraries and middleware, so service/application devs won't even see most of it.
Earlier quoted context omitted.
Rust-style error handling works better though - similar to Go, but with the addition of enums such that the precise types of errors which may be encountered can be easily documented in the type system.
Agreed, I’d love if they would take inspiration from Rust and bring that to Go.
The library ecosystem is already excellent, and the tooling is good, lack of sum types is the single wart that makes me regret it every time I pick Go up for a project.
Earlier quoted context omitted.
Indeed, our code base is littered with fmt.Errorf("...: %w", err), but that only works if enough places in the code add context. Currently only about 15% of return sites do this. And I disagree that the cost of carrying around the callstack is something to worry about. Errors are akin to exceptions in C++/Java: no happy path should rely on errors for control flow (except io.EOF, but that won't generate a call stack).…
Every error should be annotated at the call site. fmt.Errorf("...: %w", err) isn't litter, it should be a basic expectation of any code which passes code review. > Errors are akin to exceptions in C++/Java: no happy path should rely on errors for control flow (except io.EOF, but that won't generate a call stack). They should be rare enough that any cost below about 1ms and 10k is negligible. This may be true in C++ o…
Most Go programmers are too scared to panic and abort when invariants are violated. I think most codebases contain at least 2x as much error handling as is really necessary.
Earlier quoted context omitted.
lnav is neat but doesn't really do structured logging AFAICT, just predetermined fields for the format.
lnav has support for JSON-lines, logfmt, as well as the Bro and W3C Extended Log File formats that are XSV and self-describing. The contents are also accessible through SQLite tables. Is there some gap here that you're thinking of?
As far as I can tell, the lnav feature you describing allows you to define a JSON log format with predefined fields. But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output or filter on it in the UI and no ability to see it in the SQLite virtual table. That's why I say lnav doesn't appear to support structured logs.
edit: oh, I missed `"hide-extra": false`! That significantly improves things! Still, I don't see a way to access it from the SQLite virtual table. I also don't see something else I might want: a way to have a different "view" for certain log messages; maybe to switch between filtering/viewing particular ones, maybe to just have line-format be conditional based on the detected format. (I guess I can sort of do this based on `module-field`? but I might want it lighter-weight/finer-grained than that.)
Earlier quoted context omitted.
I can see the case for flat logging, aka. key value logging, as an optimization for very, very performance critical code that needs to emit string logs. That however, isn’t mainstream in my experience. The far, far more common case is logging in code with complex data-driven behaviors where the data is structured (more than one level, not flat) and where forensic debugging via logs is a critical activity. If that’s n…
I think you seem to be arguing that the end result should be a "map"-like structure, whereas the other commenter is arguing about the interface to the logging library not being based on maps. These are not the same and taking maps in the interface is likely to incur allocations, yes. Having to specify your key-value pairs without maps is the only downside to not taking fully constructed maps in the interface.
Just my opinions here. I don’t question the value of slog as it sits, just could have been better for the community at large is all.
Earlier quoted context omitted.
lnav has support for JSON-lines, logfmt, as well as the Bro and W3C Extended Log File formats that are XSV and self-describing. The contents are also accessible through SQLite tables. Is there some gap here that you're thinking of?
The idea of structured logs is that every place in the code where you define a log message, you can throw in extra attributes (key/value) pairs. As far as I can tell, the lnav feature you describing allows you to define a JSON log format with predefined fields. But if some log message uses an attribute you haven't anticipated in the format definition, there's no way to see it in the pretty-printed output or filter on…
Properties in the log message that are not in the "line-format" are displayed below the message as key-value pairs. As an example, the bunyan[1] log format[2] has a few standard properties that are used in the "line-format" to form the main message. Then, as shown in this test[3] for this log[4], the remaining properties (even ones not mentioned in the format file) are shown underneath.
> or filter on it in the UI
Since they are part of the message as mentioned above, they can be filtered on.
> and no ability to see it in the SQLite virtual table.
The "log_raw_text" column in the table can be used to access the original log message from the file. So, you can use the JSON functions in SQLite to retrieve the value:
;SELECT log_raw_text ->> '$.repository' from bunyan
[2] - https://github.com/tstack/lnav/blob/master/src/formats/bunya...
[3] - https://github.com/tstack/lnav/blob/master/test/expected/tes...
[4] - https://github.com/tstack/lnav/blob/master/test/logfile_buny...