151–160 of 232 posts
This right here is the fundamental problem because the way it's done is highly inefficient, complex and I believe only exists so cloud providers can sell their expensive offerings.
A monolith is fine 90% of the time.
If you really want to get serious, you also want some kind of continuous profiling (like pyroscope) or at the very least some periodic thread dump collector (for serious degradation diagnostics once every couple of minutes is enough).
But logging is still a great tool.
"Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue." This right here is the fundamental problem because the way it's done is highly inefficient, complex and I believe only exists so cloud providers can sell their expensive offerings. A monolith is fine 90% of the time.
So many problems would be solved if service calls were IPC instead of network calls.
> Here's the mental model shift that changes everything: Instead of logging what your code is doing, log what happened to this request. Yeah that doesn't magically fix everything. Logging is still an arbitrary, clunky, unintuitive process that requires intentional design and extra systems to be useful. The "Wide Event log" example is 949 bytes, which isn't unmanageably large, but it is 3x larger than most log message…
I hope registering an entire domain name for a blog post doesn't become a trend. I like linking to things that are likely to last a long time - a personal blog is one thing, but expecting people to keep paying the renewal fee every year for a single article feels less likely to me. A good alternative here is subdomains, since those don't have an additional annual fee. https://logging-sucks.boristane.com/ could work w…
Earlier quoted context omitted.
> These do not have to be separate services, but they are separate enough to warrant it. All of this arises from your failure to question this basic assumption though, doesn't it?
> All of this arises from your failure to question this basic assumption though, doesn't it? Haha, no. "All of this" is a scenario I consider quite realistic in terms of what needs to happen. The question is, how should you split this up, if at all? Mind that these concerns will be involved in other ways with other requests, serving customers and internal users. There are enough different concerns at different levels…
Happy to discuss why you think this is all necessary. Open to questioning assumptions of my own too, if you have specifics.
As it is, you're just quoting microservices dogma. Your auth service doesn't need a different programming language from your invoicing system. Nor does it need to be scaled independently. Why would it?
The key insight that this article hints at but doesn't quite get too: you should treat your logs as a product whose customers are the rest of the devs in your company. The way you log things is intimately connected with what you want to do with them, and you need to build systems to generate useful insights from the log statements. In some cases it literally is part of the product: many of the machine learning systems that generate recommendations, search results, spam filtering, abuse detection, traffic direction, etc. are all based on the logs for the product, and you need to consider them as first-class citizens that you absolutely cannot break while adding new features. Logs are not just for debugging.
[1] https://absl.readthedocs.io/en/latest/absl.logging.html
[2] https://research.google/pubs/dapper-a-large-scale-distribute...
Earlier quoted context omitted.
On the other hand, investing in better tracing tools unlocks a whole nother level of logging and debugging capabilities that aren't feasible with just request logs. It's kind of like you mentioned with using the user id as a "trace" in your first message but on steroids.
These tools tend to be very expensive in my experience unless you are running your own monitoring cloud. Either you end up sampling traces at low rates to save on costs, or your observability bill is more than your infrastructure bill.
While I agree with some of it, I feel like there's a big gotcha here that isn't addressed. Having 1 single wide event, at the end of a request, means that if something unexpected happens in the middle (stack overflow, some bug that throws an error that bypasses your logging system, lambda times out etc...) you don't get any visibility into what happens. You also most likely lose out on a lot of logging frameworks you…