100% agree. It should be an optional argument for your logging library and handled one time there to be used everywhere.
Log by time, not by count
91–100 of 109 posts
Re: Log by time, not by count
#92Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…
log all major logical branches within code (if/for)
Re: Log by time, not by count
#93I agree with this. Logging, as well as metrics and tracing, are such hard topics for me to wrap my head around though. From the log consumer (person) perspective, you'd want logs to provide you with sufficient information when troubleshooting. But since trouble usually happens when things go wrong in unexpected ways, the logging likely won't be well aligned to emit the right info for you to figure out what's going wr…
Everything is events. The problem is that, as you notice, you frequently encounter situations where there are too many events to handle. Metrics, logging, and tracing are just three different ways to handle that problem. Metrics handles too many events by aggregating them. You handle too many events by squashing them into a smaller number of events that aggregate the information. Logging handles too many events by sa…
Yup.
Somewhat formalized: https://peter.bourgon.org/blog/2018/08/22/observability-sign...
Re: Log by time, not by count
#94Best advice I ever got on logging: log all major logical branches within code (if/for) if "request" span multiple machine in cloud infrastructure, include request ID in all so logs can be grouped if possible make log level dynamically controlled, so grug can turn on/off when need debug issue (many!) if possible make log level per user, so can debug specific user issue - https://grugbrain.dev/ The only one I'll add is…
log all major logical branches within code (if/for) This certainly does not work for any non-trivial amount of load...
My systems scale to somewhere around 200,000+ machines at peak and seem to do fine.
Re: Log by time, not by count
#95One way to reframe this is: "as a user [of the logs], what might I want to know?" In my experience, this post is often right (and the logs are often wrong). There's a tendency to either log too much or log too little - if only a few items are getting processed, it's fine and maybe even good to log all 7 of them. But if many, many are getting processed - you'll experience semantic overload as a reader of the logs. Wha…
The "compressed form" of logs you're describing here is really just metrics...
Re: Log by time, not by count
#96Earlier quoted context omitted.
log all major logical branches within code (if/for) This certainly does not work for any non-trivial amount of load...
My systems scale to somewhere around 200,000+ machines at peak and seem to do fine.
How many customers is that serving, 20 per machine? (Or whatever a customer might be in that case)
Re: Log by time, not by count
#97There is an additional benefit to throttling by time, it is a lot easier to do it efficiently in multithreaded environments. If you log by count, you need a global counter for that event (you could do thread-local, but then your logging volume would depend on the number of threads). If the code path is hot (which may be the case if you want to throttle your logs) multiple threads will contend on the increment, and th…
For high-concurrency scenarios, sharded counters (per-thread counters with occasional global sync) or probabilistic logging (log with probability 1/N) can also solve the contention issue while maintaining count-based semantics. These approaches give you deterministic volume without the CAS overhead when timers expire.
Re: Log by time, not by count
#98Putting a few important metrics in the logs every 10s is something that the Aerospike datastore also does (https://aerospike.com/docs/database/observe/latency). This is useful because when you contact support, they run a script to generate a table of historical latencies from the log without depending on you having set up Prometheus, CloudWatch, etc.
Re: Log by time, not by count
#99> Log rate should be consistent If you want to know if an application is running, implement health checks. I hope I never have to deal with the pattern suggested in this article in a production system.
Yup, he's trying to solve monitoring and logging with the same setup. I don't think I'd appreciate it, but like all the IT horror stories can be probably made to work.
Re: Log by time, not by count
#100Earlier quoted context omitted.
log all major logical branches within code (if/for) This certainly does not work for any non-trivial amount of load...
My systems scale to somewhere around 200,000+ machines at peak and seem to do fine.
It's not a question of number of machines, it's a question of how much load any individual program can serve. Requests per second, per process. If that's O(1k) or less, then sure, do whatever you want, it's trivial load.