How To Write Good Log Messages

Post by moe » Fri, Aug 10, 2012, 2:38 PM UTC

3amOpsGuy wrote: post #50

moe wrote: post #49

Earlier quoted context omitted.

When you roll the logs at say 500Mb, there's the chance our monitoring system misses an error Sorry but then your monitoring system is broken? Why is its operation tied to log chunking? Multiple logfiles for a day are harder to search Not really. Often enough interesting events span multiple days (or cross midnight) anyways. Thus regardless of chunking you have to be prepared for that case. What benefits does a split…

>> Sorry but then your monitoring system is broken? Why is its operation tied to log chunking? Yours clearly bends some unbendable truths of the known universe ;-) A monitoring system ordinarily samples the logfile at a given point in time, that is an open followed by a close operation on the logfile (lest you run into phantom disk usage issues caused by holding an open file handle to a compressed / rotated logfile).…

Post by moe » Fri, Aug 10, 2012, 4:18 PM UTC

3amOpsGuy wrote: post #52

moe wrote: post #51

Earlier quoted context omitted.

Possible workarounds include logfile streaming (say via a pipe on the filesystem) but that introduces much larger problems and would in no way be compatible with the app rolling its log files. What you call a workaround (streaming to a central location via syslog or scribe) happens to be the standard approach in my corner of the world. Analytics and monitoring operate naturally on the stream because, as you point out…

>> streaming to a central location via syslog or scribe Ahh, "man mkpipe", you're mixing up a separate concept - streaming to a remote host vs streaming via a pipe on the filesystem. >> standard approach in my corner of the world Syslog streaming has been a common approach since the ~70s i'd guess? Likely before. >> yet feel entitled to give ops-advice that is at odds with how the rest of the world operates Hopefully…

Post by michaelcampbell » Fri, Aug 10, 2012, 5:34 PM UTC

memset wrote: post #7

I personally have found a lot of difficulty with JSON-based logs, because tools like `grep` and `cut` don't suffice; I usually have to write a more complex regex with `sed` in order to extract useful information I want. Anyone have a good utility for managing JSON in log files?

Post by gwillen » Wed, Aug 22, 2012, 4:24 PM UTC

emmelaich wrote: post #56

gwillen wrote: post #44

This may be a bit out of scope for this post, but at Google I used to get a lot of mileage out of a system I worked on called Dapper: http://static.googleusercontent.com/external_content/untrust... It uses thread-local storage and RPC library instrumentation to thread a fixed request-id through every RPC call and thread handling a particular user request. Then you can do things like automatically adding the request I…

Dapper looks fantastic. It reminded me somewhat of the ancient ARM (1996) http://en.wikipedia.org/wiki/Application_Response_Measuremen... which I thought had died. I was pleasantly surprised to see that there's a module for Apache in the works.