Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You say “cave dweller debugging”, I say debug logging

sicpers.info

Post reply on HN
Print view View original

131–139 of 139 posts

Re: You say “cave dweller debugging”, I say debug logging

#131

Post by invalidname »

geophile wrote: post #124
invalidname wrote: post #119

Earlier quoted context omitted.

The problem is that in order to add a missing log in production you need to effectively redeploy. That isn't tenable for a polyglot production cluster. There's a rising field of developer observability which is specifically designed to provide polyglot cloud "debugging" capabilities. One of the capabilities is to inject new logs dynamically.

And if you need to fix a bug in production you need to redeploy too. Bugs and logs and testing are all related. You obviously (should) do unit testing, integration testing, system testing before production. As you do these things, you will find and fix bugs. Ideally, you would add logging to help you diagnose the bugs you are fixing. Congratulations, you now have logging in code known to have been buggy, which will s…

The difference between deploying once and deploying just to add more logs. And more logs is a huge difference. I don't know about your environment but I can't push something to production on my own schedule. I need approval, process, etc.

This is painful to do just to add a log which might not include all the information I need.

Re: You say “cave dweller debugging”, I say debug logging

#132

Post by geophile »

invalidname wrote: post #131
geophile wrote: post #124

Earlier quoted context omitted.

And if you need to fix a bug in production you need to redeploy too. Bugs and logs and testing are all related. You obviously (should) do unit testing, integration testing, system testing before production. As you do these things, you will find and fix bugs. Ideally, you would add logging to help you diagnose the bugs you are fixing. Congratulations, you now have logging in code known to have been buggy, which will s…

The difference between deploying once and deploying just to add more logs. And more logs is a huge difference. I don't know about your environment but I can't push something to production on my own schedule. I need approval, process, etc. This is painful to do just to add a log which might not include all the information I need.

The more constraints you have on pushing something to production, the less it makes sense to deploy logging dynamically! Those constraints are there to minimize risk, ensure quality and security, etc. Dynamically deployed logging seems to provide an extremely large loophole. For example, consider the recent Java logging fiasco.

Re: You say “cave dweller debugging”, I say debug logging

#133

Post by invalidname »

geophile wrote: post #132
invalidname wrote: post #131

Earlier quoted context omitted.

The difference between deploying once and deploying just to add more logs. And more logs is a huge difference. I don't know about your environment but I can't push something to production on my own schedule. I need approval, process, etc. This is painful to do just to add a log which might not include all the information I need.

The more constraints you have on pushing something to production, the less it makes sense to deploy logging dynamically! Those constraints are there to minimize risk, ensure quality and security, etc. Dynamically deployed logging seems to provide an extremely large loophole. For example, consider the recent Java logging fiasco.

Excellent point. The tool we use (Lightun) runs all of that in a read-only sandbox and limits the classes you can access (they are in a sandbox too) so you can't print "anything". It throttles your access so you won't use too much CPU with expensive operations too.

It also comes with PII reduction and block lists so you won't just log user passwords in the login code...

This solves the problem of over-logging: https://www.reddit.com/r/devops/comments/udgohy/there_is_no_...

Re: You say “cave dweller debugging”, I say debug logging

#134

Post by roca »

geophile wrote: post #79
10000truths wrote: post #67

Earlier quoted context omitted.

Working with gdb is quite the learning curve. Luckily, it’s mostly unnecessary, except in the rare case where you actually need register-level debugging or hardware watchpoints. But it can still be done if you’re willing to suffer through the docs. For debugging multithreaded code, you can run a gdb script in batch/non-interactive mode, and direct the gdb output to some out-of-band channel, like a file or other termi…

... or you could just log. I don't understand the extreme attachment to interactive debugging that I see in some of the comments here. It's like people are so attached to this idea that they will do anything to make it work as each complication is added. Furthermore, the amount of attention and manual labor needed to use a debugger effectively is just staggering. One mistake, e.g. you step over a function you should…

> One mistake, e.g. you step over a function you should have stepped into, and you've got to start over.

Not with rr (rr-project.org). With rr you can step right back where you were. rr also lets you debug after the fact without interrupting your program, and lets you collect a recording on a cluster machine and then debug the recording elsewhere.

Re: You say “cave dweller debugging”, I say debug logging

#136

Post by roca »

geophile wrote: post #48

I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless. But the real problem, and top-l…

> I have become less and less enamored of gdb (or IDE equivalent) debugging over my many years in software. Debugging is often very confusing with multithreaded code. It only works in one language at a time. If you're doing Java with JNI for example, the code called through JNI is a black box. If you are doing anything involving multiple processes, or distribution, debuggers are useless.

These are true of plain-old-gdb but they're not inherent to interactive debugging. rr and Pernosco address most of these issues and with more work we could do even better.

In particular: rr is great for debugging multithreaded code; record a race once and then debug the replay as much as you want until you've figured it out. rr will also record multiprocess code and let you debug whichever process you want; Pernosco goes further and supports seamless debugging across all processes. https://pernos.co/about/javascript/ shows how Pernosco can support debugging high-level code (JS) and C++ code at the same time. rr can record processes on multiple machines; we don't have an integrated debugging experience for distributed systems yet, but it's not hard to see how that could be done.

> But the real problem, and top-level issue, is that you are constantly rooting around the very lowest level of your code, and it is difficult and labor-intensive to get a higher level view of what's going on.

That depends on the debugger. Interactive debuggers can do higher-level things if we want them to, e.g https://pernos.co/about/expressions/

Re: You say “cave dweller debugging”, I say debug logging

#137

Post by icedchai »

invalidname wrote: post #130
icedchai wrote: post #129

Earlier quoted context omitted.

Not necessarily. I've been at some places where you just "do it live." We had dozens of worker nodes. An engineer would "reserve" one for debugging purposes, editing python code live, adding logs or prints as needed and restarting services. Worst case a small percentage of requests was lost, though generally they would be retried except in the most extreme cases. This risk was considered acceptable. Obviously, depend…

For that you have developer observability which doesn't carry the HUGE gaping security risks or performance problems. But still lets you debug the cluster at scale.

I’ve worked in that space. Observability tools may not give you all you need to debug certain classes of problems. I saw your other comments about Lightrun and will check it out though!

Re: You say “cave dweller debugging”, I say debug logging

#139

Post by bckr »

shakna wrote: post #113
bckr wrote: post #80

Earlier quoted context omitted.

An idea that I've been kicking around is to, in Python codebases, have a @log decorator that first logs the name of the function, all of the parameters it's called with and a function call id, then the name of the function, its return value, and the same function call id. This can be applied to as many of the functions in the codebase as necessary to provide "replayability". Haven't got the chance to try it out yet.

import logging import threading logger = logging.getLogger(__name__) def log(fn): def log_inner(*a, **kw): logger.info('%s(%s, %s) @ %s', fn, a, kw, threading.get_native_id()) return fn(*a, **kw) return log_inner It's pretty easy to grab all the arguments and thread, but grabbing the return would depend on you not mutating anything permanent, so that probably needs some special casing.

Thanks for typing that up! The test will be whether this will help as much as I think it would in a real system.
Post reply on HN