You say “cave dweller debugging”, I say debug logging

I feel that it is the opposite. It is where the path is obscure that logging works really well. It can be iteratively refined and repeated.

FATAL = The system is basically non-functional, even in some partially-working way. Alerts fire.

ERROR = Something went wrong in some part of the system. The system may be mostly or partially working, but an important thing is broken and should be fixed. Depending on the where the error is, it may need to be immediately addressed or at least very soon.

WARNING = Something isn't working right, but it's possibly transient and the system has was a handling this via retry or other workarounds. This should be addressed at some point if possible, but shouldn't require immediate intervention.

INFO = This is the log level that the system is designed to run at. It provides enough human-readable context to see how the system is running. Most importantly, it provides just enough information provide context to WARNINGS/ERRORS/FATALS that may occur somewhere in a given request. The goal is to balance information w/performance.

DEBUG = All the things, all the time. Default mode for developers, and when diagnosing issues that require much more depth of information than provided at INFO level. It is VERY RARE to run at this level in production, as INFO logs should provide enough request context for a developer to reproduce most issues, but can be enabled for short bursts in production on one server in a cluster, or some other limited form to capture necessary information.

A guiding philosophy on all logs is to persist them in a central location (and locally w/rollover just in case persistence is broken), and is to flow some request or context ID through the system so logs from a single request can be easily identified and related log statements correlated with each other.

At my last startup, it was almost a year effort to clean up the logs. Such things like app pool crash on startup were logged at DEBUG, and the application starting successfully was logged at ERROR. Combined with an effort to improve overall quality, hit a first milestone of going 3 months without the ops team requiring escalation to developers outside of business hours (i.e. they could identify and fix things based on logs). Eventually, out of business hours alerts because a "I don't remember the last one" thing.

The stages of understanding go like this, in my experience:

1. Log nothing.

2. Upon being told to log things, write mostly useless logs: "Entering f() function", "leaving f() function". When f() fails, who knows what happened?

3. Upon being taught to add context so that logs can actually be used to troubleshoot problems, "entering f() with id 123", "id is 123", "writing 123", etc.

4. Enlightenment usually comes when logs include necessary info but are not redundant, are not noisy, and can actually be used to track down problems. Of course, improvement at step 4 is always ongoing, for all of us.