What I missed here was the importance of keeping careful notes as you go. What exactly happened when we constructed that weird input and commented out line 353? What hypotheses are we entertaining? Can we rule out any of them based on our evidence? It’s very easy to dupe yourself if you’re doing it all in your head.
Troubleshooting: A skill that never goes obsolete
101–110 of 141 posts
Re: Troubleshooting: A skill that never goes obsolete
#102Earlier quoted context omitted.
> this is why I also promote using a debugger as much as I can as well - there's nothing quite like being able to see _exactly_ what's going on. I don't know, the speed of just reading code and maybe inserting some diagnostic messages is hard to beat. It's a pretty bad day if I feel like I need to bust out a debugger—99% of "seeing _exactly_ what's going on" is not going to be relevant and will just distract you.
The code rarely resembles the runtime state of the system. Debuggers are an incredible shortcut almost all the time.
I strongly and emphatically disagree on every point (I don't even know what your statement about runtime state and code even means actually, it just seems like a category error, but how do you even write code without being able to reason about runtime state? It just makes no sense.), but I understand there are people who love their debugger and I respect them.
Basically the only time I pop open the debugger is when it's otherwise difficult to see runtime behavior—say, a certain condition in a server for which you can't easily access logs. Outside of that it feels like major overhead and distraction from getting the bug fixed. Plus iterating without print statements is a tedious, tedious affair.
Don't get me wrong, it's a critical and necessary skill that junior devs often struggle to understand and master. I just think over reliance on the debugger will slow your velocity over time when most bugs have straightforward causes easiest to see by simply reading the code (which you'll have to do anyway with a debugger). I can't tell you how many times I've had to tell devs to put away their tools so we can calmly analyze the code without flipping back and forth between views. The vast majority of the time it's the second pair of eyes that resolves the problem.
Re: Troubleshooting: A skill that never goes obsolete
#103> Realizing that I spend more time troubleshooting than I do building or doing ... That's not good. The problem with troubleshooting is that it messes up with your reward system. After you fix a hard-to-debug problem, you feel a sense of accomplishment. Which would be ok, but the problem is that this sense of accomplishment is often time higher than it should be. You go home at the end of the day thinking "well, toda…
I agree the need for troubleshooting can be born from poor decisions, but it's still a marketable skill for places that need it at scale. One of my roles was head of Linux Engineering for a Fortune 50. Sure there's the pets vs cattle thing and we all prefer cattle, but particularly in places with lots of legacy apps and infrastructure there are plenty of both that need more nuance than turning it off and back on again.
There's value in fixing things in the moment and then feeding them back to your engineer and architecture functions to address endemic issues so that everyone benefits.
Re: Troubleshooting: A skill that never goes obsolete
#104Earlier quoted context omitted.
"troubleshooting really can't be taught" Exactly: it is a gift. You have "the Knack". (Dilbert - The Knack "The Curse of the Engineer")
I don’t believe that’s true. It’s an attitude, not some kind of innate skill like reflexes. You can learn to believe in yourself, plus it’s teachable in my experience.
Yes attitudes are harder to learn than skills aren't they?
Ever notice people get more stubborn and stuck in their ways over time?
It's possible you cannot teach people to want different things.
Re: Troubleshooting: A skill that never goes obsolete
#105Earlier quoted context omitted.
"troubleshooting really can't be taught" Exactly: it is a gift. You have "the Knack". (Dilbert - The Knack "The Curse of the Engineer")
I think it has to do with interests. Some people have an inmate interest in how stuff works, and specifically how it breaks. I think you can teach someone to troubleshoot in a procedural and methodical manner, but they will always lack the creative "spark" that comes from being actually interested. Procedural troubleshooters are useful, but they won't exceed the bounds of the model they've been taught to work under.
Right - can you teach people to like different things? Maybe? Generally, no.
Re: Troubleshooting: A skill that never goes obsolete
#106> Realizing that I spend more time troubleshooting than I do building or doing ... That's not good. The problem with troubleshooting is that it messes up with your reward system. After you fix a hard-to-debug problem, you feel a sense of accomplishment. Which would be ok, but the problem is that this sense of accomplishment is often time higher than it should be. You go home at the end of the day thinking "well, toda…
I think you may be leaning too far in the other direction. I'm a troubleshooter. I fix problems. I keep my head straight in a crisis. Every job I've had across 3 decades, regardless of my actual title or formal responsibilities, I'm the firefighter. People call me when they can't figure something out. People call me when something big breaks and needs to be fixed urgently. Even if I'm not an expert in the broken thin…
Textbook survivorship bias.
> I wouldn't need to help out nearly as many people because they could handle their own crises.
They don't need to, because there's always you who can figure out boring minutiae for them while they deliver business value.
Re: Troubleshooting: A skill that never goes obsolete
#107When the CEO's vibe-coded slop gets chucked over the wall to become someone else's problem once completed in rough prototype form, and the ensuing bugs and scalability/reliability issues manifest, troubleshooting is going to be a more valuable skill than ever!
We'll get paid peanuts for it, but hey, we should be thankful for the work in the first place!
Re: Troubleshooting: A skill that never goes obsolete
#108From zen and the art of moto: ‘There’s no fault isolation problem in motorcycle maintenance that can stand up to it. When you’ve hit a really tough one, tried everything, racked your brain and nothing works, and you know that this time Nature has really decided to be difficult, you say, “Okay, Nature, that’s the end of the nice guy,” and you crank up the formal scientific method.’
Re: Troubleshooting: A skill that never goes obsolete
#109From zen and the art of moto: ‘There’s no fault isolation problem in motorcycle maintenance that can stand up to it. When you’ve hit a really tough one, tried everything, racked your brain and nothing works, and you know that this time Nature has really decided to be difficult, you say, “Okay, Nature, that’s the end of the nice guy,” and you crank up the formal scientific method.’
The more I hear about this book, the more I realize that I was way too young when I read it.
Re: Troubleshooting: A skill that never goes obsolete
#110From zen and the art of moto: ‘There’s no fault isolation problem in motorcycle maintenance that can stand up to it. When you’ve hit a really tough one, tried everything, racked your brain and nothing works, and you know that this time Nature has really decided to be difficult, you say, “Okay, Nature, that’s the end of the nice guy,” and you crank up the formal scientific method.’
The more I hear about this book, the more I realize that I was way too young when I read it.
I read it at age 18 and thought, "I should go buy a motorcycle and ride it around. That's the answer." Then I read it at age 30 and thought..."Oh, that wasn't the point at all."