121–130 of 238 posts
Rule 1 should be: Reproduce with most minimal setup.
99% you’ll already have found the bug.
1% for me was a font that couldn’t do a combination of letters in a row. life ft, just didn’t work and thats why it made mistakes in the PDF.
No way I could’ve ever known that if I wouldn’t have reproduced it down to the letter.
Just split code in half till you find what’s the exact part that goes wrong.
> Quit thinking and look (get data first, don't just do complicated repairs based on guessing) From my experience, this is the single most important part of the process. Once you keep in mind that nothing paranormal ever happens in systems and everything has an explanation, it is your job to find the reason for things, not guess them. I tell my team: just put your brain aside and start following the flow of events ch…
1. Is this mistake somewhere else also?
2. What next bug is hidden behind this one?
3. What should I do to prevent bugs like this?
For #4 (divide and conquer), I've found `git bisect` helps a lot. If you have a known good commit and one of dozens or hundreds of commits after that is bad, this can help you identify the bad commit / code in a few steps. Here's a walk through on using it: https://nickjanetakis.com/blog/using-git-bisect-to-help-find... I jumped into a pretty big unknown code base in a live consulting call and we found the problem pr…
Like, traffic going from A to B can turn ... complicated with VPNs and such. You kinda have source firewalls, source routing, connectivity of the source to a router, routing on the router, firewalls on the router, various VPN configs that can go wrong, and all of that on the destination side as well. There can easily be 15+ things that can cause the traffic to disappear.
That's why our runbook recommends to start troubleshooting by dumping traffic on the VPN nodes. That's a very low-effort, quick step to figure out on which of the six-ish legs of the journey drops traffic - to VPN, through VPN, to destination, back to VPN node, back through VPN, back to source. Then you realize traffic back to VPN node disappears and you can dig into that.
And this is a powerful concept to think through in system troubleshooting: Can I understand my system as a number of connected tubes, so that I have a simple, low-effort way to pinpoint one tube to look further into?
As another example, for many services, the answer here is to look at the requests on the loadbalancer. This quickly isolates which services are throwing errors blowing up requests, so you can start looking at those. Or, system metrics can help - which services / servers are burning CPU and thus do something, and which aren't? Does that pattern make sense? Sometimes this can tell you what step in a pipeline of steps on different systems fails.
I have been bitten more than once thinking that my initial assumption was correct, diving deeper and deeper - only to realize I had to ascend and look outside of the rabbit hole to find the actual issue. > Assumption is the mother of all screwups.
Make sure you're editing the correct file on the correct machine.
My first rule for debugging debutants: Don't be too embarassed to scatter debug logmessages in the code. It helps. My second rule: Don't forget to remove them when you're done.
My rule for a long time has been anytime I add a print or log, except for the first time I am writing some new cide with tricky logic, which I try not to do, never delete it. Lower it to the lowest possible debug or trace level but if it was useful once it will be useful again, even if only to document the flow thru the code on full debug. The nicest log package I had would always count the number of times a log msg…
A few times I looked for a bug like "something is not happening when it should" or "This is not the expected result", when the issue was with some config file, database records, or thing sent by a server.
For instance, particularly nasty are non-printable characters in text files that you don't see when you open the file.
"simulate the failure" is sometimes useful, actually. Ask yourself "how would I implement this behavior", maybe even do it.
Also: never reason on the absence of a specific log line. The logs can be wrong (bugged) too, sometimes. If you printf-debugging a problem around a conditional for instance, log both branches.
I’m not sure that doesn’t sit well with me. Rule 1 should be: Reproduce with most minimal setup. 99% you’ll already have found the bug. 1% for me was a font that couldn’t do a combination of letters in a row. life ft, just didn’t work and thats why it made mistakes in the PDF. No way I could’ve ever known that if I wouldn’t have reproduced it down to the letter. Just split code in half till you find what’s the exact…