Ask HN: How do I feel less guilty about bugs?

I see two discrete issues here. I'll post one here, and one in another comment because I'm apparently long-winded today :).

First, as a developer, you _will_ break things. There is absolutely no getting around that - it will happen. You can't choose not to break things. You _can_ choose to learn from it.

I've broken production systems many times, and will break them in the future. I've been writing software for sixteen years now; I break things less often now than I did when I was more junior, but I would say "frequency" is the biggest difference that comes with experience. Instead, I tend not to break things in the same ways.

When I review my work (and the work of teammates) I do so with the critical eye of someone who's seen how things are likely to break. I also look for ways that things are likely to go wrong and pre-plan - to the extent that's possible - how they can be fixed.

Finally, I do everything I can to make it easy to troubleshoot and recover when things (invariably) go wrong. I have a very thorough and "intentional" plan for when things are on fire, and always pause before taking a remediation step.

For example, I once deleted a major project on a scientific publishing platform. The company was very new, and while we had backups, the recovery plan was not exercised and significant work would have been lost if I'd used it. In this case, I was intending to update a single nested attribute on a MongoDB object, but used the wrong operator and replaced the entire document with the single attribute I wanted to update. As soon as I saw what happened, I took a deep breath, and made sure my next action was intentional. My first instinct was to call over the founder and tell them what happened. I estimated that it would take about 5-10 minutes to explain the issue, so I considered my other options. Restoring from backup would take too long, but mounting a backup on another DB was something I could trigger in a few minutes. I sat that aside for the moment as well, but remembered that I should initiate that if I thought I might need it later. Then I looked at what I had in my shell. I scrolled back and found that I had read the document in the Mongo shell a few minutes before performing the update... and the entire document was right there in the output! I took a picture of the screen with my phone. Yes, that's silly, but I wanted to make sure that I didn't misclick or something and completely lose the output. Then I took a screenshot on my MBP, and then I copied the output from the terminal and pasted it into a new text document in a new terminal. I saved that, then opened it in a third shell and verified that it was in fact there. Back to another shell, I built the command I would use to restore the object in Vim. It looked good. I connected to my local database, pasted the command, and executed. It worked! I verified that the data looked like it should, and it did. I then pasted the command into production, visually verified that it was correct, and executed. It worked!

In the above, the Total time from "Oh crap!" to "Whew!" was less than five minutes. As I mentioned, it would have taken me at least that long to call over a more senior person and bring them up to speed. I stood by my decision not to bring others into the emergency immediately for that reason, but that doesn't mean that I was "hiding" anything. Once I saw that it had been fixed, I sent them a Slack message to the effect of "I just broke Project X by overwriting the document in MongoDB. I believe I have resolved it, but please verify it thoroughly ASAP. I'm writing a summary of what happened right now." I then wrote the summary and sent it to him. That summary ended up being the basis for a presentation that I gave at the following all-hands meeting.

Suppose I invited you to a party and I gave you directions but they were so bad you were lost for an hour. I would feel bad about that, even if you enjoyed the party. But that's just one person. If I discover a bug that causes one user each month to lose an hour of work, I'd feel bad and try to get the team to fix it. And then I find out in a couple months that because we have more users it's now affecting 100 people? I can't keep feeling guilty forever.

So I'm sort of asking how do I make peace with that and maybe even feel happy about the work even though it occasionally pisses people off, and as our user base grows the number of people who are pissed off will increase.

Do this in the given order: 1. Add monitoring wherever you can. Not just the exceptions, but latencies, handled success/fail statuses, user flows from clicking on x to saving data on y etc. Create meaningful dashboards & alerts on them. 2. Add release canary process. If you have sufficient traffic, do an A/B test roll out. Have load balancer send 10% of traffic to new binary, 10% to old binary (and make it user sticky - so same user doesn't get bounced between different binaries) . You'd need to be mindful of caching (so version static assets etc). 3. Add tests to critical workflows. This part is usually the hardest to do and pay off is usually the longest but it's critical. However, only do this if your user flows are more or less settled and changes are limited.

Also - before doing 3 - it'd be useful to identify the kind of bugs you see - is it client side bugs or server side bugs? Client side bugs are a pain in the ass overall as UX changes can render tests useless, but try to test critical components using a realistic user flow. Also - as each bug shows up, definitely add a regression test for it (which would have caught some of your bugs that you said "repeat").

Remember - you need to adopt "swiss cheese model"[A] for catching bugs. Each layer will catch something with (1) being a catch all, and the more layers you have the better - but only over time.

The guilt goes away once you start catching bugs before client notices them, or as soon as the bug happens. And as you do (3) and (2) your confidence will improve.

Bugs are fact of human life - no human is perfect, and software is complex. The more complex it is from user interaction pov, the more likelyhood of a bug. Don't sweat, keep doing, keep improving over mid-term and long term, and they'll happen less.

[A] https://en.wikipedia.org/wiki/Swiss_cheese_model

But also pay attention to your company culture. A "blame culture" is one of the worst enemies of good software/product. If your management work by making people feel guilty, you can't have candid conversations about your problems. You'll never find what you need to fix, because you won't be able to discover the real problems.

I've worked in a company where the "blame game" mode were always on. It was terrible. People didn't want to fix a problem, just pass it to the next guy. You'd never find the original cause of problems or have your management invest to solve it.

If you have a culture problem, don't let your employer make you feel bad. Take care of your mental health. A job change may be the best option.