Ask HN: Do you spend more time coding or debugging?

For me debugging usually means "figuring out why we had an outage." This means looking at: 1) application logs, 2) server metrics, and 3) source code associated with the failed applications.

I recently had to ssh and run ngrep on 8(!) servers to see how groups of messages passed around and then look at timestamps to correlate what happened. It was very tedious. This could have been saved by better debug logging; we could have switched that on for 2 minutes, run the tests, and and the looked at everything in Logstash.

When this happens, I end up spending a ton of time tracking down errors. On a bad week, this can be half my time.

So to me, debugging is as much thinking about how you'll have to solve errors in the future and planning for it as it is writing unit tests and tweaking code.

I am working on an existing distributed system with many moving pieces, which is rather prone to outages. This is fintech, so outages mean a lot of money for a lot of people. So my job involves overhauling the existing system, as I upgrade bits of the system slowly: A full rewrite at once would be madness, but I suspect nothing in the current system will remain in two years.

The biggest time sinks are stress testing any of the newer pieces that I try to bring in, followed by incident remediation. There's an incident that requires a human hand to fix it every couple of weeks or so, and I end up spending about three days each time writing better error handling code, adding observability and alerts, and if something is really recurring, writing automation to make the problem fix itself.

This is a fact of life in any distributed system that was written fast: People are start happy because it works most of the time, but as you want the 4th and 5th nine, you need people hardening the system. This is something that is very hard to do as you build anyway: While unit tests are good, there are entire layers of behavior nobody will be able to spec properly by looking one piece at a time, so stress testing, gamedays and such are the only ways to make sure not that the system works to a spec, but that we can even come up with a spec that behaves the way we want in practice.

There's value in evaluating scenarios in your head, but I've also seen what happens when mathematicians use that as their only weapon in a distributed system: Months are spent making sure the system is correct, but then lots of effort is spent on scenarios that are more theoretical than practical, and other scenarios are ignored, even though they occur a lot in practice.

In this respect, it's not very different from entrepreneurship: Getting an MVP out the door and doing things by hand instead of using automation is going to beat spending a lot of time making a product without having any idea of what the market really wants.

80% maintenance