How to approach and prioritize technical debt

> Take an engineer who ostensibly "intended" to accrue some technical debt. Ask them to fix said debt and note that you'll find more debt accrued.

I don't see how the second statement follows from the first. Are you saying that good engineers always have enough time to implement every feature "the right way" on the first try?

> But building a system with a fixed set of value in a robust and decoupled way is not a function of immediate time available _today_ -- it's a function of skill set.

Of course it is a function of immediate time available today.

> Usually it's because there's a conflation between system "flexibility" and architectural soundness, and a misunderstanding that some code duplication has anything to do with 'tech debt'.

This sounds like it's targeted at one specific developer you used to work with (I have worked with developers like this too, it sucks)... but that has nothing to do with time/quality tradeoffs in design and development.

> The fallacy behind 'tech debt' is that it assumes as an axiom that continuously building a robust system is really just a matter of time spent -- and otherwise has little do with experience and value set.

This is like Calvinism for programmers. Yes, some programmers are better at software architecture and design than others. Yes, some programmers leave behind a mess which, if not fixed, does truly turn into "legacy code" and "tech debt".

But it's ridiculous to assert that all technical debt follows this pattern. Design is often iterative; people are not omniscient. Bad engineers fail to evaluate the tradeoffs correctly, or aren't aware of all their options, and make bad decisions. Good engineers understand the tradeoffs and know their options, and make good decisions.

Let me give an example from my actual job:

I was tasked with implementing a sort of graph lookup system. I had two options: pre-compute a large portion of the graph, leading to a nice big flat table that could be easily queried at runtime with a single join, or leave the entire thing in a normalized format and do a big multi-join lookup at runtime. I chose the second option, because this allowed me develop a system and codebase that was simple and easy to understand, offloading the complicated work to the database. Moreover, it allowed me to avoid the various operational issues and technical details around setting up and maintaining the intermediate pre-computed table. The performance requirements were such that the increase in latency and server load wasn't important for the first version of the product. The business was and remains happy with the product.

Now that it's humming along and users are starting to put more load on it, migrating to the first option is now on our engineering calendar for the year. With ~12 months of accumulated knowledge about this system and its use cases, with an increasingly stable API, and with an increasingly rigorous & thorough test suite, I feel confident that swapping out the internals (including the database layout) is a relatively low-risk operation at this point. The internals have already been subject to several incremental refactorings simplifications as the business stabilized the product requirements; these were implemented without any visible effect on users, and with minimal changes to other "internal consumers" who depended on the API.

If I had tried to implement things that way from day 1, I would have had a system that was more complicated, more difficult to maintain, more difficult to explain to other programmers, and had more operational elements that could break. But I did indeed take a loan of "tech debt" in the form of worse performance and high server load, which we will have to pay off eventually.

Perhaps if I was a better programmer, I would have had the skills to implement the first option right, on the first try. But I don't know many (any?) programmers who do have those kinds of skills, while also having the wherewithal to document the thing, write useful and nontrivial test assertions, develop tooling for OpenAPI conformance, et alia. Not to mention that the flexibility granted by simplicity actually has turned out to be very important for being able to respond to changes in product requests quickly.

It seems (so far) that I did make the right decisions in this situation. I think that makes me at least a decent software engineer.

Code gets plastered over with features and abstraction layers, but that's an active process that we are complicit in doing.

More germane for this discussion, technical debt was originally defined as a positive thing that you want to go get... It is the mismatch between our domain model and how our users think about the domain. It is positive because “enough with all of the planning and interviewing and requirements gathering and careful architecting, can I just build something and have my users criticize it and do four or five drafts until I get it right?” ... The debt is the drafts before you get it right, the interest is the constant translation between the language of your users and the language that the system is expressed in. You have a “contracts” table that contains things that are not contracts because every purchase foreign keys to a contract, but your users have since wanted to know what they do with the purchases that are not associated with any contract, where do they go. And now every query that aggregates over contracts needs to exclude the non-contract contracts, this is part of the interest you are paying.

But at some point it started to mean that we had to upgrade our dependencies, that's tech debt, or this kludge that I threw in, that's tech debt, or the fact that we never worked out a shared library between the front end and the back end and so any Python code in utils.py needs to be manually kept up to date with a file utils.js in a separate Git repo so that we are sure that we can do these things both on the front end and the back end. More broadly, anything that we no longer care for is tech debt. And that's where you really get this idea that it is deteriorating, that's more a measure of our own patience deteriorating, especially as we never seem to have time for the refactors we want.

Perversely the cause of not being able to refactor has nothing to do with tech debt and cannot be solved in this way. It's multifaceted so at different places it emerges in different ways, but usually it's an incentive problem. At some places it is that the only incentives are for feature development. At other places it is that every developer is working on a different thing rather than prioritizing one thing that the business needs and delivering it, and allowing developers to use their slack time in this equation to improve the product however they see fit... In yet another it is because the team lead resists any suggestion that the framework being used is too heavyweight for the problem being solved and so trying to keep these very clean abstraction layers is causing people to have to rewrite the same basic thing in five different unrelated places, because it has to bubble up from the data layer into the service layer into the business layer into the controller layer into the API into the consumer layer into the app state layer into the frontend model layer...

Technical debt is the answer to management requests that cannot be accomplished in the requested time with the available resources. No more, no less. The word debt refers to the fact that the shortcut needed to do so will cause problems repeatedly in the future (interest that will need to be paid).

In my experience, engineers will make this clear to management, and will point out the additional time needed to implement a solution that avoids those problems, and management will make the decision to continue as is.

For a concrete example, management requests that you implement real-time collaborative editing in an existing tool. You have enough time to replace all the backend stuff with a CRDT or OT based implementation, but the frontend architecture is not equipped to deal with collaborative editing and there is no time to do both with the people available.

So, in order to meet the management deadline, quick hacks are added to bolt on the collaborative UI on top of the existing UI. Eventually new features have to be added that simply cannot be supported in the existing UI architecture without massive workarounds. Carving out some time to slowly refactor the UI architecture is rarely done, so the problem accumulates until development speed slows to a crawl.

That is technical debt, and it's entirely a byproduct of management/development negotiations. Has nothing to do with engineers. Nobody could have predicted that the architecture would need to support an entirely different backend in the future.

EDIT: Just to be clear, technical debt happens as a result of a conscious tradeoff between one or more options, where the more problematic option is chosen to get to an "acceptable" solution in a shorter period or with less resources. Some dev writing a shitty architecture is not technical debt. Depending on a library that later gets EOLed is not technical debt.

Figuring out you have a shitty architecture or an EOL library and choosing to continue building on it despite clear evidence it is a bad idea, in service of shorter time to market, that is technical debt.

It kind of does -- if you leave a codebase alone for a long time, and you come back to it later to upgrade a lot of dependencies (sometimes making a multi-version jump), it's a lot harder than it would have been to keep them updated as new versions were released.

It would have been a lot worse if that log4j CVE had been in a library with a lot more transitive dependencies or makes breaking changes between versions, like Jersey.

One advantage of monorepos with shared dependencies is that even the parts of the code that don't need to be touched very often will still get the latest dependency updates. If those codebases are in standalone repos, they just sit there, and then one day a simple attempt to upgrade a dependency turns into hours of work.

So, it's not technically "rotting," but it's definitely the case that leaving code to sit creates more technical debt later on -- even if that code was perfectly good the last time anyone worked on it.