Infrastructure Debt

You have to walk a fine line. If you neglect your infrastructure for too long you get in a situation where you get bogged down by all the circumstantial complexity in your infrastructure. If you are too zealous about debt you automate things that don't need automating and you end up with an over-engineered infrastructure that is completely automated but never works quite right.

As with the design of software projects the design of the infrastructure takes time. People often go through phases where they take it either far too seriously (after they got burned) and through phases where they cowboy-code their way through.

The problem here is really mundane. It's difficult to set up an architecture right and you don't get there by throwing buzzwords to the wall (and see which ones stick). Even if you use multiplexed dictionary stores, Bazoop Clusters, BunnyHQ, and tricircular backups you still may end up with with a mess of an architecture if you don't know exactly which problems you're trying to fix. And you can easily waste a few weeks evaluating different deployment tools and automation software and still end up with something that works only barely. The opportunity cost is immense.

Re: Infrastructure Debt

#4

Post by narcissus » Fri, Nov 04, 2011, 1:01 PM UTC

I get the benefits of having identical environments as production (and that's what I aim for, at least, with our staging server) but to be honest, I like having a small amount of diversity across our development servers. It's the small differences that have helped us to work out that bizarre bug (turns out, it was actually a problem specific to a particular version of PHP) or to find those small assumptions about the environment (eg. assumed file paths and so on).

The latter aren't that much of an issue, but I find that being able to deal path changes, for example, makes the code that little bit more flexible, making it a little easier to make changes to the production environments going forward with less issue.

Re: Infrastructure Debt

#5

Post by valjavec » Fri, Nov 04, 2011, 1:05 PM UTC

Best quote to conclude article:

"Don’t be scared of change, be scared of the debt growing in your code base and in in your infrastructure. It won’t go away and there is no government bailout on the way to fix it."

Re: Infrastructure Debt

#6

Post by grumpycanuck » Fri, Nov 04, 2011, 1:12 PM UTC

arethuza wrote: post #2

"Version control is a 20 year old, well-understood concept." Closer to a 40 year old concept: http://code.google.com/p/pysync/wiki/VCSHistory The first, at least according to that article, being SCCS in 1972: http://en.wikipedia.org/wiki/Source_Code_Control_System

Blog post author here. Yeah, I figured it has been around a lot longer than 20 years but I thought the 20 year thing would get the point across.

Re: Infrastructure Debt

#7

Post by jader201 » Fri, Nov 04, 2011, 1:14 PM UTC

What about database design debt? Or is this still considered technical debt?

In our company, this is 90% responsible for the paralysis that is keeping us from migrating from a 10 year old enterprise architecture. It is designed in such a way that makes it hard for us to extend and scale, but the whole foundation of our business rests on this outdated model. It's to the point that the only way we can move forward is to start over.

I've read about technical debt, but it's usually in reference to code design, lack of proper testing, and tightly coupled dependencies.

And we definitely suffer from "infrastructure debt" as this article describes, but I feel this is the least of our problems.

To me, possibly the most expensive kind of debt to be in is database design debt, as everything rests on this. At least this is the case where I work.

Re: Infrastructure Debt

#8

Post by grumpycanuck » Fri, Nov 04, 2011, 1:17 PM UTC

jader201 wrote: post #7

What about database design debt? Or is this still considered technical debt? In our company, this is 90% responsible for the paralysis that is keeping us from migrating from a 10 year old enterprise architecture. It is designed in such a way that makes it hard for us to extend and scale, but the whole foundation of our business rests on this outdated model. It's to the point that the only way we can move forward is t…

(I'm the blog post author)

I would place database design debt firmly in the technical debt side of the ledger (as it were). To me, infrastructure debt deals more with consistency across environments and consistency in moving code from one environment to another.

I also agree with you that database design debt is very expensive and, as you pointed out, leads to paralysis over fixing problems with an established application.

Re: Infrastructure Debt

#9

Post by raganwald » Fri, Nov 04, 2011, 1:31 PM UTC

I like the article, but am thinking about a bike shed. For each thing, like manual deployments or failure to use source control, I ask myself: Is it “debt?" Or "friction?”

Debt and friction both accumulate, but debt must be "repaid" in a lump sum. If "technical debt" or "infrastructure debt" means your velocity is falling over time, it's friction, not debt.

Infrastructure debt would be something that is eventually going to cause everything to halt while you sort it out, and the longer you wait, the worse it will be. Not using source control is debt, because with near certainty you are going to have at least one major SNAFU requiring spelunking through backups to recover a lost file or to restore some prior release.

Developing without deploying at all is definitely debt, sooner or later you are gong to have to deploy.

OTOH, lot of "technical debt," isn't. It's just friction, and it takes some experience to know when it's a bad tradeoff.

Re: Infrastructure Debt

#10

Post by gbog » Fri, Nov 04, 2011, 1:37 PM UTC

To the author: would a bad choice of language, framework, persistence layer count as infrastructure debt? If so, then it should be noted as the most dangerous of all.