Why do web sites and software take so long to build? And why is it so hard?

So the underlying development method, nowadays denounced as 'waterfall', wasn't sufficient? I am not an Agile zealot but change and 'knowing better' needs to be embraced during the development process and not ruled out.

I've always felt that comparing software with architecture, building a car etc. is what people naturally reach for since it's something they're familiar with in the physical world and much of the same terminology is used.

I don't think that software is at that point, though. Physical things "play nice" together because they are part of the physical world. Materials have characteristics that are inherent and don't need to be conceived of where as the way things behave and interact in software needs to be defined and constructed entirely by humans.

I guess one could argue that happens to a certain extent in the world of materials science but ... I don't really think it holds up.

I arrived at a point where I started to think of building software as writing a novel. Once you start to use that as an analogy, it doesn't seem so weird that it's hard and takes ages, because so does writing a novel.

In a novel, one must define the entire world, and one can make the choice of using/re-using story lines or doing something original. People that trot out formulaic drivel make better money than the tortured geniuses on average, but the few tortured geniuses that manage to hit, hit it big and serve as an example to all the others.

I kind of stopped thinking about it at this point and got back to work, but I think that using that analogy, things really start to make more sense ... what do you reckon?

I also wonder whether teaching practitioners to diagnose and identify what type of project one is dealing with might avoid blind spots when estimating projects. For instance, an LDAP project should be identified as an integration project, where one might have to deal with unexpected schema.

The real problem with software projects at the moment is we lack a proper framework to think about the nature of each project and therefore don't understand where the risks are, and properly advise clients.

"When I had my bathroom remodelled, at first I thought it was organized amazingly well and why couldn’t software be like that? There was a designer from Expo, a primary contractor and subcontractors for tiling, painting, etc. a project workbook containing all the documents, including the design, and a logbook for every contractor to record their visit. But it turned out to be just like a software project.

As soon as they started ripping up the walls, the painstakingly drawn design for the tub/shower was hastily and arbitrarily adjusted because the casing for an outside fire extinguisher was in the way. Many of the components turned out to be incompatible with each other and many were not the ones originally ordered (the bathtub was not even the one labelled on its box!)

Communication was terrible – the city inspector would tell the primary contractor to change something, and then a subcontractor would show up and ask me what he was supposed to do. When there was a disagreement about whether the tiling was done properly, one of the contractors disfigured the tiling to force another contractor to redo it. They all had other projects, so for long stretches I had a pile of dirt (for the cement) in my patio and a non-functioning bathroom for months. And toward the end there were some quick patches, e.g. the walls were slightly curved so edges of the tiles looked bad and they just painted the patches, which worked but cost me extra. And a couple of years later when the tub sprung a leak, the plumber couldn’t figure out how to get in there, said they must have done a shoddy job on connecting the pipes, then when he finally got a good look at it, realized why they did it that way. It was exactly like a software project!"

A lot of assumptions have to be met for that to be true. It shouldn't take very long provided:

1. The programmers are very familiar with the tools.

2. The programmers are very familiar with all necessary 3rd party components.

3. The 3rd party components account for 99% of functionality.

4. The programmers have built that exact project before, preferably more than once.

I usually work in a best-case scenario for development: I'm building projects with limited scope, solo and with no other decision makers involved in the project. I'm also experienced enough to know how complex things will generally be from a high level standpoint. And I have never once met my own estimate for complexity and the amount of time something will take. Exactly 100% of the time silly, seemingly inconsequential things add up to an extra 30% or more.

It doesn't matter if you can write 1000 shippable lines of code in a day because tomorrow you'll spend most of your time hunting a bug in an external library. Suddenly your machine-like pace has been cut in half. This happens on every project I've worked on and I would assume all software projects.

I agree that poor planning adds to the amount of time needed. Of course it does. But there really is no "best case" scenario where you and a couple of buddies could hammer out Facebook in three weeks.