Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why do web sites and software take so long to build? And why is it so hard?

scottporad.com

Post reply on HN
Print view View original

81–90 of 159 posts

Re: Why do web sites and software take so long to build? And why is it so hard?

#81

Post by kaiwetzel »

tablatom wrote: post #24

To date, software engineering has failed to deliver on one critical goal, which is why we are in this mess. The world of software development needs to be divided in two -- the component creators, and the component assemblers. To stick with the given analogy, component creators are the people inventing new kinds of plumbing: easier ways of connecting pipes, taps that don't ever drip. Component assemblers are the peopl…

I'm just reading the book Flow-Based Programming (2nd Edition) by Paul Morrison[1].

I would say the main idea is to shift focus from control-flow to how data ("information packets") flow through a network of (black box) components and only use traditional control-flow style programming for the most simple, atomic components. The book presents some convincing examples from business programming but I think the idea should work very well for other areas than text processing, certainly for image manipulation or sound processing.

The split between component creators and component assemblers (application programmers) is highlighted in the book, I hate it how often I tend to slip from one role to the other in regular programming, maybe such an explicit split would help a lot (well, or certainly a lot of experience will ...)

[1] http://www.jpaulmorrison.com/fbp/

Re: Why do web sites and software take so long to build? And why is it so hard?

#82

Post by b1daly »

A lot of good points, just a couple I thought of: Software is written to control hardware. In the end that's all a user can use. Computers are a new category of machine, they are super complex, easily reconfigurable general purpose machines. So writing softare doesn't really compare with other trades. In some sense it's like a design, but since the machine is already constructed the building of the particular machine that is the app in practice is trivial (usually, click install). If you compare to building a house,let's say the program is like the complete design. One could design a house that was reused over and over. In some degree this happens (prefab houses). But since the construction costs are high relative to the design, there is usually some room and incentive to customize, just a bit.

For a software app, if it's well written, the end product builds itself. The cost of this last step is virtually nothing. Since the design can be used repeatedly, the software development costs can be spread over a lot of end users. For example, take Excel. A "spreadsheet machine" is pretty useful, it's complicated. You could compare "spreadsheet machine" to a house (or maybe something like a tractor). Aside from tweaking, the software is done, the actual machine (a PC) is a commodity. It's a done deal, anyone who wants a "spreadsheet machine" can get one relatively cheaply. But now, since the machines are out there, along with a huge amount of cheaply distributed software (the OS,languages, compilers) everyone is tempted to make something new. How about a multimedia playback machine? Brand new things that haven't even been made before. Even though there are a bunch of general purpose tools available, it is the "newness" of the app that makes it hard to predict how long it will take to code. New problems don't have existing solutions, so no one can predict how long it will take to find them.

Also, since the actual machine is super complex, which is necessary for such a general purpose device, there is looseness in the designs, leading to many different ways of doing the same thing and inevitable bugs. There are bugs in the OS, so you are going to have bugs in your app. Think about how much more complex a PC Playing back a DVD is compared to a DVD player.

Re: Why do web sites and software take so long to build? And why is it so hard?

#83

Post by yaix »

trustfundbaby wrote: post #48

brilliant comment in the original post, just in case anybody missed it "When I had my bathroom remodelled, at first I thought it was organized amazingly well and why couldn’t software be like that? There was a designer from Expo, a primary contractor and subcontractors for tiling, painting, etc. a project workbook containing all the documents, including the design, and a logbook for every contractor to record their v…

Only difference, its really hard to do "git reset HEAD" when the bath tub didn't fit.

Re: Why do web sites and software take so long to build? And why is it so hard?

#84

Post by ansgri »

mbenjaminsmith wrote: post #68
bigiain wrote: post #57

Earlier quoted context omitted.

"It doesn't matter if you can write 1000 shippable lines of code in a day because tomorrow you'll spend most of your time hunting a bug in an external library. Suddenly your machine-like pace has been cut in half. " I'm there right now - why doesn't this Concrete 5 ecommerece plugin's Paypal addon work this week, when nothing's changed since last week??? It's not just you…

I was thinking a bit lower level than that but yeah, I guess you get my point. I spent a day with a bug that I found in OpenCV (a computer vision/machine learning library) that spooks me to this day. How it works internally is that for images to be processed you create a header (with usual image header info) which contains a pointer to the actual image data. There are two ways to do it: one you create the header and…

I wouldn't call OpenCV 'about as mature as a software project can get,' as it's actively developed library with lots of contributors in yet experimental domain (more about mid-level pattern recognition than low-level image filtering). And the documentation was not up-to-date last I checked (~2 months ago).

Bugs in really mature software (i.e. libtiff, libxml, sqlite) are very rare.

Re: Why do web sites and software take so long to build? And why is it so hard?

#85

Post by bgilroy26 »

robryan wrote: post #51
drpancake wrote: post #28

Earlier quoted context omitted.

Or alternatively you could say writing assembler is the software equivalent of building physical things by putting together individual molecules. Physical builders work at a higher level of abstraction too. I think the analogy is becoming a little stretched though.

In the plumbing example in the article, the plumber isn't first making pipes from raw materials. They get them pre built in standardized lengths, diameters, thicknesses and materials. Same can be said for the compiler providing a standard, generally agreed upon abstraction.

Wow, the pipe-building factory (generating standardized parts to be assembled by the plumber) to compiler building (generating standardized machine code to be assembled by the high level language programmer) comparison is really cool!

I feel like the ISO/ANSI etc standards-making bodies are where the analogy breaks into time and space.

Re: Why do web sites and software take so long to build? And why is it so hard?

#86

Post by mbenjaminsmith »

ansgri wrote: post #84
mbenjaminsmith wrote: post #68

Earlier quoted context omitted.

I was thinking a bit lower level than that but yeah, I guess you get my point. I spent a day with a bug that I found in OpenCV (a computer vision/machine learning library) that spooks me to this day. How it works internally is that for images to be processed you create a header (with usual image header info) which contains a pointer to the actual image data. There are two ways to do it: one you create the header and…

I wouldn't call OpenCV 'about as mature as a software project can get,' as it's actively developed library with lots of contributors in yet experimental domain (more about mid-level pattern recognition than low-level image filtering). And the documentation was not up-to-date last I checked (~2 months ago). Bugs in really mature software (i.e. libtiff, libxml, sqlite) are very rare.

That's probably more accurate. I was conflating a long history with maturity/stability. I would guess most ~13 year old libraries are broadly more stable and bug-free than something a few years old but I guess anything that is being actively adapted to new problems will have inherent instability.

I seriously doubt the code for reading images into an OpenCV-native format is being actively developed though. Anything being done in pattern recognition would likely not effect the library's basic IO stuff. What I was actually using it for (their RDP implementation, image thresholding) were likely pretty stable as well.

Re: Why do web sites and software take so long to build? And why is it so hard?

#87

Post by lifeisstillgood »

Tichy wrote: post #79
bigiain wrote: post #31

My opinion? Because as much as the software industry loves to adopt the work "engineer", almost none of the sort of websites and software discussed here on HN ever gets anything much like "engineering" in the sense of, say, "civil engineering" done. I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied custome…

As I've written in my unknown blog, I think there is another misunderstanding here: software development is not the analogy to building something according to a plan. It is making the plan. http://blinker.net/2009/08/31/the-architecture-analogy-for-p...

I have one of those invisi-blogs too

http://ciocookbook.com/philosophy/thecodeisthedesign.html

Jack Reeves is the man here though - writing source code is designing the product. Before code can be written you need to take all the trade offs, the decisions and the research (or just make wild assumptions).

Once you have source code, it has been designed.

Which is why instead of sending out a 12 page form telling a client they need to think harder about something they don't understand, put out version 0.1, and ask them what they want changed.

Re: Why do web sites and software take so long to build? And why is it so hard?

#88

Post by mgkimsal »

Quite an aside, but... I see many things going wrong with processes in businesses all the time. Simple processes - things like following up with me on an inquiry that I made, or correcting an invoice and sending it to me. And those get borked constantly. And almost inevitably, I get something like "oh sorry, our computers were down" or "oh, the computer made a mistake". Hrm...

But with software development we are in charge of the computer. When there's a problem with the computer, there's no ability to shrug it off or make excuses - it's our entire purpose on a project is to make the computer do stuff. Oh, no doubt we can (and do!) make excuses - the compiler versions are wrong, the tool doesn't work with this other platform, etc... but no one outside our field can even understand those problems, nor do they want to. It's just our fault, and we have to fix it.

Re: Why do web sites and software take so long to build? And why is it so hard?

#89

Post by kahawe »

trustfundbaby wrote: post #48

brilliant comment in the original post, just in case anybody missed it "When I had my bathroom remodelled, at first I thought it was organized amazingly well and why couldn’t software be like that? There was a designer from Expo, a primary contractor and subcontractors for tiling, painting, etc. a project workbook containing all the documents, including the design, and a logbook for every contractor to record their v…

To be fair, I can see one difference, though, at least here in Euroland: you have all sorts of standards and regulations that buildings and contractors have to adhere to and especially on bigger changes or constructions you would get an independent expert to check whether your contractors and subcontractors did obey all the rules and when they frakked up somewhere along the way, they have to fix it, typically on their own bill. So it is not enough that you have a house standing there, material, construction and architecture are evaluated as well.

Contrast that to software, it is typically more than enough that you can click through the links and it basically looks like it "works" and if you delivered all that on time then it is already better than probably 95% of all software projects out there and in all but the most critical cases (money or human lives involved) absolutely nobody will check what exactly you programmed and how terrible the underlying software structures and architectures are. If anything, symptoms are being checked and there are requirements you have to meet when money or lives are involved. But that is it then.

Re: Why do web sites and software take so long to build? And why is it so hard?

#90

Post by mistermann »

hdeshev wrote: post #71
trustfundbaby wrote: post #48

brilliant comment in the original post, just in case anybody missed it "When I had my bathroom remodelled, at first I thought it was organized amazingly well and why couldn’t software be like that? There was a designer from Expo, a primary contractor and subcontractors for tiling, painting, etc. a project workbook containing all the documents, including the design, and a logbook for every contractor to record their v…

Thousands times this. I've seen situations like that in all sorts of businesses and software projects aren't unique in their inability to deal with complexities and bad communication. On the other hand, learning to manage software projects gets you an almost unfair advantage at managing "simpler" and more mundane projects where everyone and his mother underestimates the complexities and all the ways things can go wro…

> learning to manage software projects gets you an almost unfair advantage at managing "simpler" and more mundane projects

I'd add that learning to write software also gives you an unfair advantage in managing the "complexities" of all other processes you'll encounter in day-to-day life, to the degree that it's (at least for me) incredibly frustrating going through life seeing inefficiencies and "incompetence" (not meant as an insult) everywhere around you.

Post reply on HN