Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why do web sites and software take so long to build? And why is it so hard?

scottporad.com

Post reply on HN
Print view View original

61–70 of 159 posts

Re: Why do web sites and software take so long to build? And why is it so hard?

#61

Post by rrreese »

ExpiredLink wrote: post #42
bigiain wrote: post #31

My opinion? Because as much as the software industry loves to adopt the work "engineer", almost none of the sort of websites and software discussed here on HN ever gets anything much like "engineering" in the sense of, say, "civil engineering" done. I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied custome…

> I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied customer at the end. Overwhelmingy, some (or most) of the design gets "made up as we go along". Even on projects with several small forests worth of up-front documentation, there's almost always large areas of vagueness or outright contradictory requireme…

Right, I've worked on many projects where it wasn't really possible to gather all requirements up front. This is hugely frustrating to everyone involved but the fact remains that until you have at the very least a working prototype you will often not be able to generate a final requirements document.

On the other hand when requirements are nailed from the very start, the development proccess has been swift, well executed and largely bug free.

While developers love discussing the pros and cons of different languages, libraries and frameworks I often think that more work should be put into developing ways of better capturing requirements, as this has such a huge effect on the amount of time a project will take.

Re: Why do web sites and software take so long to build? And why is it so hard?

#62

Post by thwarted »

jurre wrote: post #41
bigiain wrote: post #31

My opinion? Because as much as the software industry loves to adopt the work "engineer", almost none of the sort of websites and software discussed here on HN ever gets anything much like "engineering" in the sense of, say, "civil engineering" done. I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied custome…

I think you might be underestimating how much custom code has been written by companies like twitter and Facebook, think php to c++ compilers, databases, rpc servers, log frameworks etc etc. even if all of the designs of all of that software was given upfront I doubt any small team could build it within a month or so.

And the reason that custom code gets written is because it's specific to that environment. The wide majority of the software available is focused on either a very generic target or a very specific one. If you use something that is very generic, you need to customize it because one size does not fit all. And you can't use the very specific one because it doesn't match your use case. Or what's available is designed for a single webserver and a single database server, so it's really tempting to use that when you're small then you'll most likely need to replace that at scale. And by then, you're writing something that is specific for your use case, which you've discovered has grown dynamically and organically based on changing business and technical requirements.

It's difficult to take technology or software from one place and transplant it into another. The best we can do is take what we learn on past projects and past companies and keep those lessons in mind when making new decisions.

Re: Why do web sites and software take so long to build? And why is it so hard?

#63

Post by roel_v »

"How often do you think two plumbers argue over the right way to plumb a bathroom? Almost never!"

I don't think this guy has ever been to a construction site. The people who work for me argue about stuff like this all the goddamn time. About the most minute details like making a connection from a wall point to the drain going first straight then left, or first left then straight, and a million other things that are completely irrelevant in the grand scheme of things. So just like software. They also come up with the same justifications for doing it one way or another - 'oh but it'll be easier later on' (programming language nerd wars), 'this way saves work', 'this way is more robust'. Get 10 of them together and you get 11 opinions. And you know what - in the end, the best ones are the ones who don't come up with different solutions every single time, but who just get stuff done, good enough, in time and within spec. Just like software.

Re: Why do web sites and software take so long to build? And why is it so hard?

#64

Post by kamaal »

roel_v wrote: post #63

"How often do you think two plumbers argue over the right way to plumb a bathroom? Almost never!" I don't think this guy has ever been to a construction site. The people who work for me argue about stuff like this all the goddamn time . About the most minute details like making a connection from a wall point to the drain going first straight then left, or first left then straight, and a million other things that are…

>>the best ones are the ones who don't come up with different solutions every single time, but who just get stuff done, good enough, in time and within spec. Just like software.

This looks like some who uses Dynamic/scripting languages to get the job done, like Perl. Especially 'get stuff done', 'different solutions every single time'(TIMTOWDI), 'Good enough' seem to perfectly match that definition.

Re: Why do web sites and software take so long to build? And why is it so hard?

#65

Post by vlisivka »

ExpiredLink wrote: post #42
bigiain wrote: post #31

My opinion? Because as much as the software industry loves to adopt the work "engineer", almost none of the sort of websites and software discussed here on HN ever gets anything much like "engineering" in the sense of, say, "civil engineering" done. I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied custome…

> I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied customer at the end. Overwhelmingy, some (or most) of the design gets "made up as we go along". Even on projects with several small forests worth of up-front documentation, there's almost always large areas of vagueness or outright contradictory requireme…

Use Agile for R&D, user Waterfall for D only.

E.g. use Agile and smart guys with good knowledge of application domain for first version, then drop it and rewrite from scratch using Waterfall and outsource professionals.

Re: Why do web sites and software take so long to build? And why is it so hard?

#66

Post by eric-hu »

fridek wrote: post #54
bigiain wrote: post #31

My opinion? Because as much as the software industry loves to adopt the work "engineer", almost none of the sort of websites and software discussed here on HN ever gets anything much like "engineering" in the sense of, say, "civil engineering" done. I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied custome…

I was working as a freelancer for a long time and I had a quite simple solution for lack of documentation. Before the first meeting I'd send a customer 12-page form to fill which covered most of the "what and why" of the project. It quickly turned out that most of people have no idea whatsoever about the product they wanted to buy. I know that it's partly our job to guide people through rough path, but if all custome…

I, too would like to see this form. It would provide a lot of guidance for my company. We're knee deep in the consequences of scope creep right now.

Re: Why do web sites and software take so long to build? And why is it so hard?

#67

Post by kamaal »

maaku wrote: post #19

Except that software is written by machines these days. I stopped writing in assembly language after it stopped being cool in the 90's. Even still, you'd have to be hand-crafting machine code in a hex editor to really get all the machines out of the loop. These days I write in an interpreted language using complex libraries that handle a multitude of protocol choices for me. Even if you argue that each of those tools…

>>These days I write in an interpreted language using complex libraries that handle a multitude of protocol choices for me.

Those days are gone too. In fact currently you just have to know how to press ctrl+space at the right time and wait for intellisense to do the magic. Nearly 99.99% of the Java world works like that.

In other words, these days we learn how to learn tools that write programs.

Re: Why do web sites and software take so long to build? And why is it so hard?

#68

Post by mbenjaminsmith »

bigiain wrote: post #57
mbenjaminsmith wrote: post #50

Earlier quoted context omitted.

"It doesn't take very long to 'build' the software or the website (obvious 'large scale' projects like OSes excepted)." A lot of assumptions have to be met for that to be true. It shouldn't take very long provided: 1. The programmers are very familiar with the tools. 2. The programmers are very familiar with all necessary 3rd party components. 3. The 3rd party components account for 99% of functionality. 4. The progr…

"It doesn't matter if you can write 1000 shippable lines of code in a day because tomorrow you'll spend most of your time hunting a bug in an external library. Suddenly your machine-like pace has been cut in half. " I'm there right now - why doesn't this Concrete 5 ecommerece plugin's Paypal addon work this week, when nothing's changed since last week??? It's not just you…

I was thinking a bit lower level than that but yeah, I guess you get my point.

I spent a day with a bug that I found in OpenCV (a computer vision/machine learning library) that spooks me to this day. How it works internally is that for images to be processed you create a header (with usual image header info) which contains a pointer to the actual image data. There are two ways to do it: one you create the header and then create the image data separately and the other you create the image all in one go. I ended up in a situation where the image (it was going through a couple of different filtering algorithms) would get processed and be displayed regardless of whether or not I had set the pointer to the image data. That image data was already in memory since it was getting read from disk in a separate step, but there was really no way for the library to know its location in memory. I spent a day trying to sort out what was going on -- and threw my own schedule -- because I had to assume something was seriously wrong and would lead to other unintended consequences. I never did figure out what it was but after a lot of testing I decided it wouldn't cause any other problems (it never did, the library I was writing is still the core of a product I'm selling today).

If you don't know it OpenCV is about as mature as a software project can get. It was most likely the best tool for the job I had to tackle. But even with the library's stability and widespread use there was just plain voodoo that set a project I was working on back. Other 3rd party tools can be much worse. I don't do any web stuff but I have had to integrate Facebook Connect into products. Interfacing with Facebook can be a nightmare. Their API changes fairly regularly and the documentation can be a mess. Even using a 4th party, regularly updated, helper library I've seen bugs that took ages to sort out.

And that's just talking about integrating 3rd party, hopefully mature, components. I don't care how good you are, writing new code means a lot of testing and debugging. I've never seen new code that didn't introduce some voodoo of its own into a project.

Re: Why do web sites and software take so long to build? And why is it so hard?

#69

Post by cateye »

The important question that needs to be distinguished is:

How much R&D is involved in your project?

It doesn't matter if it is a software project or a physical project. If the percentage of unknown terrain is high, the project will become complex.

When a very progressive and innovative bridge/house needs to be build with new materials and a unknown construction etc., the project will be complex and you can't design it (completely) upfront.

For example, if your software project is just a CRUD application (without complex business logic), a qualified and experienced software developer can estimate it and deliver it on time. (There are a lot of frame works and tools already available for this kind of software. So, nobody doesn't need to reinvent the wheel.)

If the wheel is not invented yet for your problem, yes, it can take a while and your luck is in the hand of gods.

The other issue is that there are a lot of developers that can't get working any shit. But incompetent people are everywhere.

Re: Why do web sites and software take so long to build? And why is it so hard?

#70

Post by redthrowaway »

lechris48 wrote: post #58
ExpiredLink wrote: post #42

Earlier quoted context omitted.

> I think in my ~20year career, I've had only 3 projects that were specified well enough up front that we just "built it according to the plans" and had a satisfied customer at the end. Overwhelmingy, some (or most) of the design gets "made up as we go along". Even on projects with several small forests worth of up-front documentation, there's almost always large areas of vagueness or outright contradictory requireme…

And that is what actually differentiates software engineering from other engineering disciplines. You cant't simply embrace change when building a house or you'd have to tear it down multiple times during building...

I've worked in construction, and that's actually not the case at all. What happens is very similar to what happens in software development: someone realizes that the spec is borked; the subcontractor and general contractor get together to look it over, and come up with a kludge that fixes the problem as easily as possible.

If the change is at the customer's request, they get charged a nice premium and the necessary changes are made. At the end of the day, everyone still gets paid regardless of how often the customer changes their mind.

This, then, is perhaps what the trades do better than us: they're very upfront with the customer about added costs. "Yes, we can make your bathtub a jacuzzi. It'll cost you $2500 more and the job will take a couple days more. Do you want to go ahead with your change?" Because there is a clear and complete plan that the customer signs off on before construction begins, it's very easy to show them what a change in requirements entails.

Everyone can envision what knocking a wall down means, and can see it will cost money and time. Many decide, upon reflection, that they really don't care enough to pay that price. Others go nuts and end up with expensive projects that take forever. Either way, the contractors get paid.

Post reply on HN