Why software projects take longer than you think – a statistical model

With the context you've provided here, this is still an estimate-able task.

The steps to do this task are enumerable and would be similar across similar tasks.

I believe that the conclusion "software estimation can't be reliably done" comes from a standpoint that claims tasks like these ("reverse engineer X-Y-Z...") are somehow completely novel each time we encounter them. In my experience, they are not... even "soft tasks" like "design a Thing" and tasks with unknown details like "reverse engineer X-Y-Z..." -- when you do them more than once -- will be seen for the predictable beasts that they are.

But only if you approach them with the attitude that they aren't these novel beasts. That almost all of the work of the working practitioner is NOT like that of the researcher on the bleeding edge of some novel discovery.

A problem with our industry in this regard is that we don't understand the difference between learning and doing.

A new grad entering industry is going to be doing a lot more learning than doing -- or rather learning while doing.

I know from experience that explicit doing is highly predictable/estimate-able for the experienced software practitioner.

I have a suspicion that the learning side would be predictable, too, if our industry could do a better job of articulating what the practitioner skill set actually is -> then a pedagogy could develop and it could be said that learning X-Y-Z takes on average this long for a set of students. Etc.

But we as an industry do not seem to be near this level of clarity -- in large part because we don't even have the vocabulary to frame it this way... in terms of learning vs doing...

Now what this means for the new CS grad is not the best story. You'll rather have to play the chaotic game a little bit, which includes a mess of things like doubling estimates and working weekends or what-have-you depending on the work culture you find yourself within.

That ^^ in the short term.

In the long term what you should do is practice on your own:

1) ALWAYS privately plan and estimate your tasks to the best of your ability--on your own, you may not get benefit by exposing your "practice" to the higher-ups

1a) hint: your tasks should be as scoped/small as you can make them and they will look pretty simple, like this: "design widget X", "design API interface", "implement API interface", "test API interface", "implement widget Y", "learn how framework X lifecycle works" (yes! even learning tasks can be estimated!), and so on. The key is that you try to keep them under a day and ideally even smaller on the order of hours.

2) RECORD the time it takes and compare to your estimates. --> LEARN from this.

3) REPEAT

If you do this conscientiously you will find that your estimates improve dramatically over time and you'll be able to out-estimate your peers to their own wild confusion.

This skill set will pay off in your ability to deliver AND have weekends and hours with your family. Because you will be able to "see time" and protect yourself. You'll have protection and ammo even in the worst pressure-cooker environments because you will be able to say "No" with confidence. Or rather you will learn how to say "Yes" and what to say "Yes" to. (Ie. you will get really good at priority negotiation etc.) And you'll cultivate delivery trust with superiors and everyone will be happy.

The main reason you get these claims that "business just doesn't understand software" and "they put too much pressure on us" is because "business" side doesn't trust us. Once you get good at estimates and delivering on them, that trust is restored and everyone can be made happy.

BUT -- and here's the rub -- it takes time and conscientious effort and variety of experience to reach this level of confidence. My advice: Ignore all the philistines who say it can't be done....because they'll just try to talk you out of the effort that they weren't or aren't willing to do themself.

I used to work for a software company that did projects for clients and charged by the hour (a consultancy) and we frequently had to estimate projects with incomplete information in order to event get the contract in the first place, so they often turned into the actual final budget. Since I was the head of engineering I ended up doing a lot of sales and estimation for new projects. Besides just doing it a lot and gaining experience from many projects, there were a few other things that really helped me:

1) Software Estimation: Demystifying the Black Art by Steve McConnell (already mentioned by others).

2) His short video on "Targets vs. Estimates" was super helpful - we watched it with engineering, sales, and project management all together and had a good discussion afterwards. https://www.youtube.com/watch?v=FY9X21HA02w

3) Keeping a running list of "Things to Remember". Every time a project went astray or we encountered something during a project that I had failed to estimate (but potentially could have) it went on the list. That was useful to share with other too, when they did estimates.

I really like the discussion of standing up to those who ask for estimates. A clear understanding of what an estimate is and what it's for is important, as is a strong sense of professionalism. I would recommend "The Clean Coder" by Robert Martin for this. It's more about professional behavior than software practices. Especially his chapters on "Saying No" and "Saying Yes". I read it and discussed it with my team often. It helped us realize when we had to refuse to give an estimate because we lacked the necessary information to do so, rather than just guess or make something up.

ah. I see now what's gone wrong. I thought I commented on the parent comment about

> under the (mistaken) belief that more hours = more software. reply

I'd delete it if I could.

What I want to know is why the software developer is expected to give estimates. Estimating how long common tasks take seems like something I would expect a software manager to be able to do effectively and care about. If you’re just a monkey who is asking everyone to do the planning and estimation job for you, I’m frankly unsure why the company allows you to collect a paycheck.