Why software projects take longer than you think – a statistical model • Hacker News Board

Re: Why software projects take longer than you think – a statistical model

#271

Post by naavis » Tue, Apr 16, 2019, 8:27 PM UTC

artworx wrote: post #150

undergrowth54 wrote: post #29

An important aspect of being a professional software engineer is having the backbone to sometimes say things like: - “I don’t know yet enough about the problem to give you even a rough estimate. If you’d like, I can take a day to dig into it and then report back.” - “This first part should take 2-3 days. 5 on the outside. But the second part relies heavily on an API whose documentation and error messages are in Chine…

I highly recommend the book "Software Estimation: Demystifying the Black Art". I work for an outsourcing company and part of my job is to come up with estimates and it helped me deal with clients and managers. The book contains a quiz that we used as part of a training exercise with management and the results were hilarious. Here is an online copy: https://scrumandkanban.co.uk/how-accurate-are-your-estimates... pleas…

I just started reading the same book. Seems really nice so far!

Re: Why software projects take longer than you think – a statistical model

#272

Post by regular_person » Tue, Apr 16, 2019, 8:30 PM UTC

maerF0x0 wrote: post #229

jnaddef wrote: post #210

Earlier quoted context omitted.

Can you add sources to justify this statement?

I was trying to say, In my experience controversial comments are better received w/ sources to back the items. At the time I made my comment the Parent comment had a negative score and I was trying to be helpful to commandlinefan .

Just curious, what sort of source would you expect for that? An article from Manager's Monthly about tips on tricking your employees to do free work?

Re: Why software projects take longer than you think – a statistical model

#273

Post by 52-6F-62 » Tue, Apr 16, 2019, 8:33 PM UTC

afarrell wrote: post #108

nahname wrote: post #104

Earlier quoted context omitted.

As long as software is a cost center to your company, it will be treated this way. Trying getting a job at a company where software is a core concern. Ideally with a CEO that is not from a marketing/business background.

For more on this sort if thing, Patrick McKenzie’s writing is good: - https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-pr... - https://www.kalzumeus.com/2014/04/09/what-heartbleed-can-tea...

In my experience, that's not necessarily actionable advice. (the first link)

Re: Why software projects take longer than you think – a statistical model

#274

Post by agumonkey » Tue, Apr 16, 2019, 8:33 PM UTC

mntmoss wrote: post #263

afarrell wrote: post #218

Earlier quoted context omitted.

The lack of a good “intro to task and small project management” MOOC or book is a serious problem for our industry’s ability to onboard junior engineers. I wish I knew how to write one, but TBH I don’t know any principles myself except one. Lots of people say “just do it and multiply by 3” without appreciating that The important part of estimating tasks is coming up with the task breakdown and identifying the risks.

The best you can do in most scenarios is to research comparable projects and calibrate against them. "They shipped in six months, therefore we can ship in six months."

Both the mentionned idea are natural, but it's ultra sad that we can only live by experiment like craftsmen for previous eras. Not that I mind craftmanship but at least school should write it in bold letters so you're not seeking a mathematical model.

Re: Why software projects take longer than you think – a statistical model

#275

Post by 52-6F-62 » Tue, Apr 16, 2019, 8:36 PM UTC

eikenberry wrote: post #260

afarrell wrote: post #108

Earlier quoted context omitted.

For more on this sort if thing, Patrick McKenzie’s writing is good: - https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-pr... - https://www.kalzumeus.com/2014/04/09/what-heartbleed-can-tea...

> In the real world, picking up a new language takes a few weeks of effort and after 6 to 12 months nobody will ever notice you haven’t been doing that one for your entire career. Spoken as someone who has never taken the time to fully master a programming language and, from the sound of it, has never worked with someone who has either. The difference between someone who has spent 6-12 months with a language compared…

Thanks for posting that Norvig post. It's fantastic.

Re: Why software projects take longer than you think – a statistical model

#276

Post by rightbyte » Tue, Apr 16, 2019, 8:51 PM UTC

teddyh wrote: post #60

According to Joel Spolsky¹, programmers are generally bad at estimating, but they are consistently bad, with the exact factor depending on the individual. So by measuring each person’s estimate and comparing it to the actual time takes after the fact, you can determine each person’s estimation factor, and then when they estimate again, you can get a pretty reliable figure. 1. https://www.joelonsoftware.com/2007/10/26…

"As estimators gain more experience, their estimating skills improve. So throw away any velocities older than, say, six months."

I don't know how Spolsky is (was) as a boss, but it's is way easier to just adjust your time reports according to the estimates and make your boss happy than to actually try for no benefits to either you or the project. As long as the estimation errors are white noise-ish and in linear relation to actual task length, which is the assumption for his model to work, manipulated time reporting and not manipulated make no difference.

I've noticed that when programmers are introduced to Agile they initially are bad at time estimates until they learn that you actually can report time according to the estimates and no-one will ever notice. Extra points for looking on the burndown chart before reporting time to make it smooth.

"Some developers (like Milton in this picture) may be causing problems because their ship dates are so uncertain: they need to work on learning to estimate better. Other developers (like Jane) have very precise ship dates that are just too late: they need to have some of their work taken off their plate."

Milton is probably doing fair estimates. When planning four months into the future you probably can say where you are within 2 months with 50% probability (or what ever boxplot he uses) of where you think you will be at that time. Janes reporting is obviously manipulated to match the estimates.

Re: Why software projects take longer than you think – a statistical model

#277

Post by afarrell » Tue, Apr 16, 2019, 9:10 PM UTC

SkyBelow wrote: post #254

golergka wrote: post #86

Earlier quoted context omitted.

> Does it all take that long? That's just a question. Asking questions is OK. If you have such a high need to please other people and such a low self-confidence that you will change your answer simply if someone asks you twice, that's the problem with you, not them. I say it as someone who have been like that and have changed it through therapy. I was a nervous mess and always thought that people around me don't like…

>Asking questions is OK. Is it? Questions have built in assumptions and implications. Most questions don't have enough to be of concern, but some questions definitely imply things beyond the question itself. When asked 'Does it all take that long?', many hear an implied 'Make it not take so long.' command. It is not implied all the time, but it is implied often enough. This does not mean someone has low self-confiden…

If you hear a 'Make it not take so long.' command, thats not an unreasonable interpretation. However, it is also one that should be questioned because it clashes with how software projects work.

If you go from "my boss gave me a command" to "I must execute the command and not question it" without exercising independent judgement, then I would assert that you do have low self-confidence. Whether someone needs therapy is a deeper question that we should be hesitant to answer for another person. But therapy was tremendously helpful for me in realising how my personal and professional life were held back by my reluctance to trust my own judgement or endure the discomfort that can come with honest, respectful, collaborative confrontation.

Re: Why software projects take longer than you think – a statistical model

#278

Post by bloorp » Tue, Apr 16, 2019, 9:39 PM UTC

One big factor that I don't THINK the article touches on is how we are using 'space' metaphors (speed and distance) for our work that's very 'time' based (working productivity and duration). And I think when we estimate, we try to estimate distance/duration but we forget that we're really trying to estimate our speed/productivity.

Where that gets painful is, say you estimate you can get something done in two days. In reality, you're twice as fast (there's that speed metaphor), so you actually get it done in one day. Yay, you saved a day! Now assume you're twice as slow as your two day estimate. Boo, you spent TWO days longer. So in terms of duration, getting it incorrect in the painful direction seems like a bigger mistake.

I don't think this is the same phenomenon as the author's mean vs. median dilemma. I'll bet both the mean vs. median and the productivity vs. duration dilemmas are real factors though.

Re: Why software projects take longer than you think – a statistical model

#279

Post by scruple » Tue, Apr 16, 2019, 9:43 PM UTC

adrianmonk wrote: post #211

dre85 wrote: post #208

I'm blown away on a regular basis by how long it takes to write software that in my opinion is super easy. Usually it just comes down to fuzzy/missing/changing requirements. A lot of times people know that they want something, but they don't know exactly what. Or they're absolutely sure they need feature x, but then later they realize they don't, but they missed out on developing other more fundamental features.

I have developed a belief about this: people don't know what they want until you show them what they said they want. Then it's immediately obvious to them what they wanted instead. This suggests that demos and mock-ups might be a valuable tool. The sooner you can get someone to try something, the sooner they can tell you what direction they really wanted you to go in instead, and the less time you waste.

They want everything but they can't even define a starting point. It's insanity. I've been going back and forth with a customer about this (through my product and sales team) since August of last year. At first the priority was high and it seemed like the scope was reasonable. But no one was making moves or delivering on my requests / asks / concerns. I suggested a phased approach, so that we could _be_ agile and get _something_ in front of the customer for feedback. But before I even got there the requirements changed again and again and again and again and again...

Thankfully all of this specific work is easy to segregate away from the rest of my team. It's proven to be very toxic and morale draining. If my entire team was involved, I've no doubt that we'd have lost people by now.

And the real kicker: It turns out that right now we can't even deliver on the first pass that I had originally suggested, despite me being basically done with my teams piece, because some other team in the company was loose with their language and convinced a bunch of product and sales folks that, yes, of course they had what the customer was asking for. They didn't. They still don't. I'm convinced today that they never will. Bunch of fucking sycophants.

I have no idea how these things happen but recently I've become convinced that this sort of confusion-on-all-fronts is just par for the course in this industry today. The only work that I've been able to do in the past couple of years that was well understood and easy to articulate, and as a result capable of being completed mostly on time and within scope, was born out of a select few individuals being able to identify a real problem and a real solution and grinding away at it in a controlled fashion. But that seems to be rare and not at all "how things are done."

Re: Why software projects take longer than you think – a statistical model

#280

Post by squarecog » Tue, Apr 16, 2019, 9:52 PM UTC

koonsolo wrote: post #59

undergrowth54 wrote: post #29

An important aspect of being a professional software engineer is having the backbone to sometimes say things like: - “I don’t know yet enough about the problem to give you even a rough estimate. If you’d like, I can take a day to dig into it and then report back.” - “This first part should take 2-3 days. 5 on the outside. But the second part relies heavily on an API whose documentation and error messages are in Chine…

> They probably don’t want you to lie, but there is a small chance that they pruessure you to. In my experience, most of the time there was pressure. I remember a funny conversation between my project manager and a colleague: PM: "We need to have X as soon as possible, how long will it take you?" Dev: "Oh that's, easy, I can finish it tomorrow." PM: "Tomorrow??? That's impossible, because it needs a and b, no?" Dev:…

Can you explain which part, in your opinion, is facepalm worthy?

In your example conversation, the developer starts off by demonstrating that they are not good at thinking through how long things take them, by giving an impossibly optimistic estimate. Even the PM, whose job is not to figure out how to build stuff or how long building stuff takes, knows the estimate is wrong, and pushes back on it, pointing out dependencies the Dev appears to not have considered.

The dev comes back with another estimate, also provided on the spot, without actually thinking through all the tasks, problems, and potential ways to streamline / accelerate. The PM, having just had the dev's lack of thorough thinking with regards to complexity estimation amply demonstrated, asks the Dev to think more carefully, a third time.

Are we facepalming at the dev who can't be bothered to actually consider what it will take to get the thing done, or the PM who for some reason accepts the 3rd estimate, even though the dev appears to just be randomly generating numbers? Both?