Live data from Hacker News

Why software projects take longer than you think: a statistical model (2019)

erikbern.com

151–160 of 178 posts

Re: Why software projects take longer than you think: a statistical model (2019)

#151
imposing uncertainty on task estimates is a natural way to push back on management pressure to low-ball estimates

if everything has a potential 2x blowup, you monte carlo all the tasks to produce a distribution of finish dates, not a single 100% delivery date

institutions naturally push risk away from political power, so mostly this uncertainty is applied to line workers by managers. but senior managers have an incentive to push uncertainty to middle managers, not to line workers. uncertainty is a management trick to do this.

Re: Why software projects take longer than you think: a statistical model (2019)

#152
post #58
post #20

Earlier quoted context omitted.

> What I find ironic is that everyone in the industry knows that estimating is very hard, thus estimations are nearly useless, yet everyone insists on the importance of planning and on basing such planning on void estimates. Obviously, planning is critical. Planning means resource allocation. How is this not a critical aspect of any project? The mistake you're making is presuming that if estimates are not crisp then…

> Obviously, planning is critical. Considering it very often fails even for successful projects, obviously, it is not. > This belief is detached from reality. Failing to provide estimates means a failure to scope how much work is required Your line of reasoning is detached from reality. The real world is full of project leader who do everything in their power to know as soon as possible how much work and time are req…

> Considering it very often fails even for successful projects, obviously, it is not.

You are using a very personal and peculiar definition of failure, which is not shared by anyone else.

Back in the real world, companies hire project managers and put them in charge of projects without any concern or regret regarding the methodology. Do you think the whole world is wrong and you're the only one getting it right?

> Your line of reasoning is detached from reality.

Please explain exactly why you think scoping efforts have no relation with estimates.

> The real world is full of project leader who do everything in their power to know as soon as possible how much work and time are required,

Yes, it's their job. What exactly do you struggle to understand?

> still they never get the right answer until the project is completed.

Yes, what about it? Exactly what do you have trouble understanding?

Re: Why software projects take longer than you think: a statistical model (2019)

#153
post #58

Earlier quoted context omitted.

> Obviously, planning is critical. Considering it very often fails even for successful projects, obviously, it is not. > This belief is detached from reality. Failing to provide estimates means a failure to scope how much work is required Your line of reasoning is detached from reality. The real world is full of project leader who do everything in their power to know as soon as possible how much work and time are req…

> > Obviously, planning is critical. > Considering it very often fails even for successful projects, obviously, it is not. This is not a relevant strawman (if it fails for some, it works for none?). Working without a plan isn't how humans work. You want to argue about scale, that's fine, but far from absolute. > > This belief is detached from reality. Failing to provide estimates means a failure to scope how much wor…

> This is not a relevant strawman (if it fails for some, it works for none?). Working without a plan isn't how humans work. You want to argue about scale, that's fine, but far from absolute.

OP clearly fails to understand that the role of a project manager is to navigate ambiguity and uncertainty to deliver results in spite of them, and project managers are fully aware there is no such thing as crisp estimates and absolute certainties. Therefore OP puts strawmen regarding absurd notions of rigour that no one in the world adheres to.

I mean, hasn't OP noticed Project Manager's don't carry around stopwatches?

Re: Why software projects take longer than you think: a statistical model (2019)

#154
This is great, and reminds me of estimating random walks and stochastic processes.

Basically, the time estimates for uncorrelated steps can be added together, but if there's correlation then the times should be multiplied. Because a change in one step can require a rewrite of one or more of the other steps.

Which means that a 3 step process with average step time t can take t * 3! (factorial) units of time instead of t * 3.

Also it might have been exponential instead of factorial, I wish I could remember.

Re: Why software projects take longer than you think: a statistical model (2019)

#155

Earlier quoted context omitted.

Indeed. Development should not begin until all requirements have been finalized. Every project I've ever been on that had wildly wrong estimates was because management didn't actually plan jack shit either due to inexperience, laziness, malice, or all of the above. They then play the blame game to try to keep their job, and man it's so hilarious to witness the sheer carnage as everyone tears them a new asshole nowada…

> Development should not begin until all requirements have been finalized. I agree, but this runs directly counter to Agile methodologies which are all the rage these days.

"How to end up shipping your project three years after the planned release"-starter-kit.

Re: Why software projects take longer than you think: a statistical model (2019)

#156

Start from an empty file and write to an imaginary API or pseudocode what you actually want to do and coding is easy and fast! I think the high level imaginary API/pseudocode of any system is straightforward and fast when you start from scratch. (Because you're not focused on detail and just intent) But when you start from an existing codebase, existing infrastructure, existing APIs, it slows to molasses. You've got…

> way of saying that the behaviour is the product of every statement about the behaviour that is desired, not an explicit instruction of what to do next I believe you'll enjoy reading "Notes on the Synthesis of Form" by Christopher Alexander [1]. There are many ideas in it, but at the later chapters the author proposes a system for solving design problems as a large graph of interacting requirements, where smaller su…

Thanks a lot for your comment!

I once read "The Timeless Way of Building" by Christopher Alexander but not "A Pattern Language"

I actually owned that book "Notes on the Synthesis of Form" but I never read it, I gave all my books away. Maybe I'll have to get it after I finish reading "Database System Concepts" which I am slow to read.

I would like to hear YOUR ideas too! Send me an email, I hope you shall tell me your ideas and thoughts and we can bounce off eachother. Please (and anybody else reading) do ping me - I love reading other people's thoughts and encouraging others.

The idea of using graphs to solve problems and breaking them into re-useable patterns resonates with me, I didn't realise that particular book had that in it. One of my programming projects was using the A* algorithm to do code generation in Python ( https://replit.com/@Chronological/SlidingPuzzle3 ) based on generating graphs of potential instructions.

> Sounds similar to this idea of yours, just with a key difference in that the author sees it as a two-pass solution: first you "carve" out your solution by establishing a context boundary, and then do you "fill" it with an implementation - such that the solution is the minimal implementation that fits the context, and not anymore.

Thanks for sharing that. It reminds me of Java interfaces but more powerful.

Re: Why software projects take longer than you think: a statistical model (2019)

#157
post #54
post #22

Earlier quoted context omitted.

I think the fundamental misunderstanding is that we aren’t building a product. We are doing research and development. We are figuring out how to build something novel, otherwise the customer could just go out and buy it already. Once we’re done with discovery, the computer builds it. So, the customer is like any other who is paying for research and development. It’s not a blank check, but they should go into it witho…

This is hubris of the field of software engineering. The no-code market and the existence of tools as old as MS Access prove that there are software projects for which a better analogy is not R&D, but construction. I do note that construction projects are also not good at estimating, but it's folly to suggest that at some level we are not _manufacturing_ rather than _developing_ a widget.

If we're going to talk about construction as the metaphor, then it's done by the computer (e.g. compilation), and it's so cheap or trivial on many projects that developers will ask the computer to build the software hundreds or thousands a times a day (e.g. via test-driven development). In this metaphor, I think the developer would be more akin to a designer or architect, but where the building in question is at least partially novel. If it's not novel in product, people, process, etc., then the client could just go out and buy or use one that has already been produced. The developer wouldn't be necessary.

Re: Why software projects take longer than you think: a statistical model (2019)

#158

All this smokes and mirrors, when the reality is: if you gave accurate estimates, you'd get reprimanded, not get the project approved, etc. Same reason that infrastructure project go over budget. The way anything actually gets done is by initially being overly optimistic, ignoring potential future problems, getting the project approved, then lock it in via sunk costs, now as problems turn up, you can claim it could n…

I'd also add once the project gets going and leadership realizes it's starting to fall behind they'll ask what they can do to help. I'll reply w/ very real answers like we need wireframes, or language strings or this endpoint isn't ready and we don't even have a spec blah blah but that never seems to go anywhere. Anyone w/ real power is so far out of the weeds and usually busy w/ other things and those of us in the k…

Very similar experience in my ~decade of experience in software dev as an entrepreneur.

Re: Why software projects take longer than you think: a statistical model (2019)

#159
post #78

Earlier quoted context omitted.

I'll have a go: If you ask developer A how long it will take, the best outcome for them is that they AND EVERYONE ELSE go high. So they should go high. BUT if everyone else goes low, but they go high, then they look bad. So they're forced to go lower. But then not too low, or they won't be able to deliver in time. So perhaps this settles on an estimate that's "as low as possible but no lower"?

> the best outcome for them is that they AND EVERYONE ELSE go high But it's really not. The best outcome for them is that they are 100% accurate. Now, I understand that that's generally not feasible, but assuming that you need to pad everything by a huge amount is part of the problem. As a general rule, I've found that the kind of answer that gets the best response is on the order of "about X time to complete develop…

Very well put and so true. You can exercise much more leverage by exuding professionalism and experience.

I've successfully talked stubborn execs out of bad ideas by framing things in terms they understand (risk, costs, reputation). And by doing so earned more respect which I can further leverage in the future.

Re: Why software projects take longer than you think: a statistical model (2019)

#160

Another factor here is in a meeting-heavy agile environment, many engineers can't even give a good forward estimate of the number of "in the zone" hours they will have in the coming week. Will I be on 0 outage calls or 5? Doesn't matter if I am on pager duty rotation or not.. Will I get pulled into 3 calls this afternoon or 0? Is my boss gonna come bother me with some urgent non-ticket task tomorrow morning? Etc. I'm…

Another factor here is in a meeting-heavy agile environment, many engineers can't even give a good forward estimate of the number of "in the zone" hours they will have in the coming week. As a developer my highest level of productivity was on a 2 person team with a bi-weekly meeting with our manager. We would have impromptu meetings when needed, but for the most part I was free to work 6-15 hours in a row whenever I…

This is a big reason I'm still 100% satisfied being a solopreneur and working with specific clients I've chosen with lots of autonomy. Soooo much more productive and effective in terms of Getting Things Done™
Post reply on HN