Software effort estimation is mostly fake research • Hacker News Board

Re: Software effort estimation is mostly fake research

#111

Post by randomsearch » Mon, Jan 18, 2021, 8:56 PM UTC

BowBun wrote: post #100

It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates. According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather. If industry…

I'd say the problem is more from the academic side. If good data isn't available, then academics should not be publishing papers on toy data. It's meaningless. The goal is not to publish papers but to advance science.

Re: Software effort estimation is mostly fake research

#112

Post by randomsearch » Mon, Jan 18, 2021, 8:59 PM UTC

shihab wrote: post #27

I have been involved in Software engineering research a bit, even have a first-author short paper. I was struck by just how much pointless, low-effort papers there are in this domain. People have been researching about bug prediction for over two decades now, and judging by the paper quantity, this isn't a niche area. Yet how many organizations do actually employ those systems in real-world? Can't comment on industry…

Couldn't agree more. Software engineering is a very practical, fast-moving subject, and that makes most academic work even more irrelevant - software engineering academics often don't have the first clue about what real software engineering is about, because they don't have recent industrial experience.

Re: Software effort estimation is mostly fake research

#113

Post by [deleted] » Mon, Jan 18, 2021, 8:59 PM UTC

randomsearch wrote: post #111

BowBun wrote: post #100

It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates. According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather. If industry…

I'd say the problem is more from the academic side. If good data isn't available, then academics should not be publishing papers on toy data. It's meaningless. The goal is not to publish papers but to advance science.

[deleted]

Re: Software effort estimation is mostly fake research

#114

Post by robertlagrant » Mon, Jan 18, 2021, 9:00 PM UTC

durandal1 wrote: post #21

jakubp wrote: post #2

If someone can conclusively teach inexperienced programmers good approach to estimates (methodology) + help embed this into sales process of a software house-type company, I know some folks who'd love to have this :) My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software dev…

Apple executes massive feature releases on a yearly waterfall-like schedule across hundreds of teams. It requires honesty, transparency, strong leadership (ruthless prioritization), strong cross-team goodwill, strong cross-team collaboration as well as a solid body of experienced top-tier engineers. Other than that there's no magic. While there are teams within apple that does scrum/agile, none of the core OS/framewo…

How do you know it's waterfall-like? If you ship working software every 2 weeks and every 26th one you release, hooray, you have yearly software that's super likely to work well.

Re: Software effort estimation is mostly fake research

#115

Post by corry » Mon, Jan 18, 2021, 9:01 PM UTC

snarf21 wrote: post #98

didibus wrote: post #82

The issue with estimates are expectations. While nobody acknowledges it, you're not actually asked for an estimate, you're being asked for a quote. The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and…

The issue is that you are being asked to estimate something that has never been done before. Even houses always go over time and money and that is fairly straight forward. These days, I only give estimates in terms of units but without numbers. Hours, days, weeks, months, quarters or years. Some relatively small number of those units. If you want a quote it will take an extra 1/4 of the estimate worth of time for an…

'I really wish we would treat sales people the same way. "How much money is this contract for? What date will it be signed?"'.

In well-run sales teams, that is exactly what happens. There is a constant move towards more accuracy and tightness in forecasting.

In poorly run sales teams, the reps basically set their owns quotas (padded, of course, to mitigate against massive uncertainty) and then are very imprecise at managing the WIP. Which sounds like a poorly run engineering team too. In sales, though, at least there is a very objective measure ($$$ closed) at the end of the period. Also, the period is fixed typically (months, quarters, etc).

Engineering is obviously different... what's the measure? LOCs? Obviously not...

Edit: just to be clear, I'm saying that well-run sales teams and well-run engineering teams probably have a lot in common, not that they are exactly the same.

Re: Software effort estimation is mostly fake research

#116

Post by jeffreygoesto » Mon, Jan 18, 2021, 9:02 PM UTC

didibus wrote: post #107

quickthrower2 wrote: post #90

Earlier quoted context omitted.

That is the degenerate scenario, but it’s not always true. Often I see this haggling down of estimates and then microagression if estimates are not met but no need to work for free to make up for it. And I’ve also seen wise use of estimation but that is rarer!

> And I’ve also seen wise use of estimation but that is rarer I've got a question for you here, what is a wise use of estimation? What problems are estimates the ideal solution for? In the context of software development? I can think of only two: planning headcount and priorities (aka planning the order in which you do things). When you focus on those problems, you realize there's no need for an accurate estimate. Al…

Get something ready for Christmas business? Get a car on the road? Finish the cruise ship in time? ...

Re: Software effort estimation is mostly fake research

#117

Post by randomsearch » Mon, Jan 18, 2021, 9:02 PM UTC

ThomPete wrote: post #60

In his book “Code Complete”, Steven McConnel speaks about metaphors. He reasons that metaphors are necessary to be a good developer as it helps visualize the act of coding. The metaphor he prefers is the “construction” metaphor. This metaphor he argues best explain the act (some would say art) of programming and gives developers a language to speak in that brings clarity to the development process. When in constructi…

The reality is that software development is nothing like engineering or construction, it's totally different. You don't build a quick house, let people live in it and start building the walls whilst they live there.

Humans like to think via metaphor because it's a least-effort mode of thought but sometimes there just isn't one and it's just tough luck and start thinking from first principles instead.

Re: Software effort estimation is mostly fake research

#118

Post by robertlagrant » Mon, Jan 18, 2021, 9:03 PM UTC

BowBun wrote: post #100

It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates. According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather. If industry…

What data would be useful here? There are so many confounding factors.

Re: Software effort estimation is mostly fake research

#119

Post by didibus » Mon, Jan 18, 2021, 9:14 PM UTC

jeffreygoesto wrote: post #116

didibus wrote: post #107

Earlier quoted context omitted.

> And I’ve also seen wise use of estimation but that is rarer I've got a question for you here, what is a wise use of estimation? What problems are estimates the ideal solution for? In the context of software development? I can think of only two: planning headcount and priorities (aka planning the order in which you do things). When you focus on those problems, you realize there's no need for an accurate estimate. Al…

Get something ready for Christmas business? Get a car on the road? Finish the cruise ship in time? ...

> In the context of software development?

While your examples I'm not sure are super relevant from a software development context. I think you're pointing out that the estimate would help you decide when you need to start so you are done before the "real life events that needs the thing".

Is that correct?

Like for example I'm not sure it's useful to learn that no, you will not be ready for the Christmas event. Or no the cruise ship will not be ready in time. But it might be useful to know: "If I want to be ready for the Christmas event when should I get started and how many developers do I need?"

That's a good example I guess, though I haven't seen this issue arise very often in software. I could imagine someone needing to know when to start feature freeze and prep work for scaling up leading up to a big sale event if you run an online store for example.

My take on this is that asking devs to guess that time would obviously not work. And if you did this type of forecasting for any other discipline, you'd instead gather historical data about how much time things took in the past and plot that to forecast how much you'd think it takes in the future. Which is a process that can be done without needing to ask a dev for their personal not data based guesstimate.

Re: Software effort estimation is mostly fake research

#120

Post by alkonaut » Mon, Jan 18, 2021, 9:14 PM UTC

wyldfire wrote: post #8

How do other disciplines estimate NRE? Do they have the same problems with missed predictions?

You only quote the fixed bits and charge by the hour for the rest. If you need to give a quote for an unknown you multiply the quote by N to cover a worst case.