Software effort estimation is mostly fake research • Hacker News Board

Re: Software effort estimation is mostly fake research

#201

Post by Const-me » Tue, Jan 19, 2021, 12:54 AM UTC

csours wrote: post #10

"Estimates" are for things you've done before - like you can estimate building a house, because people have built houses before. The more like an existing house, the better you can estimate it. Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word th…

I mostly agree, but for research, estimating the estimation is often good enough in practice.

For me it often something like this:

I don't have a clue how the hell to do what you asking for. But maybe implementing .. might help. Can't guarantee but it might. I think I can confirm or disprove that spending .. on the prototype subject to following limitations.. If then we find out it actually works for you, we'll go from there but approximately gonna take .. extra to rework the prototype into production-quality stuff. If it won't, I'll think about something else to try.

Re: Software effort estimation is mostly fake research

#202

Post by perl4ever » Tue, Jan 19, 2021, 1:00 AM UTC

taeric wrote: post #89

My problem with this is all estimation is hard. Period. Quantitative discussions of something that has been done? Fairly easy, if still not accurate. Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done. Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to p…

Estimation is hard if you are stuck in the mindset that you have to make a point estimate.

It's not hard at all if you are willing to be (and are allowed to be) honest about the uncertainty of the inputs and calculate the uncertainty of the final result based on that.

It's true that people may demand precision that you can't give them. But at the same time, you know something and it is simple to compute what you know.

It's like Fermi estimation, that everyone hates so much and claims is so useless to interview for.

Re: Software effort estimation is mostly fake research

#203

Post by perl4ever » Tue, Jan 19, 2021, 1:09 AM UTC

a_wild_dandan wrote: post #58

csours wrote: post #10

"Estimates" are for things you've done before - like you can estimate building a house, because people have built houses before. The more like an existing house, the better you can estimate it. Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word th…

You've perfectly nailed the fundamental issue here. For many software projects, you simply cannot make meaningful granular estimations for parts of them. It doesn't matter how many story-point poker sessions you hold. Some software work cannot be reduced to a positive integer. In our business scopes creep, bugs plague us, and myriad issues make precise estimations pointless. It's engineering Numberwang. I get that fo…

>I get that folks want to tame a chaotic world. But sometimes you can't.

No matter how little you know about something you want to estimate, there is an infinite difference between what you do know and knowing nothing. The right thing to do is to represent your knowledge without fabricating it.

Saying "I can't make a point estimate, therefore I can say nothing" is a false dichotomy.

Even if you say, x is between 0.001 and 10^10, you are narrowing things down.

There is a great paper mentioned periodically on HN, where they examined the famous Drake equation and show that when you get rid of point estimates and use reasonable ranges for each variable, the "paradox" that we don't see aliens vanishes. Despite many of the variables having uncertainty of many orders of magnitude.

Re: Software effort estimation is mostly fake research

#204

Post by x0x0 » Tue, Jan 19, 2021, 1:14 AM UTC

didibus wrote: post #119

jeffreygoesto wrote: post #116

Earlier quoted context omitted.

Get something ready for Christmas business? Get a car on the road? Finish the cruise ship in time? ...

> In the context of software development? While your examples I'm not sure are super relevant from a software development context. I think you're pointing out that the estimate would help you decide when you need to start so you are done before the "real life events that needs the thing". Is that correct? Like for example I'm not sure it's useful to learn that no, you will not be ready for the Christmas event. Or no…

Inter-team dependencies. Marketing pushes. Customer commitments. New sales dependent on certain features. Building features to increase TAM. Partner commitments for interoperability / joint deployments. Even presentations of your roadmap to customers and prospects to get them bought into your vision of where the product is going.

There's tons of things that require broadly knowing when X is going to be done beyond capacity issues.

Re: Software effort estimation is mostly fake research

#205

Post by BowBun » Tue, Jan 19, 2021, 1:16 AM UTC

froh wrote: post #147

BowBun wrote: post #100

It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates. According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather. If industry…

> How large are these datasets that have attracted so many research papers? > The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset con…

I think the term 'cargo cult' carries some religious history and other baggage that I wouldn't want conflated here, but your feeling that anything we've heard about software estimation is questionable is probably warranted.

IMO the theories have sprouted more from management which unfortunately is probably the best equipped today to view the trends over time and compare them to actual results (given the lack of empirical data). If we admit that our ideas of estimation have come from management, we should also admit their conflicts of interest in the matter and the various management fads that come and go. I think as humans we all search for patterns and try to generalize rules to solve our problems, even when we have an incomplete view of things to begin with.

Finally, even with a generalized rule we should realize how different people are team-to-team. I haven't worked on two teams where the same exact processes worked for everyone. The best teams and managers I've had will observe a team over time and tweak the various processes according to what works at that point in time. The worst managers were the ones who idolized a specific style or person and copy-pasted their opinions into their workplace without listening to the team.

Re: Software effort estimation is mostly fake research

#206

Post by smichel17 » Tue, Jan 19, 2021, 1:16 AM UTC

meheleventyone wrote: post #140

dkarl wrote: post #127

Earlier quoted context omitted.

> Even houses always go over time and money and that is fairly straight forward. This is an incredibly important point. I was "raised" in commercial software at a time when there was a lot of condescension internally towards ourselves as an industry because we couldn't do estimation and planning, and building was the favorite comparison. A $100 million building couldn't run late or over budget. That would be unimagin…

It doesn’t even have to be a big construction project to hit budget and schedule issues. Rebuilding our house had us going right back to the drawing board having to get new plans drafted and approved. It worked out much better in the end but the unforeseen cost and time overrun was quite anxiety inducing at the time. But yeah the idea anyone has magically solved accurately predicting the future is completely bonkers.

I'll raise (shrink) you an estimated-at-1-week landscaping project for my house that ended up taking just over 2. Wasn't particularly anxiety inducing, though, as we expected some overrun and both parties (re)negotiated in good faith.

Re: Software effort estimation is mostly fake research

#207

Post by jrumbut » Tue, Jan 19, 2021, 1:19 AM UTC

daanlo wrote: post #134

didibus wrote: post #82

The issue with estimates are expectations. While nobody acknowledges it, you're not actually asked for an estimate, you're being asked for a quote. The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and…

Software doesn‘t exist in a vacuum. The marketing team needs to know when the landing page / qr code (whatever) is implemented, because they need to plan their marketing budget. The customer care agent that gets shouted at because feature xyz is broken / not working as the customer expects, would like to give the customer an answer when it will be fixed. The customer care manager is also worried about her team member…

That's the thing, if other departments have a timeline they need that should be the estimate.

I think it's a lot more natural in business software to say something like "we need the best version of XYZ you can do in three months" rather than saying "we need XYZ" and hoping the estimate is three months. That way everyone's cards are on the table and can work together on getting what they need.

Re: Software effort estimation is mostly fake research

#208

Post by dragonsky67 » Tue, Jan 19, 2021, 1:20 AM UTC

koonsolo wrote: post #136

didibus wrote: post #82

The issue with estimates are expectations. While nobody acknowledges it, you're not actually asked for an estimate, you're being asked for a quote. The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and…

I once had a manager that thought he could negotiate estimates. That is like negotiating with the weatherman about the weather. Sure, maybe in the end you can convince them it will be finished sooner. But the fact is that it really doesn't change reality.

at that point you are negotiating just how much of your personal time/stress you are going to burn to meet their expectations. Also maybe how many other people you are going to have to short change because they have jumped the queue.

Re: Software effort estimation is mostly fake research

#209

Post by ChrisMarshallNY » Tue, Jan 19, 2021, 1:28 AM UTC

perl4ever wrote: post #198

ChrisMarshallNY wrote: post #71

Earlier quoted context omitted.

You are exactly correct. That is why Honda is a bigger company than Mercedes. They do decent quality (Hondas cost more than Kias), but at scale. I worked for a "Mercedes-level" photographic equipment company for years. People were often quite surprised, when I told them the size of the company. It was like Roadhouse "I thought you'd be bigger." Our prototypes were incredibly expensive. A $2,000 (retail) body prototyp…

>That is why Honda is a bigger company than Mercedes I was curious if this was accurate. Here are some recent numbers for revenue and unit sales: Honda -> $144B revenue - 12.4M motorcycles, ATVs, etc. - 3.3M automobiles - 5.7M generators, small engines, etc. Daimler -> $207B revenue - 2.4M automobiles - 0.5M trucks - 0.4M vans - 0.03M buses So, Daimler seems to be about 40% larger in revenue, but Honda sells almost 4…

Good point. I was thinking cars, but Europe seems to have every cab as a Mercedes.

That's a lot of cars.

Re: Software effort estimation is mostly fake research

#210

Post by wheelie_boy » Tue, Jan 19, 2021, 1:31 AM UTC

dkarl wrote: post #127

snarf21 wrote: post #98

Earlier quoted context omitted.

The issue is that you are being asked to estimate something that has never been done before. Even houses always go over time and money and that is fairly straight forward. These days, I only give estimates in terms of units but without numbers. Hours, days, weeks, months, quarters or years. Some relatively small number of those units. If you want a quote it will take an extra 1/4 of the estimate worth of time for an…

> Even houses always go over time and money and that is fairly straight forward. This is an incredibly important point. I was "raised" in commercial software at a time when there was a lot of condescension internally towards ourselves as an industry because we couldn't do estimation and planning, and building was the favorite comparison. A $100 million building couldn't run late or over budget. That would be unimagin…

I've seen a lot of industries compared to programming, like construction and manufacturing. For example, people trying to apply lessons from the Toyota Production System to programming.

This doesn't make sense, because we have systematized the construction of executable programs with extremely high reliability and predictability: we call it "Copy File"

It's the design that is the hard part, and the process of coding is the process of iterating on the design. Even turning the design into an executable is extremely reliable, either running via a VM or scripting engine, or compiling. Those are all very repeatable and reliable.

There are other disciplines that focus on creating and improving designs (like writing, painting, architecture, 3D modeling), but it's not like they're reliable or predictable either, unless you're designing very similar things over and over again. That's true of coding as well, but you don't really end up doing that, because once you've done it 3 times you should make a library and then never do that part again, just the unique bits.

In short, yeah, coding is its own thing, and I doubt it'll ever be predictable unless you're wasting a lot of time and effort doing very similar things over and over.