The danger of the mediocre success when testing startup hypotheses (2023)

First, you need a sample size that will produce a statistically significant result. Cold calls are expected to have a low success rate and be highly variable. 2 in 100 could very well be a massive success. 0 in 100 wouldn't rule out its viability. If instead you had a sample size of 2000, then you'd get a very good signal to noise ratio.

Then, you need controls. Look up statistical design of experiments (DOE), you'll find efficient ways for finding how much various different factors affect your results. Basically split up those 2000 calls into groups and vary things as you go, so instead of testing whether a specific cold call technique is working, you can see if cold calls in general work.

Finally, understand what success means. A business does not need to rigorously prove that it's methods are optimal, it needs only to find an adequate strategy to achieve its objectives. You should know what it costs to make 2000 cold calls, you should know how much revenue is generated from a sale, there should be a specific threshold where the number of sales justifies the cold calls. You should be doing a process capability study that you are sufficiently over this threshold that, given expected variation, you will remain above it most of the time. What is your Cpk? How much does it vary among the different sub-experiments you performed?

At the end of the day, you're still going to need to make decisions with incomplete information, but don't pretend that you're making a data driven decision when you're not. The worst thing is not an inconclusive experiment, the worst thing is an experiment with an erroneous result you mistake for conclusive.

Broadly, assuming more data and tighter calculations == more certainty is a folly endemic to engineering-focused organizations and industries. In reality, you're increasing precision without increasing accuracy: a dangerously misleading state. It's understandable: organizations obviously need to play to their strengths, but there's real danger in succumbing to the "when you're a hammer, everything looks like a nail" mindset. Firstly, It's much easier for non-engineers to parse the difficulty of engineering problems than the other way around. Outsiders easily see that they don't have the requisite chops to do engineering work. Secondly, cutting through ambiguity and unpredictability to reveal factors you can control is critical to engineering, while non-engineering jobs like management, design, and community outreach are hard because you must confront those things. Engineering types often reason about non-engineering problems either like they have the same level of predictability, or simply disregard the ambiguous and unpredictable factors because they don't fit into an equation.

It's easy to see why that micromanaging non-technical manager screwed up by insisted on using some technology or approach they read a snappy article about; an engineer's authority is cut-and-dried in that situation. It's harder to see why purely formulaic approaches often fail dealing with people, nature, markets, etc. Most things in the world are far more complex, temperamental, and less predictable than cache invalidation.

Are you referring to the slightly edited title someboty has put here at HN? But still, no dissonance with the content

Take a company like Pdfsimpli for instance.