Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The danger of the mediocre success when testing startup hypotheses (2023)

pivotal.substack.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#71

Post by vharuck »

brabel wrote: post #62
jjk166 wrote: post #61

The problem here comes from poor experiment design. Seeing how many sales you get from 100 cold calls has no control, and produces a single datapoint. No matter what number you get, be it 0, 2, or 37, will be useless for predicting future sales. First, you need a sample size that will produce a statistically significant result. Cold calls are expected to have a low success rate and be highly variable. 2 in 100 could…

> If instead you had a sample size of 2000, then you'd get a very good signal to noise ratio. In modern statistics, even a small number of samples can be considered enough to get a satisfactorily small error range, as long as the sample is random and representative of the population. I would think 2,000 samples is far more than strictly required if you're able to sample from your target market.

It's true that people often incorrectly dismiss results with sample sizes below 100. But, for rare events, you really do need large samples. Otherwise, you'll only ever be able to confidently identify massive differences.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#72

Post by bcrosby95 »

sumanthvepa wrote: post #38

As an entrepreneur, my observation is that the vast majority of the experiments I run result in an outcome where the null hypothesis cannot be rejected. That is the case for most social science experiments (which is what marketing experiments really are.) But since I don't have to publish anything, I don't p-hack my results. If you think that scientific rigour will help you avoid the need for good judgement, you're i…

And if you don't reject the null hypothesis in the vast majority of experiments, then a large portion of those where you do were probably also just a statistical anomaly.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#73

Post by henry2023 »

mkl95 wrote: post #10

> So when a startup comes to me with an idea for an experiment, the one thing I tell them is: make sure that there’s a well-defined distinction between success and failure. Don't fall in the messy middle. Mediocrity is often enough to put food on the table. The world is full of companies aiming to survive for a few more quarters with their little mediocre product. And once a company embraces that idea there's usually…

I know a guy who settled with a mediocre product that put food on the table. He kept it running for the most part of the 2010s. Then, out of nothing someone came and acquired his company for ~ $14mm. He had no investors and just a handful of workers.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#74

Post by jjk166 »

brabel wrote: post #62
jjk166 wrote: post #61

The problem here comes from poor experiment design. Seeing how many sales you get from 100 cold calls has no control, and produces a single datapoint. No matter what number you get, be it 0, 2, or 37, will be useless for predicting future sales. First, you need a sample size that will produce a statistically significant result. Cold calls are expected to have a low success rate and be highly variable. 2 in 100 could…

> If instead you had a sample size of 2000, then you'd get a very good signal to noise ratio. In modern statistics, even a small number of samples can be considered enough to get a satisfactorily small error range, as long as the sample is random and representative of the population. I would think 2,000 samples is far more than strictly required if you're able to sample from your target market.

Average results for cold calling are a 4.8% success rate, meaning with 2000 cold calls you'd expect less than 100 hits for a good campaign. This in turn is highly variable with field, a 1% success rate might be high, especially for a product with no pre-existing market presence. And it's not enough to see whether you're above a certain number, you need to know the variance to predict future performance. Maybe the magic number is 1500 or something, the number will no doubt vary based on the peculiarities of the experiment, including the product and the company, but to see a signal with 2 standard deviations of significance, you need a sample size that you would expect to produce about 20 times the expected noise level.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#75

Post by jjk166 »

jantissler wrote: post #58
jjk166 wrote: post #54

Earlier quoted context omitted.

An article with a deceptive title and lacking substance is textbook clickbait.

The title is "The Worst Outcome is a Mediocre Success" and that's what this is about. How is this deceptive? The "substance" part might be debatable. I personally don't think every post has to go in-depth on everything. I enjoyed his nugget of insight.

It's deceptive because the title is being used in a way that you would not guess without reading the article and suggests highly that it is referring to something completely different. "Mediocre success" is not a synonym for "ambiguous result" and in context most would assume it is referring to financial success.

The updated title is much better.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#76

Post by ben_jones »

new2this wrote: post #14
pinkmuffinere wrote: post #6

> make sure that there’s a well-defined distinction between success and failure. Don't fall in the messy middle. I agree this is the right thing to do, but it isn’t always possible to do for a given experiment. For instance, with the cold-calling example, how do you ensure that it’s either a smashing success or definite failure a-priori? Is there a principled way to choose the threshold number of conversions? Do you…

I've built a few sales team for early stage startups, and invariably the question comes up: "Inbound sales is working. Will outbound sales work?" The successful experiments I've seen look like this: hire 2 SDRs at the time, focus them exclusively on outbound, then measure their results to identify the CAC. Then, you can make a call if that CAC is something you want to scale. The failed experiments look like this: the…

CAC as in Customer Acquisition Cost if anyone else was wondering

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#77

Post by sgarman »

kijin wrote: post #55
mkl95 wrote: post #10

> So when a startup comes to me with an idea for an experiment, the one thing I tell them is: make sure that there’s a well-defined distinction between success and failure. Don't fall in the messy middle. Mediocrity is often enough to put food on the table. The world is full of companies aiming to survive for a few more quarters with their little mediocre product. And once a company embraces that idea there's usually…

Settling down with a mediocre product that sustains your company should probably count as success. The vast majority of startups fail, after all. You might have failed to produce a unicorn, but at least you've got a decent workhorse.

VC's have twisted the idea of what a success is for normal entrepreneurs because it doesn't fit VC's 100 failures, 1 unicorn model.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#78

Post by gopher_space »

darkhorse13 wrote: post #67

My literal life goal is to have a software startup with moderate success WITHOUT VC funding. One that can sustain my lifestyle, but also keep me busy and happy without going completely overboard.

Go through a list of dead YC startups and steal ideas that wouldn't scale. A five-person company putting a quarter of a million into everyone's account per year, in perpetuity, is an abject VC failure.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#79

Post by doctorpangloss »

sumanthvepa wrote: post #38

As an entrepreneur, my observation is that the vast majority of the experiments I run result in an outcome where the null hypothesis cannot be rejected. That is the case for most social science experiments (which is what marketing experiments really are.) But since I don't have to publish anything, I don't p-hack my results. If you think that scientific rigour will help you avoid the need for good judgement, you're i…

> that scientific rigour will help you avoid the need for good judgement

This is what all big company PMs and directors of engineering believe. But their judgement (good or bad), their metrics (scientific or pseudoscientific), their career trajectory (up or out), whether these are known or unknown in the first place, and their actual material success: who knows how correlated it all is. They don’t have the words to express this stuff, so they wouldn’t be able to see the difference between science and pseudoscience for example, and you'd have a hard time communicating this thing about science to these people.

So while you are right, it’s not saying much that the real problem is communications, and that is the mainstream opinion of people who study microeconomics / the structure of firms. Put differently, people have built cathedrals of bullshit in their minds in service of the status quo where they “test” “everything” in lieu of having falsifiable, forward-looking opinions (aka judgement). You can’t just go, Martin Luther all of corporate white collar bureaucracy.

Re: The danger of the mediocre success when testing startup hypotheses (2023)

#80

Post by pseudalopex »

chefandy wrote: post #63
sumanthvepa wrote: post #38

As an entrepreneur, my observation is that the vast majority of the experiments I run result in an outcome where the null hypothesis cannot be rejected. That is the case for most social science experiments (which is what marketing experiments really are.) But since I don't have to publish anything, I don't p-hack my results. If you think that scientific rigour will help you avoid the need for good judgement, you're i…

> If you think that scientific rigour will help you avoid the need for good judgement, you're in for a great deal of distress. Broadly, assuming more data and tighter calculations == more certainty is a folly endemic to engineering-focused organizations and industries. In reality, you're increasing precision without increasing accuracy: a dangerously misleading state. It's understandable: organizations obviously need…

The McNamara fallacy infects business types at least as much as engineering types in my experience.
Post reply on HN