PG,
In part you are correct, but you are making some huge mistakes and, in a saying from my past, are "straining over gnats and forgetting elephants".
> That's made harder by the fact that the best startup ideas seem at first like bad ideas. I've written about this before: if a good idea were obviously good, someone else would already have done it.
That's likely true from what you and Silicon Valley see, but that's just a gnat.
Basically you are saying that success is a shot in the dark and it is impossible for an entrepreneur and their financiers to design success with good reliability. That is, from enormous data, obviously nonsense.
Paul, instead of shooting in the dark, you need to turn on a light!
I claim that we can design success, on just clean sheets of paper, with high reliability and that how to do so is well known with a fantastic track record with the long track record right in front of us.
Also, we need to address your:
> if a good idea were obviously good, someone else would already have done it.
Nonsense. Paul, where do you get such stuff? You have one of the best backgrounds of anyone in Silicon Valley entrepreneurship, and yet you still fall for that nonsense that fills the trash and the offices on Sand Hill Road.
What you mean by "a good idea" is just a good 'business idea', e.g., a very short, nearly superficial, description of the business to, say, an early customer. But, Paul, that's nearly irrelevant.
Want a good business idea? Make a billion dollars quickly? A sure fire winner? Okay, this is your lucky day. May I have the envelope, please (drum roll). Yes, here it is: One pill to take once that provides a safe and effective cure for any cancer. That's your guaranteed, 100% true, died in the wool, sure fire, billion dollar "good business idea". And it's obvious, and no one has done it. Done.
Since the 'idea' is obvious, why has no one done it? Sure: No one knows HOW to do it.
Are you beginning to understand?
So, for a hand up, in case you were up all night drinking beer with the boys, here is a 'generic method' (ALL TRUE hackers just LOVE such verbiage):
Step 1. Think of a big, unmet need, something a billion people will pay a little for or thousands of people will pay a lot for. E.g., the one pill cure for any cancer. That is, think of an important, unsolved problem.
Step 2. Find a way to meet this need. E.g., for the cancer pill, have to do some quite good biomedical research. Uh, did I say that it was all easy? I don't remember saying it was all easy. If Step 2 is too difficult, then return to Step 1 (we're writing this in the form of an 'algorithm' since ALL TRUE hackers just LOVE algorithms). Else proceed to Step 3.
Step 3. Sell the solution to the customers and take the money to the bank.
There it is, in just three steps. Of course, likely the bigger the unmet need from Step 1, the more difficult will be the research in Step 2. Whatever, the key to the 'algorithm' is the research in Step 2.
Did I mention 'research'? Gee, does anyone on Sand Hill Road consider research? Well, Paul, you and Y Combinator have high qualifications in evaluating research, but a cursory look at several of the best known venture capital firms in Silicon Valley, Boston, Winter Street, and NYC show little to no ability or willingness to evaluate research. Uh, to evaluate research, as usual, we want at least a relevant Ph.D., a tenure track faculty position in a research university, and some relevant peer-reviewed publications of original research.
So, research is being ignored! Opportunity knocks!
Now, should we have any faith in the promise of research to get solutions to practical problems, or is all of research just ivory tower intellectual self-abuse that has yet to be proven totally useless forever?
Let's see: It turns out that there is in all of the world in all of history exactly one, unchallenged, unique grand champion of doing research to get powerful solutions to important practical problems. And the track record is much better and much longer than that of Silicon Valley.
Without further ceremony, here's the envelope. Yes, the answer is, the US DoD. They got going about 70 years ago and did little projects like "the bomb, the hydrogen bomb", radar, synthetic aperture radar, spread spectrum radar, encoded with shift register sequences, adaptive beam forming passive sonar, inertial navigation, GPS, and on and on.
So, for the question, is it possible to do research that yields powerful solutions to important practical problems? Sure. Done.
So, should Silicon Valley fund research? Well, perhaps not. But, if an entrepreneur has selected a good "unmet need" in Step 1 and done some good research to get a powerful solution in Step 2, should Silicon Valley consider the research?
Hmm ...? But, such research is rare! Right! Did I notice that you have already noticed that big winners are rare, 'black swans', outliers, "few"? Yup.
How many $200 billion winners does a $100 million venture fund need to give good returns to its limited partners? Many? Several? A few? How about just one?
Silicon Valley is shooting in the dark into a pond with nearly only small fish. Silicon Valley needs to turn on some lights, look, pay attention to powerful research for big unmet needs, and then evaluate pulling the trigger.
My guess is that the real problem is the limited partners (LPs) who really prefer to look at reports from accountants. So, the LPs tell their venture funds to do all evaluations as close to accounting as possible. Since the usual accounting metrics are not yet available, the venture firms use surrogates, and their favorite is 'traction'. They want their coveted 'traction' to be high and growing rapidly. E.g., in the case of the bomb, they would have said "You build and test one and get one ready for delivery, and we will chip in for the gas for the Enola Gay.".