In fields like art, programming, and physics, you'd
better be doing something
weird. If you're doing something
mainstream, the same thing thirty other people around the world are doing, you're all competing to make the same thing — paint the same painting, write the same text editor, prove the same theorem about black holes. Twenty-nine of you are going to get scooped by whoever is the hardest-working, the smartest, the best-supported institutionally, or whatever combination of those turns out to be the deciding factor. Your chance of being in that 97% who have totally wasted their efforts? 97%.
And if you're doing something really mainstream, like writing the next big client-side JavaScript framework, the one that will replace React, watch out! Your chances are a lot worse than that, because there are hundreds of thousands of people who fight with React every day and are frustrated with its shortcomings. Your chances are literally millions to one, unless you work at Microsoft or Google and have management support to beat those Facebook fuckers.
But if you're doing something offbeat, working on a problem† that's mainstream enough to be interesting if you're successful but not mainstream enough that dozens of people are already spending their weekends trying to solve it, you have a much better chance of finding a niche for your project. Maybe it's non-mainstream because people take for granted that it can't be solved (in which case they might be right, as with the reactionless drive — the objective here is to be weird in your project goal, not your epistemology); maybe because they don't understand why it would be important to solve it ("Where's the market?"), and you do; maybe, as with OpenSSL, it's an important problem, but there's no way to get paid for solving it.
A thousand hackers writing a thousand versions of the same library in the same way are only epsilon more productive than one hacker. A thousand hackers writing a thousand different libraries are almost a thousand times as productive.
Winning the lottery? Well, that's pretty much out of your control — but if you do decide to waste your money on the lottery, don't pick a number lots of other people are picking. Then you'll have to split the already-improbable winnings N ways.
But what if you're determined to solve a mainstream problem anyway, one where a thousand people are also trying to solve it? Then you need all the outcome variance you can get! If, of those thousand hackers, 500 are using a very conservative approach that is guaranteed to solve the problem with some quality metric 10 ±1 in 26 weeks ±2 weeks (these being the standard deviations, not some kind of 95% confidence interval), while the other 500 are using all kinds of wild approaches that give them quality metric exp(ln(4)±ln(4)) in time exp(ln(52)±ln(4)) weeks, it's pretty much guaranteed that the "winner" is going to be someone with a totally insane approach that hacked together a library with quality 49 in only 14 weeks. It's not going to be one of the 26-week plodders, because 14 weeks is 12 standard deviations out on their distribution (vs. 0.95 out on the crazy hackers' distribution), and quality 49 is 39 standard deviations out (vs. 1.3 out on the crazy hackers' distribution).
In fact, it's even worthwhile to sacrifice expectation to get higher variance in these situations. If your only hope of winning is to beat everyone else in a single round, you should do whatever will increase your minuscule chance of a home run, regardless of how it affects your chances of striking out.
It's still not socially optimal for people to behave this way — note that here you have 1000 hackers whose aggregate productivity is only about 20× the productivity of an average plodder — but if you've gotten suckered into competing for a mainstream niche, that's the way to play the game.
All of the above is for the simplified situation where you're working on a project by yourself. In a teamwork situation, the relevant actor is your team, not you individually. Do not write your code in Clojure if the rest of the team is working in Ruby. Do not try to solve only problems that nobody else on the team thinks are important.
And this advice definitely does not apply to a situation where doing the same thing someone else already did is valuable. If you're making a sandwich, there's no reason it needs to be different from the sandwich someone else is making across the street. They're two different sandwiches. If someone eats the sandwich across the street, it's gone and it can't feed your customer. They're going to be happy if you make them a sandwich they like, even if it's a little worse than the sandwich across the stret; even if there are many just like it, this sandwich is theirs. This is very different from the situation in software, where one sandwich feeds everybody in the world at once, except people with celiac disease. Nobody is going to be happy that you wrote a web browser from scratch for them instead of just installing Firefox.
† I recognize that a painting is not "solving" a "problem", but many of the same principles apply.