It's actually more interesting than that: several of the key people
working on the project believed it was impossible and could never succeed.
The short answer to your question is "because Larry and Sergey really wanted it". It's amazing how much attention projects that the founders are personally interested in get.
That would be very misleading, though, because it's not like anyone was forced to work on Instant, nor is the typical Googler a mindless automaton who always does what he's told. Everyone who worked on Instant was there because they wanted to work on Instant, even the ones who felt it didn't have a prayer of success.
The longer answer is that Google's created a culture that tolerates ambiguity remarkably well. Y'know how they say that it's a sign of intelligence (and insanity) to be able to hold multiple conflicting ideas in your head at once? That's the same drive that lets people say "I think this is a terrible idea, and it'll never succeed...but it's interesting, and it deserves an honest chance, so let me help you with that." Strong opinions lightly held.
This starts with the hiring process, I think, which tends to select for people who can appreciate nuance and tend to explore lots of technologies outside those that are immediately useful to them (contrary to popular opinion, Google does not solely care about GPA and alma mater when hiring). Then new recruits get to the Googleplex, and they're immersed in a culture that's just awash in ideas, most of which are terrible, and all of which are listened to. When you see that your coworker's reaction to the brain-dead idea that was just spouted forth is "Hmm. That's interesting - what could you do to test that and develop it?" instead of "That's stupid and it'll never work", you start to reevaluate your snap judgments. And of course, if you work there for any length of time, you'll see a number of those stupid ideas actually launch and become giant successes with millions of users.
I remember clearly telling my coworker, about Instant, "I think it's a terrible idea and will fail miserably, but I am really, really glad that I work in a company where we can place these sorts of bets."
I wish more places had a culture like this. FaceBook, I've heard, has it. So do many prominent investors in Silicon Valley, and I wonder if that's where Google got it. Top-flight research labs and universities do too. But it's sorely lacking in the general public, and on Reddit, and among our politicians & government, and in many big corporations. It, unfortunately, is not the sort of thing you can pick up from a book or a few blog or forum posts. You really have to be immersed in the culture to absorb it, and even then, your natural inclination is going to say "This is stupid" to most new ideas. It seems to take a lot of mental effort to suspend that judgment and say "Well, this sure looks stupid, but I'll stick with it a little longer and see where it ends up."
I've found, though, that that culture is perhaps the most valuable thing that I've gotten out of my time at Google.