In my opinion, yes, Google will keep crawling the web and indexing in a similar (but continually optimized) way for the foreseeable future. Here's why - let's say you work at Google and come up with an entirely new way to do search. Here are your options:
1 - Go to Larry/Sergi with the proposal to completely rip and replace millions/billions of dollars of infrastructure and knowledge in favor of a completely unproven new idea. The cost and risk are so huge they'd never go for it.
2 - Go to an outside investor with the proposal to build it from scratch. If it's truly an amazing, innovative new approach, they might fund it and you'd be able to build it from scratch without the political nightmare of trying to rewrite the core product of a multi-billion dollar public company. This is exactly what both the Cuil and FriendFeed guys did (and yes, FriendFeed is very, very, very much a Google competitor).
One thing I always think about is in 100 years (or 1000 years), will we still be opening a web browser, typing a few words into the same plain Google text box, and hitting the "Search" button to get a page with 20 blue links and a bunch of ads? I doubt it. So there's definitely a better way to do search, we just haven't discovered it. My personal opinion is that some combination of social search (i.e. FriendFeed) plus human-powered search (i.e. Wikia/Mahalo) plus semantic search (PowerSet) will be involved in the next evolution. Of course, if I knew exactly what that looked like, I'd be on a beach right now instead of hanging out here on Hacker News. :)