My personal opinion is that open sourcing googlebot[0] and tensorflow, as well as a the numerous other tools like elastic search and frameworks like hadoop make it pretty easy to make narrow search tools.
So someone like me, a pretty shitty programmer, can use an language built on top of googlebot, like nodejs, and just index the stuff I care about and instead of that information being returned to google, I can just send it back to bots viewport directly and not even see the webpage, just the info if I want.
The problem is discovery (what google solved in 1999), no one knew where to find websites, like not even info, just what sites even were on the web. This is a huge pain point again because while there are a lot of great websites there are a ton of shitty ones. This is pretty obvious if you look at sites like reddit or hacker news. They are developed around the idea of aggregating good resources because there are so many it is overwhelming.
So, I think, that of the 50% of the people that are already running googlebot, some subset will start probably using it to get info they care about. Eventually, someone will write code you can use to make this easier for normal people, and then google whill either sell the silos of info to them instead of advertising, or use it in it's new meta structure to solve big problems.
[0]http://ipullrank.com/googlebot-is-chrome/
[1]http://www.sitepoint.com/browser-trends-august-2015-chrome-e...