There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.
Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…
> I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then query efficiently.
That's what Apify in original post does - including a public database of scrapers, so there is a high chance you could use already finished scraper :).