71–80 of 113 posts
google will lose, and I'm surprised they are even trying. hiQ v. LinkedIn already settled this: scraping public web pages isn’t “unauthorized access,” even if the site says no via robots.txt or ToS. Those aren’t locks.
Google really doesn't have a leg to stand on here. They scrape the Internet. They replace content against the wishes of users multiple different times, such as with AMP. Their entire business model recently has been to provide you answers they learned from scraping your website and now they want to sue other people who are doing the same. Data wants to be free. They knew that once. EDIT: Also to be clear I am not say…
You can search Google _for free_ (with all the caveats of that statement), part of their grievance is that serpapi use the scraped data as a paid for service
Lots of Google bot blocking is also circumvented, which they seem to have made a lot of efforts towards in the past year
- robots.txt directives (fwiw)
- You need JS
- If you have no cookie you'll be given a set of JS fingerprints, apparently one set for mobile and one for desktop. You may have to tweak what fingerprints you give back in order to get results custom to user agent etc.
Google was never that bothered about scraping if it was done at a reasonable volume. With pools of millions of IPs and a handle on how to get around their blocking they're at the mercy of how polite the scraping is. They're maybe also worried about people reselling data en masse to competitors i.e. their usual all your data belongs to us and only us.
"Google follows industry-standard crawling protocols, and honors websites’ directives over crawling of their content." Is that true with how they trained Gemini? Doesn't everyone with a foundational model scrape the web relentlessly without regard for robots.txt?
[flagged]
Following that same logic, may I inform you that your income going forward is different: it has to be directed to my bank account, because the account needs the money! :-)
google will lose, and I'm surprised they are even trying. hiQ v. LinkedIn already settled this: scraping public web pages isn’t “unauthorized access,” even if the site says no via robots.txt or ToS. Those aren’t locks.
"Google follows industry-standard crawling protocols, and honors websites’ directives over crawling of their content." Is that true with how they trained Gemini? Doesn't everyone with a foundational model scrape the web relentlessly without regard for robots.txt?
[flagged]
Testimony https://medium.com/@brianwarner/celebritynetworths-statement...
CNW ended up putting up content for fake celebrity's after declining Google's request for API usage to prove that Google was scraping them.
Google really doesn't have a leg to stand on here. They scrape the Internet. They replace content against the wishes of users multiple different times, such as with AMP. Their entire business model recently has been to provide you answers they learned from scraping your website and now they want to sue other people who are doing the same. Data wants to be free. They knew that once. EDIT: Also to be clear I am not say…
SERP API just assumes everybody wants to be scraped, and doesn't give you a choice.
(whether websites should have such a choice is a different matter entirely).
"Google follows industry-standard crawling protocols, and honors websites’ directives over crawling of their content." Is that true with how they trained Gemini? Doesn't everyone with a foundational model scrape the web relentlessly without regard for robots.txt?
Almost everybody wants to appear in search, so disallowing the entirety of Google is far more costly than E.G. disallowing Openai, who even differentiates between content scraped for training and content accessed to respond to a user request.