Live data from Hacker News

Why we're taking legal action against SerpApi's unlawful scraping

blog.google

91–100 of 113 posts

Re: Why we're taking legal action against SerpApi's unlawful scraping

#91
I had an idea - take SerpAPI and save top-10 or 20 links for many queries (millions), and put that in a RAG database. Then it can power a local LLM do web search without ever touching Google.

The index would just point a local crawler towards hubs of resources, links, feeds, and specialized search engines. Then fresh information would come from the crawler itself. My thinking is that reputable sites don't appear every day, if you update your local index once every few months it is sufficient.

The index could host 1..10 or even 100M stubs, each one touching on a different topic, and concentrating the best entry points on the web for that topic. A local LLM can RAG-search it, and use an agent to crawl from there on. If you solve search this way, without Google, and you also have local code execution sandbox, and local model, you can cut the cord. Search was the missing ingredient.

You can still call regular search engines for discovery. You can build your personalized cache of search stubs using regular LLMs that have search integration, like ChatGPT and Gemini, you only need to do it once per topic.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#92
post #83

Earlier quoted context omitted.

Unfortunately they do have a couple of points that may prove salient (though I fully agree about them being scrapers also). You can search Google _for free_ (with all the caveats of that statement), part of their grievance is that serpapi use the scraped data as a paid for service Lots of Google bot blocking is also circumvented, which they seem to have made a lot of efforts towards in the past year - robots.txt dire…

> You can search Google for free I thought the ads counted as payment? That seems to be the logic used to take technical measures against adblockers on YouTube while pushing users towards a paid ad-free subscription, at least. If viewing ads is payment, then Google isn't a free service. If viewing ads isn't payment, then Google should have no problem with people using adblockers.

I don't disagree with the logic and it definitely is/was their business model, scraping/crawling the web and subsidising the service with ads. But clicking on ads are optional.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#93

Google really doesn't have a leg to stand on here. They scrape the Internet. They replace content against the wishes of users multiple different times, such as with AMP. Their entire business model recently has been to provide you answers they learned from scraping your website and now they want to sue other people who are doing the same. Data wants to be free. They knew that once. EDIT: Also to be clear I am not say…

As the post says, Google only scrapes the websites that want to be scraped. Sure, it's opt-out (via robots.txt) rather than opt-in, but they do give you a choice. You can even decide between no scraping at all and opting out on a per-scraper basis, and Google will absolutely honor your preferences in that regard. SERP API just assumes everybody wants to be scraped, and doesn't give you a choice. (whether websites sho…

requiring me to explicitly opt-out of something is NOT the same thing as getting my consent. So your argument breaks down there.

You know what getting my consent would look like? Google hosting a form where i can tell them PLEASE SCRAPE MY WEBSITE and include it in your search results. That is what consent looks like.

Google has never asked for my consent. Yet they expect others to behave by different rules.

Now where google may have a reasonable case is that google scrapes with the intention of offering the data “for free”. SerpAPI does not.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#94

I bet SerpApi is getting more business than ever due to the Streisand effect. I hadn't heard about them, but if I want an API for Google results I'm definitely going to choose the one that was so hard for Google to block that they had to sue them instead. I see on their website they even advertise a "legal shield" where they assume scraping liability for their customers.

Can confirm, I just signed up /because/ of this announcement.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#96

"Google follows industry-standard crawling protocols, and honors websites’ directives over crawling of their content." Is that true with how they trained Gemini? Doesn't everyone with a foundational model scrape the web relentlessly without regard for robots.txt?

No, but AFAIK they pulled some shenanigans with "bundling" Gemini scraping and search engine scraping. Almost everybody wants to appear in search, so disallowing the entirety of Google is far more costly than E.G. disallowing Openai, who even differentiates between content scraped for training and content accessed to respond to a user request.

While there isn't a way to differentiate between scraping for training data and content accessed in response to a user request, I think you can block Googlebot-extended to block training access.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#97

Google really doesn't have a leg to stand on here. They scrape the Internet. They replace content against the wishes of users multiple different times, such as with AMP. Their entire business model recently has been to provide you answers they learned from scraping your website and now they want to sue other people who are doing the same. Data wants to be free. They knew that once. EDIT: Also to be clear I am not say…

As the post says, Google only scrapes the websites that want to be scraped. Sure, it's opt-out (via robots.txt) rather than opt-in, but they do give you a choice. You can even decide between no scraping at all and opting out on a per-scraper basis, and Google will absolutely honor your preferences in that regard. SERP API just assumes everybody wants to be scraped, and doesn't give you a choice. (whether websites sho…

If this is about protecting third parties from being scraped, why does Google have an interest at all? Surely Google won't have the relevant third-party data itself because, as you say, Google respects robots.txt. So how can that data be scraped from Google?

I don't think this suit is actually about that, though. I think Google's complaint is that

> SerpApi deceptively takes content that Google licenses from others

In other words, this is just a good old-fashioned licence violation.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#98
post #91

I had an idea - take SerpAPI and save top-10 or 20 links for many queries (millions), and put that in a RAG database. Then it can power a local LLM do web search without ever touching Google. The index would just point a local crawler towards hubs of resources, links, feeds, and specialized search engines. Then fresh information would come from the crawler itself. My thinking is that reputable sites don't appear ever…

Fetching web pages at the kind of volume needed to keep the index fresh is a problem, unless you're Googlebot. It requires manual intervention with whitelisting yourself with the likes of Cloudflare, cutting deals with the likes of Reddit and getting a good reputation with any other kind of potential bot blocking software that's unfamiliar with your user agent. Even then, you may still find yourself blocked from critical pieces of information.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#99

google will lose, and I'm surprised they are even trying. hiQ v. LinkedIn already settled this: scraping public web pages isn’t “unauthorized access,” even if the site says no via robots.txt or ToS. Those aren’t locks.

HiQ lost on appeal, Microsoft won

Incorrect. OP's view is present day 9th Circuit precedent.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#100
post #91

I had an idea - take SerpAPI and save top-10 or 20 links for many queries (millions), and put that in a RAG database. Then it can power a local LLM do web search without ever touching Google. The index would just point a local crawler towards hubs of resources, links, feeds, and specialized search engines. Then fresh information would come from the crawler itself. My thinking is that reputable sites don't appear ever…

Fetching web pages at the kind of volume needed to keep the index fresh is a problem, unless you're Googlebot. It requires manual intervention with whitelisting yourself with the likes of Cloudflare, cutting deals with the likes of Reddit and getting a good reputation with any other kind of potential bot blocking software that's unfamiliar with your user agent. Even then, you may still find yourself blocked from crit…

No, I think we can get by with using CommonCrawl, pulling every few months the fresh content and updating the search stubs. The idea is you don't change the entry points often, you open them up when you need to get the fresh content.

Imagine this stack: local LLM, local search stub index, and local code execution sandbox - a sovereign stack. You can get some privacy and independence back.

Post reply on HN