Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we're taking legal action against SerpApi's unlawful scraping

blog.google

Post reply on HN
Print view View original

101–110 of 113 posts

Re: Why we're taking legal action against SerpApi's unlawful scraping

#101

Post by gnfargbl »

bitbasher wrote: post #75
randyrand wrote: post #67

google will lose, and I'm surprised they are even trying. hiQ v. LinkedIn already settled this: scraping public web pages isn’t “unauthorized access,” even if the site says no via robots.txt or ToS. Those aren’t locks.

HiQ lost on appeal, Microsoft won

No, it's more complicated than that: https://www.morganlewis.com/blogs/sourcingatmorganlewis/2022...

The short answer is that scraping isn't a CFAA offence but might be a terms and conditions violation, depending on the specifics of the access.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#102

Post by jofzar »

ricardo81 wrote: post #92
crote wrote: post #83

Earlier quoted context omitted.

> You can search Google for free I thought the ads counted as payment? That seems to be the logic used to take technical measures against adblockers on YouTube while pushing users towards a paid ad-free subscription, at least. If viewing ads is payment, then Google isn't a free service. If viewing ads isn't payment, then Google should have no problem with people using adblockers.

I don't disagree with the logic and it definitely is/was their business model, scraping/crawling the web and subsidising the service with ads. But clicking on ads are optional.

No google's business model is showing you ads, not clicking on them. That's the job of the person who designs the ad.

Google would like you to click through as it looks better for their stats, but they don't actually care.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#103

Post by ricardo81 »

visarga wrote: post #100
ricardo81 wrote: post #98

Earlier quoted context omitted.

Fetching web pages at the kind of volume needed to keep the index fresh is a problem, unless you're Googlebot. It requires manual intervention with whitelisting yourself with the likes of Cloudflare, cutting deals with the likes of Reddit and getting a good reputation with any other kind of potential bot blocking software that's unfamiliar with your user agent. Even then, you may still find yourself blocked from crit…

No, I think we can get by with using CommonCrawl, pulling every few months the fresh content and updating the search stubs. The idea is you don't change the entry points often, you open them up when you need to get the fresh content. Imagine this stack: local LLM, local search stub index, and local code execution sandbox - a sovereign stack. You can get some privacy and independence back.

CC is not on the same scale as Google and not nearly as fresh. It's around 100th of the size and not much chance of having recent versions of a page.

I imagine you'd get on just fine for short tail queries but the other cases (longer tail, recent queries, things that haven't been crawled) begin to add up.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#104

Post by sublimefire »

I bet the core of the problem for Google is that more folks use programmatic access to search which is not great on their side. Naturally you end up using Serp or other similar search APIs as they are great for the job. I believe this is also an issue especially in cases where search is performed on behalf of the user (scripts, ai tools). Google is just losing ground here, why would they bother otherwise, think what will happen to their stock if the search usage will drop? Another thing is that this builds pressure to whoever is integrating such a search tool in their products, clearly Google wants to grab that market as well.

Re: Why we're taking legal action against SerpApi's unlawful scraping

#105

Post by WhatsName »

Isn't search engine results a product that Google offers? [1] I find the argument quite strange that website owners agreed to Google being able to do anything with that data beyond displaying them in their search results when they wrote that robots.txt maybe ten years ago, but others shall not access those results programatically.

I certainly did not and find using the content google scraped from my website for money or AI (which they also sell on a token basis) more questionable than some third party offering API access to it.

[1] https://docs.cloud.google.com/generative-ai-app-builder/docs...

Re: Why we're taking legal action against SerpApi's unlawful scraping

#107

Post by falloutx »

modeless wrote: post #76

I bet SerpApi is getting more business than ever due to the Streisand effect. I hadn't heard about them, but if I want an API for Google results I'm definitely going to choose the one that was so hard for Google to block that they had to sue them instead. I see on their website they even advertise a "legal shield" where they assume scraping liability for their customers.

And its API seems really easy to use.
Post reply on HN