The State of Web Scraping 2022 • Hacker News Board

Re: The State of Web Scraping 2022

#141

Post by tzs » Thu, Jan 13, 2022, 10:34 AM UTC

KieranMac wrote: post #90

RobSm wrote: post #81

Earlier quoted context omitted.

How many contracts google breaches scraping billions of pages every month?

Given the nuances of browsewrap contract enforceability, perhaps not as many as you suggest. The tricky part with navigating this gray area is knowing the likely circumstances when a contract of adhesion may give rise to an actual legal claim. There are patterns.

What if the scraping occurs as part of web crawling?

Suppose I point a scraper at site S1, which has terms of service that say scraping them is OK, and my scraper finds a link on S1 to S2 and follows that, and follows a link from S2 to S3, and so on.

At some site Sn far enough down that chain is it really possible to use the scraper accessing that site to infer my intent to accept Sn's contract? The connection between me and Sn seems tenuous enough that it might be hard to even argue that I intended to visit Sn, let alone use that to infer acceptance of their contract.

Re: The State of Web Scraping 2022

#142

Post by Chris2048 » Thu, Jan 13, 2022, 2:16 PM UTC

nsonha wrote: post #118

RobSm wrote: post #91

Earlier quoted context omitted.

This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'. And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.

are you a lawyer? Your opinion doesn't really mean anything if you still lose the case at the end. By your logic there isn't a clear way to define DDoS either. Sounds like there is though?

> there isn't a clear way to define DDoS either

It isn't clear to me that there is. The difference seems to lie in intent.

You could maybe nail a group making many requests without using the data for anything as making many spurious requests and hence having ill-intent, I suppose. Maybe having dedicated servers for such a tasks prove it even more?

Re: The State of Web Scraping 2022

#143

Post by rronalddas » Thu, Jan 13, 2022, 4:30 PM UTC

RobSm wrote: post #139

KieranMac wrote: post #90

Earlier quoted context omitted.

Given the nuances of browsewrap contract enforceability, perhaps not as many as you suggest. The tricky part with navigating this gray area is knowing the likely circumstances when a contract of adhesion may give rise to an actual legal claim. There are patterns.

So in the scale of google, 'not many' would be some few million per month? And all is good then, right? Even you use their scrapped data probably daily and are totally fine with that, right? You think google bots read contracts before scraping website? really? :) If you had any experience in creating websites and launching them online, you would know how fast and often they arrive and how they do not care about your…

They read robots.txt right? You can easily add a Disallow rule for google-bot there

Re: The State of Web Scraping 2022

#144

Post by slvrspoon » Thu, Jan 13, 2022, 4:35 PM UTC

for those in this thread with super-serious experience scraping and automating at scale, looking for work (ethical!) please contact me directly.

Re: The State of Web Scraping 2022

#145

Post by KieranMac » Thu, Jan 13, 2022, 6:13 PM UTC

jasonfarnon wrote: post #130

KieranMac wrote: post #84

Earlier quoted context omitted.

Prof. Eric Goldman's blog is probably the #1 site historically on scraping and the law. I've contributed to it a few times. https://blog.ericgoldman.org/archives/2021/06/more-perspecti... The name of my firm is McCarthy Garber Law. I write about scraping there when I have time (which I rarely do)!

I agree that Eric's blog is great for getting updates on what's going on, and I've been following it for years. But he is very one-sided in his opinions about decisions, particularly on controversial issues like section 230. I have to remind myself he's an academic (though at a law school) and I'm not just reading some defense firm's memos.

Eric is brilliant, and he has an encyclopedic knowledge of internet law. He's also an incredibly kind, generous, and open-minded person. That said, I will refrain from any commentary on Section 230, as I have zero expertise on that issue!