Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

Post reply on HN
Print view View original

211–220 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#211

Post by setr »

vectorEQ wrote: post #178
powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

i'm wondering if that robots.txt might then get you sued due to blocking scrapers / bots?

Robots.txt doesn't interfere with anything. It's a suggestion.

Re: Congrats! Web scraping is legal! (US precedent)

#212

Post by herval »

vectorEQ wrote: post #178
powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

i'm wondering if that robots.txt might then get you sued due to blocking scrapers / bots?

A robots.txt file doesn’t block scrapers, it’s the equivalent of a no trespassing sign. I don’t think putting up signs will suddenly become illegal?

Re: Congrats! Web scraping is legal! (US precedent)

#213

Post by Tactic »

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

The issue here for some, if not many, is a matter of scale. It is one thing if an end-user, whom I am trying to service, comes to my site and gets my publicly available data. Maybe I monetize with ads, maybe not. It doesn't matter, that is the audience I am trying to service, regardless of size.

But when you scrape it my load goes up dramatically. A load I have to pay for.

It is analogous to the privacy debates going on with one said saying "hey, don't track everywhere I go and tag me with facial recognition" and the other side saying "hey, you are in public and people can see you." The issue is not complete privacy, but one of scale. And of intent.

I believe society is soon going to have to come to grips with the scale of things and legislate what are acceptable scales of action as it seems to be becoming a large issue in a growing number are areas.

Re: Congrats! Web scraping is legal! (US precedent)

#214

Post by frandroid »

thfuran wrote: post #149
candiodari wrote: post #77

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Tortious_interference This would mostly mean that you cannot start interfering with webscraping you previously allowed merely because you learned that they're making money with the scraped data.

It seems absurd if the 'interference' only directly affects their own property. Like, if my neighbors start monetizing livestreaming my backyard, suddenly I can't put up a fence? Except worse because in actuality, this third-party contract is costing them money through server load and bandwidth.

Your backyard can be a walled garden--this is about the public front of the property.

Re: Congrats! Web scraping is legal! (US precedent)

#216

Post by phkahler »

scotty79 wrote: post #80
powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

> Seems to imply that every business is somehow beholden to every contract signed by anyone. Implied contract is that if you publish something, it's public and you have no right to dictate what software people use to consume it.

You also have no obligation to them.

Re: Congrats! Web scraping is legal! (US precedent)

#217

Post by Dayshine »

safsexisgoodsex wrote: post #215

Reading this while heading in to work on a LinkedIn shuttle brought a smile to my face. I've always been embarrassed my company took this position given how much scraping LinkedIn does. Do as I say, not as I do I guess?

And as a user of LinkedIn this news is terrible.

I share my personal data with LinkedIn under the terms you promise, which includes not sharing it with third parties for commercial use.

If you can't enforce that then I can't use LinkedIn...

I've already gone through several 6+ month long GDPR chases to find out marketing companies are scraping my data from LinkedIn without permission.

Re: Congrats! Web scraping is legal! (US precedent)

#218

Post by derefr »

Dayshine wrote: post #206
derefr wrote: post #202

Earlier quoted context omitted.

There’s a difference between “my information” and “the public webpage that I went through a publishing workflow to create from a curated selection of my information.” Let me put it this way: if I have a Wordpress blog, I’d certainly be miffed if Wordpress let bots see my drafts... but I’d also be miffed if Wordpress didn’t let bots (Google, for one!) see the published blog itself. It’s a blog; a public website! Anyon…

Well, my name, my job title, my employer, my job history. These are all my information, and selling them to marketing companies is definitely not archiving. Would you be OK with a company scraping your blog and selling it?

> Would you be OK with a company scraping your blog and selling it?

Selling it how? If they put my blog posts in a book and try to sell that book, that’s copyright infringement. If they put my blog posts in an ML model corpus to train a translation service, and they then charge pay-per-use access to the resulting service... I don’t think I’d care, nor do I think there’s anything morally or legally wrong with that. If they scrape my name and phone number and generate a Yellow-Pages-like index from them? That’s explicitly allowed by law; and heck, that’s why I embedded the information onto my site in vCard microformat in the first place!

To put my philosophy succinctly: if web.archive.org can scrape your data without you having an explicit relationship with them granting them that right, then bad.evil.com can too. You can allow both (= publicizing your information), or neither (= protecting your information), but you can’t allow one but not the other. “Third parties you don’t have a relationship with, who access your data through the public sphere without entering into a specific licensing arrangement with you” are legally one big amorphous blob. You can’t make a law that splits that blob up, because it’s an opaque blob; in the ACL system that is contract law, all entities you don’t have contracts with are just one entity—“the public.” If you want some specific entities to have access to your information, that’s what protecting your data (= setting an ACL “the public = disallow”) and then explicitly licensing it out by entering into contracts (= setting an ACL “entity X = allow”) is for.

Re: Congrats! Web scraping is legal! (US precedent)

#219

Post by etxm »

If your site has a substantial amount of crawlers, that’s an opportunity to offer an API subscription.

No one wants to “crawl” your data, they just want your data.

No software developer or PM ever said “it’d be great if this broke every couple of days and we had to scramble to figure out how to fix it” or “I’d love to jump through a bunch of hoops to read that H1”

Bot detection and crawling is an arms race. Bots will always win. I’ve droned on about this here in the past. [1]

Anecdotal evidence: Previous co-founder and software architect of a “crawler” that crawled ~17000 store and deal sites a day (like BestBuy, Gap, RMN). We circumvented all bot detection and were only busted and c&d’d once. By CouponCabin of all people.

[1] https://news.ycombinator.com/item?id=16182405

Re: Congrats! Web scraping is legal! (US precedent)

#220

Post by Nasrudith »

Wowfunhappy wrote: post #29
kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

I want web scraping to be legal—but, is it really contradictory to say "I want this data to be accessible to real humans only"? Any person can post on Hacker News. However, if someone made a bot to post to Hacker News, I think most of us would be pretty upset.

Humans cannot directly access websites - a machine is always involved. I know perfectly well what you mean but the distinction on a deeper level is fundamentally imaginary.

The best is some sort of heuristic like captchas and even they can be outsourced so that the human doing them isn't actually viewing the content.

The thing about a bot which bothers people is the behavior anyway. A bottish acting human would get people just as upset.

Post reply on HN