Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

Post reply on HN
Print view View original

161–170 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#161

Post by stanferder »

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

I don't think the direction would ever be clear, even if the legality were clearly established. The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field.

Companies want to provide some information to some people; but providing all information to all people is analogous to allowing customers to make a meal of free food samples, on a recurring basis.

Re: Congrats! Web scraping is legal! (US precedent)

#162

Post by Nasrudith »

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

It sounds like the same old issue - lack of understanding of the fundamentals underlying technologies.

Otherwise they would realize what they demand is contradictory and incoherent like demanding to be both viewable by all and not viewable. DRM is one fundamental example of it.

Re: Congrats! Web scraping is legal! (US precedent)

#163

Post by msla »

kerkeslager wrote: post #70
kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want i…

Making organization membership public would trample on personal privacy quite effectively in some respects, such as with disease support groups or PACs; medical privacy is taken seriously, but is there such a thing as political affiliation being private? Is it a violation of someone's privacy to reveal they give to the ACLU?

Re: Congrats! Web scraping is legal! (US precedent)

#164

Post by TrackerFF »

buboard wrote: post #93
TrackerFF wrote: post #45

Earlier quoted context omitted.

I can understand why some do not want scrapers - increased traffic (with practically zero benefits to the owners) is one obvious reason. (Some people will then say "But why not just offer APIs", but that's a lot of extra work and maintenance). It's like with instagram and other social media platforms. The content creators put in the hard work, while the leeches are stealing content for their own benefit, giving zero…

Instagram is not the content creator tho

What I'm trying to say is: For every popular content creator on IG, there are tens and hundreds of (more or less automatic) content curators that do nothing more than scrape content with lots of likes, and re-post on their own channels. Then when they get sufficient followers, they make money through paid product placements, account flipping, pay-to-play sharing, and what not. More often than not, there's no linking to the original pages / creators.

Re: Congrats! Web scraping is legal! (US precedent)

#166

Post by sjg007 »

levosmetalo wrote: post #152
sjg007 wrote: post #89

Earlier quoted context omitted.

It's not the same thing at all. The film is broadcast without re-transmission rights.

How is that different? Where on my website I gave anyone "re-transmission" or "re-publishing" rights?

It's different for broadcast films and tv https://en.wikipedia.org/wiki/Retransmission_consent

Re: Congrats! Web scraping is legal! (US precedent)

#167

Post by mindcrime »

stanferder wrote: post #161
kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

I don't think the direction would ever be clear, even if the legality were clearly established. The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field. Companies want to provide some information to some people; but providing all information to all people is analogous to allowing customers to make a meal of free food samples, on a recurring basis.

The arms race would intensify, and detecting/blocking/deceiving scrapers might become a lucrative field.

It already is. There are entire companies, like Distil Networks, who exist solely to protect companies from bots/scrapers/etc. Actually, looks like Distil got acquired and are now part of Imperva, but anyway, the idea is the same. This is definitely an existing field.

https://www.imperva.com/products/bot-management/

Disclosure: former Distil employee, but I have no financial stake in this discussion, and have mixed feelings about scraping. Clearly it can be beneficial in some situations, but when I think about having to pay exorbitant prices to scalpers for tickets to an event, because they used a bot to buy up all the tickets, that is less appealing.

Re: Congrats! Web scraping is legal! (US precedent)

#168

Post by magduf »

Nitramp wrote: post #36

German copyright has the concept of a "Datenbankwerk" (since the 90s). E.g. the telephone book contains lots of boring facts that are each in themselves not copyrightable. However the collection in itself is copyrightable, as it required substantial effort to create. It seems odd that US copyright law wouldn't have a similar provision, or that it doesn't apply here?

I'm pretty sure the US has the same thing, basically. You can't copyright facts, but you can copyright presentation. So someone can't copy your map directly and sell it, because there's an artistic component to it. But they can make their own map with the same data, with a different style to it, and sell that.

Re: Congrats! Web scraping is legal! (US precedent)

#169

Post by barrkel »

magduf wrote: post #147
Symbiote wrote: post #138

Earlier quoted context omitted.

I work on a fully open data repository. The website has the API linked in 3 places, so when I find inappropriate scraping I block it with "HTTP 420 ... see or contact ". Some people probably switch to using the API, but no-one has ever contacted us. They either give up, or run their scraper on a different computer -- I've seen the same scraper move between university computers, departments, then (in the evening) to a…

I really don't understand why anyone would bother writing and using a web scraper when an API exists. Does the API not provide all the same data/functions as the website? Scrapers are a big PITA compared to just using an API: they're much harder to write to be reliable, and they can break at any time, whenever the site makes even the smallest change. APIs avoid all that mess, and make performance far better too (on b…

APIs are often not as complete as the web interface, since the customer sees the web interface and normally the customer is what drives the revenue model of the company.

If pages are driven via an API, then the API is preferable, but publicly facing websites are often a mix of server-side HTML generation and API enrichment, for caching if nothing else.

Re: Congrats! Web scraping is legal! (US precedent)

#170

Post by andrewmutz »

nathanlied wrote: post #84
powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

> Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second? ToS are subservient to the law; you can (probably) terminate a service account from a user that breaks your ToS, but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS do…

> but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS does not apply, since you've technically not entered a binding legal contract with them.

Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account.

I have interpreted the LinkedIn ruling to mean that scraping public data is no longer criminal activity but it still leaves you open to civil lawsuits for violating the ToS of the website you are scraping.

Post reply on HN