Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

Post reply on HN
Print view View original

131–140 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#131

Post by TremendousJudge »

vorpalhex wrote: post #90
DougWebb wrote: post #65

Earlier quoted context omitted.

One of my clients is involved in property tax collection and reporting. Property Tax records are public info, and their website allows looking up the records for any property without a login. However, the data behind this website it the _source_ of the public records, and not the public records themselves (which would be local government databases). For years now we've been in an arms race with someone using a botnet…

Wouldn't the solution be to offer a streamlined download (maybe even as a torrent if you're worried about bandwidth) of all the data then?

probably, but as GP said, "The operator of the botnet has NEVER tried to contact my client to ask for an efficient api to access the data"

Some people just don't care for the commons

Re: Congrats! Web scraping is legal! (US precedent)

#132

Post by kerkeslager »

whatshisface wrote: post #117
kerkeslager wrote: post #70

Earlier quoted context omitted.

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want i…

What if the organization is one person in an LLC? Do they get rights? If so then a big company can hire a bunch of little LLCs to act as rights-having proxies for any task that requires them.

I'm going to assume you're asking in good faith and try to address the confusion here.

The human does get rights, the organization doesn't.

In some cases, believing that humans have rights and believing that organizations have rights might lead one to the same action. In those cases, I'd take the action. I wouldn't want to violate a human's rights out of some vindictive dislike of organizations: that's not the point. The point is that I'd take that action because I believe in human rights, not because I believe that the organization has rights.

And with organizational transparency: the entire point of organizational transparency is to protect human rights. In cases where organizational transparency would trample human rights to privacy, I would go with human rights every time. Violating the privacy of humans to achieve organizational transparency would defeat the entire purpose.

Re: Congrats! Web scraping is legal! (US precedent)

#133

Post by superkuh »

Of course it's legal. Anyone holding other ideas has a fundemental misunderstanding of how the web and public spaces work. If you want something to be private then you make it private. If you make it public for the whole world to see, explicitly going out of your way to send the data to anyone who requests, it's all on you. Just because someone isn't using Chrome doesn't make them unethical.

Re: Congrats! Web scraping is legal! (US precedent)

#134

Post by bkor »

raxxorrax wrote: post #125

Since this is probably granted for anyone with a technical understanding, it is nice to see that the legislative powers are on board with this.

Just because something is possible technically doesn't make it ok legally. I think there's still various issues though, as per GPDR I don't think another company can just copy that data from Linkedin. That it's easily visible doesn't matter for GDPR.

Re: Congrats! Web scraping is legal! (US precedent)

#135

Post by kijin »

whatshisface wrote: post #117
kerkeslager wrote: post #70

Earlier quoted context omitted.

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want i…

What if the organization is one person in an LLC? Do they get rights? If so then a big company can hire a bunch of little LLCs to act as rights-having proxies for any task that requires them.

The person and the LLC are legally separate entities, although an LLC is often passed through for tax purposes.

Re: Congrats! Web scraping is legal! (US precedent)

#136

Post by mar77i »

The article's title gives me the cognitive dissonance. Glad it was changed for HN. To add my own interpretation, "US court appears to legalizes X and technically prohibits it" -- what is this "it" the court prohibits, and why do I want to assume the title's subject, which doesn't make sense at all. You don't "legalize X and technically prohibit X". Was the originally intended title truncated somehow?

Re: Congrats! Web scraping is legal! (US precedent)

#137

Post by jcranmer »

dsr_ wrote: post #55
Nitramp wrote: post #36

German copyright has the concept of a "Datenbankwerk" (since the 90s). E.g. the telephone book contains lots of boring facts that are each in themselves not copyrightable. However the collection in itself is copyrightable, as it required substantial effort to create. It seems odd that US copyright law wouldn't have a similar provision, or that it doesn't apply here?

It does, and the result is that phone books and maps get fictional entries inserted in order to prove copying -- because it is perfectly legal to do your own work to amass the same data set. https://en.wikipedia.org/wiki/Fictitious_entry

If you scroll down to the "Legal Actions" section on Wikipedia, you'll find that these copyright traps have generally failed to serve their purpose when attempted in courts.

Re: Congrats! Web scraping is legal! (US precedent)

#138

Post by Symbiote »

vorpalhex wrote: post #90
DougWebb wrote: post #65

Earlier quoted context omitted.

One of my clients is involved in property tax collection and reporting. Property Tax records are public info, and their website allows looking up the records for any property without a login. However, the data behind this website it the _source_ of the public records, and not the public records themselves (which would be local government databases). For years now we've been in an arms race with someone using a botnet…

Wouldn't the solution be to offer a streamlined download (maybe even as a torrent if you're worried about bandwidth) of all the data then?

I work on a fully open data repository. The website has the API linked in 3 places, so when I find inappropriate scraping I block it with "HTTP 420 ... see or contact ".

Some people probably switch to using the API, but no-one has ever contacted us. They either give up, or run their scraper on a different computer -- I've seen the same scraper move between university computers, departments, then (in the evening) to a consumer broadband IP.

Re: Congrats! Web scraping is legal! (US precedent)

#139

Post by michaelbuckbee »

nathanlied wrote: post #84
powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

> Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second? ToS are subservient to the law; you can (probably) terminate a service account from a user that breaks your ToS, but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS do…

I'm trying to puzzle out how this works in practice. So if LinkedIn has truly public data (no login required to view) then it can be scraped no problem.

But if it's only accessible with a login, then it falls under TOS and they can be blocked?

Re: Congrats! Web scraping is legal! (US precedent)

#140

Post by icedchai »

francasso wrote: post #15

So, if company sells a product based on scraping google search results, google trying to block scraping would constitute "malicious interference with a contract"?

This is interesting to to think about, considering Google gets its own results from scraping...
Post reply on HN