Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

Post reply on HN
Print view View original

321–330 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#321

Post by moron4hire »

ChrisMarshallNY wrote: post #306
moron4hire wrote: post #246

Earlier quoted context omitted.

The data does remain free, as long as LinkedIn still provides it for free. The data without the noise is what you're paying for. The service of winnowing out what you care about from what you don't care about. Considering how big of an effort it is, and that the source from which it came is still available, why should the cleaned data be free? If I collect fallen trees from public land, chop it into usable firewood,…

Absolutely spot-on. I'm thinking of processed GIS data. If you have ever tried using the various formats that are supplied by government sites, you know what a huge pain it is. I'm happy to pay a reasonable price for an interpreted and bowdlerized version.

I actually have! I had to import a huge file of all of the culverts around storm drains in a state, and each culvert was multiple pieces of geometry, none of them grouped together in any logical way. It was just a huge list of rectangles that looked like culverts when viewed visually but no way to identify them as being one culvert without heuristics on how close each rectangle was to others. Massively long process that should not have been so.

Re: Congrats! Web scraping is legal! (US precedent)

#322

Post by alok-g »

protanopia wrote: post #297
powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

> If LinkedIn wanted to force users to sign in to view profile info Do they not already do this? Every link I've ever seen for LinkedIn has redirected me to sign up page rather than showing me the content.

As a user of LinkedIn, I can pick which portions of my profile information I would like to be publicly available. This is not by default, so most people do not have it public. You can try seeing my profile without logging in. :-)

Re: Congrats! Web scraping is legal! (US precedent)

#323

Post by gyvastis »

grepfru_it wrote: post #8
amelius wrote: post #7

> Now many site owners are trying to put technical obstacles to competitors who completely copy their information that is not protected by copyright. For example, ticket prices, product lots, open user profiles, and so on. Some sites consider this information “their own”, and consider web scraping as “theft”. Legally, this is not the case, which is now officially enshrined in the US. Does this mean we can now scrape…

Yes you can scrape them, no you cannot repubilsh them. Everything you listed is protected by copyright. You cannot infringe on copyrights because of this ruling. >hiQ argued that LinkedIn’s technical measures to block web scraping interfere with hiQ’s contracts with its own customers who rely on this data. In legal jargon, this is called” malicious interference with a contract”, which is prohibited by American law Do…

I'm curious how entities like https://www.omdbapi.com/ can continue their activity, get $$$ and not get shut down.

Re: Congrats! Web scraping is legal! (US precedent)

#324

Post by CJKinni »

echelon wrote: post #184

So many ideas start to come to mind if scraping is legal. Can we start to scrape Google Search in order to bootstrap building an alternative to Google Search? Search is a really hard problem (that somebody should tackle), but if we can leverage what Google has already scraped from the web and associated with popular search terms, we can use that to help train and validate our search model. Can we scrape Reddit, Twitt…

> Can we scrape Reddit, Twitter, or Facebook in order to stand up a competing service that strips out all the ads?

Even if web scraping was definitively legal (this preliminary injunction doesn't mean that), that doesn't mean you can bypass the content creator's copyright. Non-copyrightable functional data is one thing, but copying all of Reddit, for example, would include copying https://www.reddit.com/r/WritingPrompts/ and that would definitely be violating the rights of the authors.

Re: Congrats! Web scraping is legal! (US precedent)

#325

Post by senorjazz »

jackfoxy wrote: post #299

> The CFAA is adopted to prevent deliberate intrusion on someone else’s computer — in particular, computer hacking hacking is commonly used to mean different things, and we imply from the context what the author really meant. Do any lawyers know if computer hacking has a legal definition, or if this decision will lead to a specific legal definition of the term?

From what I understand it is very loose and can mean "you accessed our systems legitmately but we didn't want you to".

I think it was last year that facebook went after fake likes / fake followers companies who were logging via the login page and they liking / following. Companies in China / New Zealand and I think NY? got threatened with the CFAA (unsure if it went further) but it made the tech news at the time.

I don't think anyone hear would call writing a bot that logged in with actual details and performed an action (the same a real person could) as hacking, but facebook were saying as it broke the TOS, it was unauthorised, thus hacking laws apply. With their budget, I guess they get to decide what hacking is

Re: Congrats! Web scraping is legal! (US precedent)

#326

Post by markandrewj »

andrewmutz wrote: post #295
beerandt wrote: post #241

Earlier quoted context omitted.

Terms of Service is a form of contractual agreement, which requires there be an offer and subsequent agreement by the parties. I don't think criminal law was ever part of this.

From the article, the LinkedIn decision was that scraping data does not violate the Computer Fraud and Abuse Act. Violating that act was considered to be criminal activity. ( https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act )

The CFAA is overreaching, and used often as a catch all. 'Reply All' has a good episode which explores this. This is actually what was used against Aaron Swartz when he was charged for downloading academic journals from MIT, and why his charges were unjustly severe.

Reply All - #43 The Law That Sticks https://gimletmedia.com/shows/reply-all/rnhoxb

Re: Congrats! Web scraping is legal! (US precedent)

#327

Post by munk-a »

hannasanarion wrote: post #305
Tactic wrote: post #213

Earlier quoted context omitted.

The issue here for some, if not many, is a matter of scale. It is one thing if an end-user, whom I am trying to service, comes to my site and gets my publicly available data. Maybe I monetize with ads, maybe not. It doesn't matter, that is the audience I am trying to service, regardless of size. But when you scrape it my load goes up dramatically. A load I have to pay for. It is analogous to the privacy debates going…

So you throttle your users. We have http status codes for "too many requests" and all scraper software comes with a delay setting by default. Everybody who does scraping is supposed to know that its rude to blast a thousand requests per second.

This ruling has left open a big question of how much you need to spend to support scrapers and where the line between scraping and a DoS attack lies - and that's going to be a weird line. If my site is producing a big report off of data that changes quarterly then re-downloading that report every 20 minutes is possibly excessive and might wander into the realm of an attack - while if we looked at the same frequency with twitter it seems a lot more reasonable - maybe even a bit on the slow side.

Re: Congrats! Web scraping is legal! (US precedent)

#328

Post by munk-a »

munificent wrote: post #298
0x445442 wrote: post #269

Earlier quoted context omitted.

My guess is hiQ (and others) would happily pay for an API over the data they're scraping right now.

Why buy the cow when you get the milk for free?

This isn't a great analogy here - getting the data delivered via API is simply more useful than having to re-assemble that data out of fragments parsed off of different web calls.

Could I suggest:

"Why buy the cheese when you get the milk for free?"

Re: Congrats! Web scraping is legal! (US precedent)

#330

Post by mic47 »

Dayshine wrote: post #193
Wowfunhappy wrote: post #29

Earlier quoted context omitted.

I want web scraping to be legal—but, is it really contradictory to say "I want this data to be accessible to real humans only"? Any person can post on Hacker News. However, if someone made a bot to post to Hacker News, I think most of us would be pretty upset.

What you might mean is: - I want my data to be publicly available - I don't want my data to be processed/distributed/sold without my permission E.g. individual use is fine, profit making is not. Which is my expectation with LinkedIn. I want people to see my profile, I don't want them to sell it as marketing leads!

Agree with this, and I would also add this - I want to be in control of my data and change the setting. - I want to be able to delete my data.

If scraping on linked-in is banned (and linked-in is enforcing it), then I do have have control of my data, since I can change the setting, and it will no longer be public (It's not perfect, since some might already scraped it, but the extend would be much smaller). Also, if I decide to delete my data, linked can do that for data it control, but not for scraped data.

Post reply on HN