Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

Post reply on HN
Print view View original

311–320 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#311

Post by orf »

Tactic wrote: post #242
orf wrote: post #237

Earlier quoted context omitted.

Provide an API for public data to reduce the costs associated with rendering a full blown page, and deliver just the information needed.

Entirely feasible. Also reasonable for you to pay me for the service as it is taking my development efforts to meet your business model. The advantage to you is you have a defined interface that I won't prevent.

I guess you missed the comment I was replying to: it may cost you more money, in bandwidth and per page resources, to not provide an API than it does for you to provide one.

So no, I won’t pay you for the privilege of you saving money.

Re: Congrats! Web scraping is legal! (US precedent)

#312

Post by MarioMan »

protanopia wrote: post #297
powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

> If LinkedIn wanted to force users to sign in to view profile info Do they not already do this? Every link I've ever seen for LinkedIn has redirected me to sign up page rather than showing me the content.

They block you out after the first few profiles you view. Try a private browser and you can still see them.

Re: Congrats! Web scraping is legal! (US precedent)

#313

Post by KaoruAoiShiho »

Jolter wrote: post #9
amelius wrote: post #7

> Now many site owners are trying to put technical obstacles to competitors who completely copy their information that is not protected by copyright. For example, ticket prices, product lots, open user profiles, and so on. Some sites consider this information “their own”, and consider web scraping as “theft”. Legally, this is not the case, which is now officially enshrined in the US. Does this mean we can now scrape…

Youtube videos are definitely protected by copyright, though.

Shouldn't the copyright belong to the creator not to youtube? Basically youtube shouldn't be able to sue you, it should be up to the creator to do so.

Re: Congrats! Web scraping is legal! (US precedent)

#315

Post by munk-a »

moron4hire wrote: post #246
Cerium wrote: post #203

Earlier quoted context omitted.

It is true in the current situation, though I would prefer that we ensure free data must be free. In that case buyers of data would be incentivized to pressure providers of free data to improve the data quality.

The data does remain free, as long as LinkedIn still provides it for free. The data without the noise is what you're paying for. The service of winnowing out what you care about from what you don't care about. Considering how big of an effort it is, and that the source from which it came is still available, why should the cleaned data be free? If I collect fallen trees from public land, chop it into usable firewood,…

I think this is especially relevant when it comes to things that fall under disclosure & transparency requirements - a lot of information that is legally required to be made available isn't legally required to be convenient. So, as a patient, you may have the absolute right[1] to a free copy of the charge master[2] of a hospital you're admitted to but it could be required that you pick it up in person or that it is only supplied in microfliche form... so a company that's aggregated this and is reselling it can deliver real value.

1. This specific example is BS but plausible - I just wanted something more specific than the vagaries around things like FOIAs or shareholder reports which both have specific facts that can be rendered useless unless you have the context.

2. Basically, list of how much procedures cost.

Re: Congrats! Web scraping is legal! (US precedent)

#316

Post by make3 »

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

they want there data to be public for a specific use, I feel like that's pretty easy to understand. LinkedIn: they want their info to be public -> to get jobs

Re: Congrats! Web scraping is legal! (US precedent)

#317

Post by Pigo »

bhousel wrote: post #24
Pigo wrote: post #16

I always thought scraping was a fun idea, I just couldn't find the right use-case for it. I'm not a sports guy, and the big sites have pretty extensive API's. Something music related would be of interest to me, but I can already get updates on events like concerts (since the monopolies make a fortune selling tickets). I'm not sure what could be useful.

We can use your help mapping all the businesses of the world! https://www.alltheplaces.xyz/

Interesting, I'll check that out.

Re: Congrats! Web scraping is legal! (US precedent)

#319

Post by zackmorris »

AznHisoka wrote: post #5

Linkedin is taking this to the Supreme Court: https://www.law360.com/articles/1237505/linkedin-will-go-to-... No ultimate decision was ever made, and no, this doesn't make web scraping 100% legal. Wake me up when there's a new announcement because anyone interested in this already know this old news.

This is a really big deal. Currently (IMHO) the US Supreme Court is a wholly-owned subsidiary of multinational corporations due to the shenanigans that happened with Obama, McConnell and Garland, so will likely side with LinkedIn since it's the larger corporation:

https://www.npr.org/2018/06/29/624467256/what-happened-with-...

I feel like siding with LinkedIn here would open up the web to extortion though, like troll companies that would send cease and desist letters to all scrapers (even search engines). I think it could be argued that letting one company scrape when another is denied is discrimination.

Then again, I don't know how conservative and republican-leaning courts decide corporate law. Maybe in this case since so much money is at stake, they might worry that banning scraping would infringe on something like free speech and ruffle the feathers of some of the wealthier contributors in their base. Especially on the media side since I imagine they use bots in one form or another to find newsworthy stories.

IANAL (obviously!), I just find it entertaining/dismaying to ponder these things in these times.

Re: Congrats! Web scraping is legal! (US precedent)

#320

Post by turc1656 »

cabaalis wrote: post #244
michaelmior wrote: post #223

Earlier quoted context omitted.

IANAL, but it seems like ToS could still govern your use of the data which you viewed. Sure, it seems like you couldn't claim any violation based on visiting a random page. But if the ToS is clearly identified on the page and you do something with the data that violates them, perhaps the owner of the site has a case.

> perhaps the owner of the site has a case. Except it sounds like the owner doesn't. If the information is on the page made public, the owner of the page can't place terms on what is done with the data downstream. They'd have to implement some real binding system such as authentication where CFAA would apply. (IANAL)

Correct, but all of that is void if the data presented is any sort of protected information (copyright, IP, etc.). You can't, for example, scrape Yahoo Finance for pricing and dividend history and republish on your own stock tools website. They have a license to redistribute that data and publish on their own website. Similar story for copyrighted text and things of that nature.
Post reply on HN