Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

331–340 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#331

So many ideas start to come to mind if scraping is legal. Can we start to scrape Google Search in order to bootstrap building an alternative to Google Search? Search is a really hard problem (that somebody should tackle), but if we can leverage what Google has already scraped from the web and associated with popular search terms, we can use that to help train and validate our search model. Can we scrape Reddit, Twitt…

> Can we finally scrape and get rid of IMDB? I'd love to put all of their content on a wiki and be done with it.

Just because you can scrape the content legally does not mean you can also republish it on your own website.

Re: Congrats! Web scraping is legal! (US precedent)

#332

Earlier quoted context omitted.

> Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second? ToS are subservient to the law; you can (probably) terminate a service account from a user that breaks your ToS, but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS do…

There's a long, long history (probably hundreds –if not thousands– of years old) of selling aggregated or processed publicly-available information. I'm not particularly thrilled with it, but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free. LinkedIn users (as opposed to the company) might actually like what HiQ is doing, as it may help their o…

[deleted]

Re: Congrats! Web scraping is legal! (US precedent)

#333
I guess this means the web is lawfully seen as belonging to the public which seems both good and bad, because while it might be publicly accessible as a whole, each individual site is privately owned.

I'm pretty sure with any brick and mortar business, as a business owner you are legally allowed to disallow service to anyone without having to provide a reason.

That would be the equivalent of allowing to block bots / unwanted traffic on your site, except the brick and mortar case is way easier to prevent. If the person won't leave your physical business, you can have them removed by the police but to prevent a bot the burden is fully on you as a site owner.

Re: Congrats! Web scraping is legal! (US precedent)

#334
post #242
post #237

Earlier quoted context omitted.

Provide an API for public data to reduce the costs associated with rendering a full blown page, and deliver just the information needed.

Entirely feasible. Also reasonable for you to pay me for the service as it is taking my development efforts to meet your business model. The advantage to you is you have a defined interface that I won't prevent.

No, I'll happily just scrape your site instead. But if you'd rather not have that happen, provide an API.

Re: Congrats! Web scraping is legal! (US precedent)

#335
Please not that I am personally more pro scraping that anti. However, this article's interpretation of this ruling (which is from September 2019) is overblown and suspect:

1) This is the 9th circuit only. The 9th circuit is known for being very liberal in these kids of rulings. There is a high likelihood that this ruling could be interpreted differently in other circuits and when it goes to the supreme court.

2) Saying "it makes it legal" is more of a technicality with this ruling and NOT practicality. There is still a LOT open to interpretation with this ruling (see 1, above). And lot of what it's saying just isn't found in the case. And again, it doesn't apply everywhere.

3) This is actually a much better summary of the case: https://www.natlawreview.com/article/data-scraping-survives-... In fact, there's a number of better summaries are out there by people who actually understand both technology and law from when it happened (again, back in September 2019). Here's some others:

https://www.cooley.com/-/media/cooley/pdf/reprints/2019/2019...

https://www.techdirt.com/articles/20190909/17571342951/big-n...

Honestly, it comes back to a lot of existing issues, like whether facts can be copyrighted (e.g., this is why recopies alone can't be), whether logins imply privacy, etc.

I personally think the ruling is in the right direction but again, I don't think this source deals with any of the complexity or the reality of what the ruling does (or still doesn't) mean.

Re: Congrats! Web scraping is legal! (US precedent)

#336
post #262

Earlier quoted context omitted.

Your analogy doesn't hold. Your backyard is private property. The data that LinkedIn publishes is intended for the public. That's why Google can index the pages and give you results from LinkedIn.

Does that mean that ia grocery store offers free samples, I can go in every day and take all the samples, and the grocery is not allowed to selectively prevent me access?

It means that if they're offering free samples and refuse to offer you the same service they're offering to other customers they might be in hot water - which is consistent with what a lot of folks consider ethical. Offering an item for free to some folks and not to others is a form of discrimination - it's usually not a particularly troubling form of discrimination but in this case Google is allowed to walk up and take all the samples and the grocery store manager just smiles and nods - but when you (hiQ in this example) try and get one you're hit with an injunction and barred from entry.

Re: Congrats! Web scraping is legal! (US precedent)

#337
post #333

I guess this means the web is lawfully seen as belonging to the public which seems both good and bad, because while it might be publicly accessible as a whole, each individual site is privately owned. I'm pretty sure with any brick and mortar business, as a business owner you are legally allowed to disallow service to anyone without having to provide a reason. That would be the equivalent of allowing to block bots /…

Yeah, but this is slightly different. This is about a business owner trying to control who can look through the windows.

Re: Congrats! Web scraping is legal! (US precedent)

#339
post #310

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

> This is almost weirder. If LinkedIn wanted to force users to sign in to view profile info, would they be not allowed to do that because some company had signed a contract that implicitly assumed access to that data? If someone writes a web scraper for my site, and I unknowingly change my site in a way that breaks that scraper, can a court force me to revert the change? LinkedIn has long wanted to have their cake an…

This reminds me about Twitter, when I click to see a thread for a tweet it asks me to login, but if I open the link in a new tab it loads the thread just fine.

Re: Congrats! Web scraping is legal! (US precedent)

#340
post #333

I guess this means the web is lawfully seen as belonging to the public which seems both good and bad, because while it might be publicly accessible as a whole, each individual site is privately owned. I'm pretty sure with any brick and mortar business, as a business owner you are legally allowed to disallow service to anyone without having to provide a reason. That would be the equivalent of allowing to block bots /…

> as an owner you are legally allowed to disallow service to anyone without having to provide a reason

as a website owner you absolutely have the right to limit who does or doesn't visit your site, but that's up to you to enforce. Scraping is literally equivalent to reading a giant public billboard and writing what it says somewhere else. How could that be illegal?

Post reply on HN