Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Legality of Web Scraping

crawlnow.com

Post reply on HN
Print view View original

21–30 of 35 posts

Re: The Legality of Web Scraping

#21

Post by paxys »

fiddlerwoaroof wrote: post #13
paxys wrote: post #10

Earlier quoted context omitted.

Accessing otherwise publicly accessible data with a different user agent is neither illegal nor unethical.

I don’t see why “publicly accessible data” is relevant: if I want to scrape my Facebook news feed and consume it as an RSS feed, I don’t see any justification for preventing me from doing this. The fact that I can see someone’s post in my web browser means I’m authorized to access that post.

The difference is when you created a Facebook account and logged in using your username & password you agreed to their terms of service.

Re: The Legality of Web Scraping

#22

Post by fiddlerwoaroof »

paxys wrote: post #21
fiddlerwoaroof wrote: post #13

Earlier quoted context omitted.

I don’t see why “publicly accessible data” is relevant: if I want to scrape my Facebook news feed and consume it as an RSS feed, I don’t see any justification for preventing me from doing this. The fact that I can see someone’s post in my web browser means I’m authorized to access that post.

The difference is when you created a Facebook account and logged in using your username & password you agreed to their terms of service.

A document no one reads can’t really be a contract, regardless of what the law says.

Re: The Legality of Web Scraping

#23

Post by pdonis »

fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

If your scraping program is fetching data at a similar rate and in a similar way to a browser, then yes, it shouldn't make any difference to the website owner. But most scraping programs are not like that. A web browser is fetching data for a human to read; that means it is not going to, for example, fetch a thousand pages from your website in just a few seconds using many parallel connections. But many scraping programs do things like that.

Re: The Legality of Web Scraping

#24

Post by paxys »

fiddlerwoaroof wrote: post #22
paxys wrote: post #21

Earlier quoted context omitted.

The difference is when you created a Facebook account and logged in using your username & password you agreed to their terms of service.

A document no one reads can’t really be a contract, regardless of what the law says.

Well sadly what the law says matters

Re: The Legality of Web Scraping

#25

Post by matheusmoreira »

mod wrote: post #20
matheusmoreira wrote: post #12

Earlier quoted context omitted.

Yeah. Why does a website operator care if I use a browser to view it or a script to extract some specific bit of data? It makes absolutely no sense to me.

Because you can't be targeted for ads if you don't see them. You are their product, and they want to enforce the parameters of your relationship with them.

I won't see ads either way. If I choose to use a browser, uBlock Origin will take care of any ads.

> You are their product, and they want to enforce the parameters of your relationship with them.

They're in for a shock then. I'm not a passive "product" to be sold to advertisers. Treating human beings like that is abuse.

Re: The Legality of Web Scraping

#26

Post by Jensson »

paxys wrote: post #24
fiddlerwoaroof wrote: post #22

Earlier quoted context omitted.

A document no one reads can’t really be a contract, regardless of what the law says.

Well sadly what the law says matters

Are you sure? Doesn't courts typically throw out cases where companies try to argue that those are binding?

Re: The Legality of Web Scraping

#27

Post by mewse »

fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

Ethically, if someone is quite clearly using the user agent string to function as a lock restricting access to their IP, does it really matter that it’s an objectively terrible lock?

I mean, since you’ve brought ethics into it.

Re: The Legality of Web Scraping

#28

Post by 1vuio0pswjnm7 »

pdonis wrote: post #23
fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

If your scraping program is fetching data at a similar rate and in a similar way to a browser, then yes, it shouldn't make any difference to the website owner. But most scraping programs are not like that. A web browser is fetching data for a human to read; that means it is not going to, for example, fetch a thousand pages from your website in just a few seconds using many parallel connections. But many scraping prog…

This seems to suggest the only thing that should matter to the website owner is the manner of retrieving data, not the data being retrieved. If it is public data, that makes sense.

I like to use sitemaps to download websites. I only use a single TCP connection and HTTP/1.1 pipelining. This can of take several successive requests depending on the number of URLs and the website's request limit per connection. For example, Akin Gump's sitemap has 9174 URLs and allows at least 2000 requests per connection. OTOH, I have downloaded sites with more URLs than that in a single connection. Pipelining is slower than parallel connections, more akin to downloading a large file. Some websites are faster than others.

According to the original RFCs, this is the proper netiquette. Even though the practice is of using parallel connections is widespread, I have never been able to find an RFC that advocated that. Popular browsers today open dozens of connections. All in the name of advertising.

I take the absence of a sitemap as an indication the website owner may have issues with automated data retrieval. For example, LinkedIn has no sitemap. However, most websites I encounter have sitemaps. Downloading websites without using parallel connections is not difficult. I do not need special software, only a small binary written in C to generate HTTP and a netcat. (Neither curl nor wget nor similar clients can do HTTP/1.1 pipelining.) I do not use a user-agent header. I have a filter I wrote for chunk transfer doecoding from stdin.

Re: The Legality of Web Scraping

#29

Post by paxys »

Jensson wrote: post #26
paxys wrote: post #24

Earlier quoted context omitted.

Well sadly what the law says matters

Are you sure? Doesn't courts typically throw out cases where companies try to argue that those are binding?

No, it depends on what courts consider "unconscionable". We will likely never know for simple cases like changing your user agent because Facebook isn't going to sue someone over it, but they have fought (and won) against wholesale scraping of user profiles, which is mentioned in the linked article.

Re: The Legality of Web Scraping

#30

Post by rootlocus »

fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

> Ethically, if I can see the data in a web browser, I already have access to it

Someone on the other side is offering you a service by providing that data. They may literally have a business, and it may be in their business's best interest to dictate how you access that data. They can filter your UA, they can rate limit, they can ban you, etc.

Realistically, if the way you consume and use their public data affects their revenue, they can (and should) act to protect their income.

Post reply on HN