Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Legality of Web Scraping

crawlnow.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: The Legality of Web Scraping

#11

Post by skissane »

repiret wrote: post #6
fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

If you change your user agent header to avoid blocking, and the blocking is to prevent some harm, however trivial, it’s not hard to fit that into the definition of fraud: misreprenting a fact that the hearer relied on and then was harmed by that reliance [1]. [1]: https://www.law.cornell.edu/wex/fraud

Suppose you have a batch job which does web scraping. The job accepts a parameter which tells it what User-Agent value to use.

And you have a web page which launches the job. And that web page passes the User-Agent header from the incoming HTTP request as that job parameter.

You call the web page using curl to scrape a site. The batch job sends a curl User-Agent to the site, and the site blocks the request.

Next you try calling the web page using Google Chrome. This time the crawler batch job sends a Google Chrome User-Agent to the site, and the site allows the crawling to happen.

Did you commit fraud in the above scenario? And at which step was the fraud committed?

Re: The Legality of Web Scraping

#12

Post by matheusmoreira »

fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

Yeah. Why does a website operator care if I use a browser to view it or a script to extract some specific bit of data? It makes absolutely no sense to me.

Re: The Legality of Web Scraping

#13

Post by fiddlerwoaroof »

paxys wrote: post #10
fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

Accessing otherwise publicly accessible data with a different user agent is neither illegal nor unethical.

I don’t see why “publicly accessible data” is relevant: if I want to scrape my Facebook news feed and consume it as an RSS feed, I don’t see any justification for preventing me from doing this. The fact that I can see someone’s post in my web browser means I’m authorized to access that post.

Re: The Legality of Web Scraping

#14

Post by Jensson »

> A website is the property of the website’s owner.

No, for example the information a user puts on linkedin is that users property. The user put it on linkedin since the user wants the world to see it, so scraping linkedin to find candidates for a job doesn't violate anyone's property rights. Linkedin might still complain about server costs which is a valid concern, but they can't say that they own the data users themselves submitted regardless of what their EULA says.

Treating user submitted data as property of the host just creates lock in, I don't see any reason why that would be a good policy.

Re: The Legality of Web Scraping

#15

Post by repiret »

snidane wrote: post #9
repiret wrote: post #6

Earlier quoted context omitted.

If you change your user agent header to avoid blocking, and the blocking is to prevent some harm, however trivial, it’s not hard to fit that into the definition of fraud: misreprenting a fact that the hearer relied on and then was harmed by that reliance [1]. [1]: https://www.law.cornell.edu/wex/fraud

If you can twist the wording into a fraud, I can twist my words too. I don't 'change' user agent. There is no mandated default value. I 'set' it to a value that the service accepts. I don't set it to avoid blocking, I set it to be served a response. You seem to imply that some act of 'changing' a value results in a fraud. Nonsense. There is nothing like an open standard of authentication or identity without the servi…

If you send the googlebot user agent, and you’re not googlebot, that’s misrepresenting a fact. Same if you use curl and send a Firefox user agent. That’s all a fraudster needs to do - if the hearer of misrepresentation relies on it and is harmed then it’s fraud.

It doesn’t matter that the user agent is trivial to misrrpresent. Just because it’s trivial for me to phone someone up and tell them that I’m from Windows support and there’s a hold on their social security number and they need to wire me money to help get their grandson out of jail in Nigeria doesn’t make it not fraud if someone relies on those misrepresentations and is harmed by that reliance.

Re: The Legality of Web Scraping

#16

Post by matheusmoreira »

repiret wrote: post #6
fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

If you change your user agent header to avoid blocking, and the blocking is to prevent some harm, however trivial, it’s not hard to fit that into the definition of fraud: misreprenting a fact that the hearer relied on and then was harmed by that reliance [1]. [1]: https://www.law.cornell.edu/wex/fraud

And what harm does a simple web scraping script cause?

Re: The Legality of Web Scraping

#17

Post by hafizhamid »

htrp wrote: post #8

I would strongly suggest looking at a guide from an actual law firm like Akin Gump [1] vs a web scraping site that provide a call to action like the below >Speak to a CrawlNow data expert today to explore new opportunities for using data to fuel growth for your business. [1] https://www.akingump.com/a/web/soxXRQ6Nw48FehNvwpdjJ1/2jiuhx...

That guide from Akin Gump is listed under the Further Reading section at the end of the article. So yeah, read that too if you can!

By the way, a call to action, especially at the bottom, doesn't make the content any less credible. Nobody works hard to create quality content on the internet for the sake of it. There is always a direct or indirect motive/promotion involved. Let the readers judge whether the content is authentic or not.

Re: The Legality of Web Scraping

#18

Post by zarzavat »

repiret wrote: post #15
snidane wrote: post #9

Earlier quoted context omitted.

If you can twist the wording into a fraud, I can twist my words too. I don't 'change' user agent. There is no mandated default value. I 'set' it to a value that the service accepts. I don't set it to avoid blocking, I set it to be served a response. You seem to imply that some act of 'changing' a value results in a fraud. Nonsense. There is nothing like an open standard of authentication or identity without the servi…

If you send the googlebot user agent, and you’re not googlebot, that’s misrepresenting a fact. Same if you use curl and send a Firefox user agent. That’s all a fraudster needs to do - if the hearer of misrepresentation relies on it and is harmed then it’s fraud. It doesn’t matter that the user agent is trivial to misrrpresent. Just because it’s trivial for me to phone someone up and tell them that I’m from Windows su…

You could easily use Firefox as curl with a bit of scripting, then of course it would be using Firefox's user agent by default. There's no fraud because there is no difference between a browser user agent or a custom one. They both mean the same thing.

Re: The Legality of Web Scraping

#19

Post by Jensson »

repiret wrote: post #15
snidane wrote: post #9

Earlier quoted context omitted.

If you can twist the wording into a fraud, I can twist my words too. I don't 'change' user agent. There is no mandated default value. I 'set' it to a value that the service accepts. I don't set it to avoid blocking, I set it to be served a response. You seem to imply that some act of 'changing' a value results in a fraud. Nonsense. There is nothing like an open standard of authentication or identity without the servi…

If you send the googlebot user agent, and you’re not googlebot, that’s misrepresenting a fact. Same if you use curl and send a Firefox user agent. That’s all a fraudster needs to do - if the hearer of misrepresentation relies on it and is harmed then it’s fraud. It doesn’t matter that the user agent is trivial to misrrpresent. Just because it’s trivial for me to phone someone up and tell them that I’m from Windows su…

So when Firefox says it is chrome you consider it fraud? The user agent is there make the server better support your request, it serves no other purpose. Servers blocking certain user agents are misusing the field.

Re: The Legality of Web Scraping

#20

Post by mod »

matheusmoreira wrote: post #12
fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

Yeah. Why does a website operator care if I use a browser to view it or a script to extract some specific bit of data? It makes absolutely no sense to me.

Because you can't be targeted for ads if you don't see them.

You are their product, and they want to enforce the parameters of your relationship with them.

Post reply on HN