Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Legality of Web Scraping

crawlnow.com

Post reply on HN
Print view View original

31–35 of 35 posts

Re: The Legality of Web Scraping

#31

Post by fiddlerwoaroof »

rootlocus wrote: post #30
fiddlerwoaroof wrote: post #2

I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.

> Ethically, if I can see the data in a web browser, I already have access to it Someone on the other side is offering you a service by providing that data. They may literally have a business, and it may be in their business's best interest to dictate how you access that data. They can filter your UA, they can rate limit, they can ban you, etc. Realistically, if the way you consume and use their public data affects t…

They can adjust how they serve traffic to me in various ways but I see no reason why they should be able to tell me that I can’t use an alternative program to view the data (a new browser I’m writing, etc.): if they’re worried about bots mass-scraping data, they have rate limits and other similar tools. If they’re worried about me saving a copy of the data I can see in normal usage of their site on my hard drive for later reference, they should mind their own business.

Re: The Legality of Web Scraping

#32

Post by JeffCarterXerox »

I think we should be looking more at intent rather than the semantics of how web scraping can be achieved.

Whether you're setting user agent strings or taking screenshots of content doesn't really matter. What matters is what you do with the content/data.

I could build a scraper to mine data on a mass scale to stick it all in a db and instantly clear it. What are my intentions here? Learn a new skill, experiment?

One example in the comments was about phone scammers. Similar phone calls have been made in jest on radio talk shows, maybe not about scamming but impersonating famous people. What differs is the intent.

Proving intent is a also difficult, as initial intent could be disguised to hide a more sinister agenda, akin to a money laundering operation. But at the root of everything will ly intent and that's what you have to get to regardless of the moral arguments.

Re: The Legality of Web Scraping

#33

Post by Dayshine »

Jensson wrote: post #14

> A website is the property of the website’s owner. No, for example the information a user puts on linkedin is that users property. The user put it on linkedin since the user wants the world to see it, so scraping linkedin to find candidates for a job doesn't violate anyone's property rights. Linkedin might still complain about server costs which is a valid concern, but they can't say that they own the data users the…

No, I didn't put it on LinkedIn for the world to see. I put it there for people to use following the T&C which prevent commercial scraping to spam me.

Re: The Legality of Web Scraping

#34

Post by hnzix »

Jensson wrote: post #14

> A website is the property of the website’s owner. No, for example the information a user puts on linkedin is that users property. The user put it on linkedin since the user wants the world to see it, so scraping linkedin to find candidates for a job doesn't violate anyone's property rights. Linkedin might still complain about server costs which is a valid concern, but they can't say that they own the data users the…

I've had my non-public LinkedIn account scraped a number of times. Thankfully waving the GDPR cannon now gets the information taken down (eventually).

Re: The Legality of Web Scraping

#35

Post by snidane »

repiret wrote: post #15
snidane wrote: post #9

Earlier quoted context omitted.

If you can twist the wording into a fraud, I can twist my words too. I don't 'change' user agent. There is no mandated default value. I 'set' it to a value that the service accepts. I don't set it to avoid blocking, I set it to be served a response. You seem to imply that some act of 'changing' a value results in a fraud. Nonsense. There is nothing like an open standard of authentication or identity without the servi…

If you send the googlebot user agent, and you’re not googlebot, that’s misrepresenting a fact. Same if you use curl and send a Firefox user agent. That’s all a fraudster needs to do - if the hearer of misrepresentation relies on it and is harmed then it’s fraud. It doesn’t matter that the user agent is trivial to misrrpresent. Just because it’s trivial for me to phone someone up and tell them that I’m from Windows su…

So are then all browsers committing fraud when they put 'Mozilla' in the user agent despite not being Mozilla based?
Post reply on HN