The Legality of Web Scraping
crawlnow.com
The Legality of Web Scraping
1–10 of 35 posts
Re: The Legality of Web Scraping
#2I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.
Re: The Legality of Web Scraping
#3I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.
In my experience of trying to calculate accurate click and open analytics, defining a user agent tells the software that they may want to exclude them and gives them an easy way to do it.
Some bots use a user agent that looks like a normal browser, so you have to try to determine patterns based on timing of events, IP address, and other data. If the request has a custom user agent, all you have to do is exclude the datapoints that have that user agent on it.
I personally always appreciate the bots that have custom user agents.
Re: The Legality of Web Scraping
#4> Trespass To Chattels is a law that governs the wrongful use of someone’s digital property.
Statements like that make me suspicious of the quality of the rest of the analysis.
Re: The Legality of Web Scraping
#5> Trespass To Chattels is a law that governs the wrongful use of someone’s digital property. Statements like that make me suspicious of the quality of the rest of the analysis.
In some situations it has, but in general this isn't true. Putting it in here as a general fact does not look good. If you look it up, it's actually controversial this sort of application. You're more likely to be hit under some version of unauthorized use of a computer system, or something of that nature.
Re: The Legality of Web Scraping
#6I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.
If you change your user agent header to avoid blocking, and the blocking is to prevent some harm, however trivial, it’s not hard to fit that into the definition of fraud: misreprenting a fact that the hearer relied on and then was harmed by that reliance [1].
Re: The Legality of Web Scraping
#7I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.
> Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.
But you have ethical arguments against Facebook and Google using the same argument to read your posts and email messages and show ads and related items on the side.
Principled technologist.
Re: The Legality of Web Scraping
#8I would strongly suggest looking at a guide from an actual law firm like Akin Gump [1] vs a web scraping site that provide a call to action like the below
>Speak to a CrawlNow data expert today to explore new opportunities for using data to fuel growth for your business.
[1] https://www.akingump.com/a/web/soxXRQ6Nw48FehNvwpdjJ1/2jiuhx...
Re: The Legality of Web Scraping
#9I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.
If you change your user agent header to avoid blocking, and the blocking is to prevent some harm, however trivial, it’s not hard to fit that into the definition of fraud: misreprenting a fact that the hearer relied on and then was harmed by that reliance [1]. [1]: https://www.law.cornell.edu/wex/fraud
If you can twist the wording into a fraud, I can twist my words too.
I don't 'change' user agent. There is no mandated default value. I 'set' it to a value that the service accepts. I don't set it to avoid blocking, I set it to be served a response.
You seem to imply that some act of 'changing' a value results in a fraud. Nonsense. There is nothing like an open standard of authentication or identity without the service authenticating explicitly = there is no fraud.
The only grey territory is if the requester denies service to others. There is a higher chance of that happening using a script or a bot, but it can equally happen if one hires a grandma to keep clicking all day. Or 1000 grandmas at a time.
Re: The Legality of Web Scraping
#10I’ve never understood why using a different user agent should make a difference. Ethically, if I can see the data in a web browser, I already have access to it and no one has any business dictating to me the programs I may use to access that data.
Accessing otherwise publicly accessible data with a different user agent is neither illegal nor unethical.