Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CloudScrape – Cloud-based web scraping platform

cloudscrape.com

Post reply on HN
Print view View original

91–100 of 109 posts

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#91

Post by manigandham »

fauigerzigerk wrote: post #56
manigandham wrote: post #50

Earlier quoted context omitted.

What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular. Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't…

>What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to…

Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline.

We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#92

Post by manigandham »

yxdfasdjkljasdf wrote: post #76
manigandham wrote: post #49

Earlier quoted context omitted.

Since there's no easy way to always reliably identify the requester, this gets complicated. Most scrapers - including this one - advertise how they use multiple servers/locations/ips/etc to get around this.

I fail to see a problem you are trying to present. Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.

> If you don't want someone to access your page, then don't respond to their request

> there's no easy way to always reliably identify the requester

That's the problem: you can't identify the person to block them in the first place.

Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else.

Not sure what your last sentence means.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#93

Post by fauigerzigerk »

manigandham wrote: post #91
fauigerzigerk wrote: post #56

Earlier quoted context omitted.

>What's the WWW or the way it's built have to do with it? What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default. Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to…

Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline. We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against ac…

">Putting a up a webserver that can be publically reached is not authorization to access it."

This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server.

">I really can't say this in any other way - just because you can do something doesn't mean you are allowed to"

You are allowed to do everything that is not expressly forbidden by law. Accessing a public web server is not forbidden by law unless the owner takes steps to prevent you from accessing it or at least clearly signals that intention. Terms of service do not constitute an implied contract, so you are not required to read the TOS before accessing a public page.

">We already have an explicit signal called robots.txt"

Exactly. That is part of what I meant when I said that any restrictions had to be implemented on top of the default, which is that everyone who can is allowed to access anything on a public webserver.

[Edit] But initially, this wasn't a thread about legality but about ethics. I think there are unethical reasons to scrape and there are unethical reasons to block scrapers. We simply need to know more about the purpose of any scraping before making a judgement.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#94

Post by jumperjake »

chinathrow wrote: post #85
jumperjake wrote: post #47

Earlier quoted context omitted.

You're confusing redistribution with scraping. Scraping publicly-accessible content is legal. Redistributing it is not (at least not in the US). Google scraps webpages as a core competency.

Google also adheres to the robots.txt standard. Most of the scrapers I block don't.

Not correct. Google will completely ignore the rules in robots.txt if it deems it acceptable. I think there's a link to this somewhere in this comment page.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#95

Post by kuschku »

chinathrow wrote: post #86
kuschku wrote: post #7

Earlier quoted context omitted.

Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.

Why? My site, my robots.txt - if the scrapers do not obey, then I'll block them.

No. Your site is public. If it is available via HTTP without authorization, I can scrape it (but not republish). There have been in many countries (except the US) court decisions about this.

There are ways to force authorization, if you don’t use them, your fault.

If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public.

And if I want to store the site offline for later reading, or read it now in a browser, is nothing that you have to decide.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#96

Post by chinathrow »

jumperjake wrote: post #94
chinathrow wrote: post #85

Earlier quoted context omitted.

Google also adheres to the robots.txt standard. Most of the scrapers I block don't.

Not correct. Google will completely ignore the rules in robots.txt if it deems it acceptable. I think there's a link to this somewhere in this comment page.

They do not index the content but might add the URL, correct. You can have a meta noindex present and they won't index even the URL.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#97

Post by chinathrow »

kuschku wrote: post #95
chinathrow wrote: post #86

Earlier quoted context omitted.

Why? My site, my robots.txt - if the scrapers do not obey, then I'll block them.

No. Your site is public. If it is available via HTTP without authorization, I can scrape it (but not republish). There have been in many countries (except the US) court decisions about this. There are ways to force authorization, if you don’t use them, your fault. If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public.…

Well, copyright does prohibit redistributing the scraped content. And that is what all 3rd party content scraping services essentially do. That violates copyright law in quite a few countries. If you scrape with your own tools, own servers and use it only within your company, I would think it's a different legal matter.

No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreciate that.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#98

Post by kuschku »

chinathrow wrote: post #97
kuschku wrote: post #95

Earlier quoted context omitted.

No. Your site is public. If it is available via HTTP without authorization, I can scrape it (but not republish). There have been in many countries (except the US) court decisions about this. There are ways to force authorization, if you don’t use them, your fault. If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public.…

Well, copyright does prohibit redistributing the scraped content. And that is what all 3rd party content scraping services essentially do. That violates copyright law in quite a few countries. If you scrape with your own tools, own servers and use it only within your company, I would think it's a different legal matter. No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreci…

It is public because the HTTP standard defines it as public, and, if you claim to use this standard, its definitions apply to you (if you claim to support a standard, you have to accept that its definitions will be used in court against you)

HTTP directly says you should use authorization to make stuff non-public.

And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it freely.

Otherwise services like Opera Turbo – which scrapes a website, removes cruft, compresses data, and sends it to your phone – would also break copyright law, and there is a nice exception for this case, under US law, it’s covered under fair use.

(Unless the scraping service makes your content accessible publicly – and even then, fair use applies usually, as in the case of pure archival like archive.org or archive.is)

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#99

Post by kuschku »

troymc wrote: post #10
kuschku wrote: post #7

Earlier quoted context omitted.

Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.

There have been lawsuits where the company running the site-being-scraped sued the scraper and won. So while it's technically possible, there can be legal issues. See https://en.wikipedia.org/wiki/Web_scraping#Legal_issues

If I republish, yes.

If I just download a site – for local use, or similar cases – it’s not an issue.

And if I provide a scraper where the user customly has to enter the URL, it’s also technically just a very advanced webproxy, and also okay.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#100

Post by troymc »

kuschku wrote: post #99
troymc wrote: post #10

Earlier quoted context omitted.

There have been lawsuits where the company running the site-being-scraped sued the scraper and won. So while it's technically possible, there can be legal issues. See https://en.wikipedia.org/wiki/Web_scraping#Legal_issues

If I republish, yes. If I just download a site – for local use, or similar cases – it’s not an issue. And if I provide a scraper where the user customly has to enter the URL, it’s also technically just a very advanced webproxy, and also okay.

That is incorrect. Even if you don't republish the data, the scraping alone can be a violation of the site's Terms of Service, and you can be sued (and lose). See the Wikipedia article I linked above.

Think about it from the point of view of the site being scraped. Suddenly your webserver is being slammed with traffic that's way beyond what any human would cause. It's costing you bandwidth and compute resources. You have every right to say no, scraping is not allowed (in your Terms of Service). It's then up to a court to decide if those Terms of Service are legally binding. Some courts have decided that yes, they are.

Post reply on HN