Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CloudScrape – Cloud-based web scraping platform

cloudscrape.com

Post reply on HN
Print view View original

41–50 of 109 posts

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#41

Post by bduerst »

angry-hacker wrote: post #30
danielmiessler wrote: post #15

Probably a super dumb question, but isn't this fairly unethical? The "automatic IP rotation" feature isn't there for no reason.

As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!

You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used.

That doesn't mean that it doesn't happen.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#42

Post by angry-hacker »

RealGeek wrote: post #37
chdir wrote: post #20

- Auto-resolve CAPTCHA’s - Automatic IP rotation That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?

Google's business is to crawl all websites, but they IP block crawlers from their website. IP rotation is a solution to crawl such hypocrites.

I agree. It's ironic they protect their page so aggressively against scraping, being the biggest scraper in the universe at the same time.

Often they ignore robots.txt also, they still continue scraping, sometimes they even publish things in their SERPS.

From Google's FAQ:

However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#46

Post by jumperjake »

bduerst wrote: post #41
angry-hacker wrote: post #30

Earlier quoted context omitted.

As much as unethical as using adblock or disabling javascript. If you don't want your content to be scraped, don't put it online!

You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used. That doesn't mean that it doesn't happen.

IANAL, but copyright governs redistribution of content not consumption (That's what pirates get busted for). I aslo recall that there was a ruling that footer TOSs aren't enforceable unless the user actively and explicitly agrees to them.

I agree with the GP in that public content is fair game. How do you thing google works?

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#47

Post by jumperjake »

brwnll wrote: post #23
kuschku wrote: post #7

Earlier quoted context omitted.

Why? People will just scrape your data anyway. What you put online is public . Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.

Well that sure is an arrogant reply, and not even close to true. In the US everything published (online or not) is copyrighted. Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.

You're confusing redistribution with scraping. Scraping publicly-accessible content is legal. Redistributing it is not (at least not in the US).

Google scraps webpages as a core competency.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#48

Post by snehesht »

yxdfasdjkljasdf wrote: post #27
chdir wrote: post #20

- Auto-resolve CAPTCHA’s - Automatic IP rotation That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?

That's just wrong. You haven't explained how is it wrong and why. None of those thing are "wrong" by itself. It is the malicious use, of any tool, that is unethical. Is there a website where we can blacklist IP addresses of such violators ? What exactly do you think is being violated here?

No offense but ...

First thing, it's just a suggestion. Don't use duplicate account to reply to the comments. Do it from your original account.

I think what the user "chdir" means by wrong is that you're not honoring "robots.txt" and how do you account for ethical scraping ( eg. running 50 concurrent connection on a single website and overloading the website, technically its DoS attack)

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#49

Post by manigandham »

yxdfasdjkljasdf wrote: post #36
manigandham wrote: post #33

Earlier quoted context omitted.

That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.

That is not how HTTP works; your analogy is not correct. Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.

Since there's no easy way to always reliably identify the requester, this gets complicated.

Most scrapers - including this one - advertise how they use multiple servers/locations/ips/etc to get around this.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#50

Post by manigandham »

angry-hacker wrote: post #40
manigandham wrote: post #33

Earlier quoted context omitted.

That's not a good argument. Do you ever leave your stuff lying around? I guess we can just take it then right? Just because you have access to something doesn't give you permission to access or access it in any manner possible.

You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple. Web is for people from people, not solely for company(s) financial interests.

What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular.

Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.

Post reply on HN