Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: CloudScrape – Cloud-based web scraping platform

cloudscrape.com

Post reply on HN
Print view View original

71–80 of 109 posts

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#71

Post by cookiecaper »

chdir wrote: post #62
yxdfasdjkljasdf wrote: post #27

Earlier quoted context omitted.

That's just wrong. You haven't explained how is it wrong and why. None of those thing are "wrong" by itself. It is the malicious use, of any tool, that is unethical. Is there a website where we can blacklist IP addresses of such violators ? What exactly do you think is being violated here?

Websites use captchas & IP based limits to prevent abuse of their resources & make it harder for copycats to mirror their data. There are often cases where copycats outrank original content in search rankings. (see this example : https://news.ycombinator.com/item?id=10103545 ). If I were a content owner/producer and I see automated scraping from IP addresses owned by Cloudscrape that violate the ToS, I would sadly tr…

There's already a tool to stop "copycats". It's called copyright (and for inventions, patents). You can and should use that to enforce your rights to your IP. It's not too hard to start issuing DMCA requests, and it's not even that expensive to have a lawyer do it if you're making money. It doesn't matter whether the illegal copy is obtained by a bot or a human.

While I agree that captchas and IP blocks can be employed by target sites, I don't agree that it should be illegal to circumvent them. I also don't agree that it's necessarily unethical (though in some cases, it may be). If you have public information posted on the public web, I don't think you have the right to mandate that it only be accessed by certain tools. You should plan and expect that it will be accessed by every tool capable of doing so.

If something is disrupting your business by "clogging the tubes" or whatever, that's another thing, and they can be held liable for that. But it doesn't matter that they clogged the tubes with one type of program or another; what matters is that the tubes were clogged by their actions, and that's the part that should be focused on in the subsequent legal proceedings. The specific tool or tools used to clog the tubes is at most a tangential curiosity. We don't want to make certain programs illegal.

Maybe we need a new amendment with "the right to bear code". We do not want to get down a rabbit hole where certain programs are legal and certain programs are not (at least not anymore than we already are with the DMCA et al). Down with code control!

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#73

Post by suneilp »

henrik1409 wrote: post #61

It's great the see discussions going on here - would like to tie a few comments to the questions of ethical aspects of web scraping: As some has pointed out scraping is not exactly a new thing and a lot of the biggest sites out there are built on the basis of web scraping or crawling. We provide a tool and expect you use that tool while abiding the law - and if not we will of course shut your account down immediately…

Saying 50 concurrent agents is no big deal for most websites is kinda flippant. It all adds up and not every popular website can handle loads of traffic. By using a headless browser to gain a proper version of a webpage, you also strain a webserver with serving static files costing them bandwidth as well.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#74

Post by bduerst »

jumperjake wrote: post #46
bduerst wrote: post #41

Earlier quoted context omitted.

You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used. That doesn't mean that it doesn't happen.

IANAL, but copyright governs redistribution of content not consumption (That's what pirates get busted for). I aslo recall that there was a ruling that footer TOSs aren't enforceable unless the user actively and explicitly agrees to them. I agree with the GP in that public content is fair game. How do you thing google works?

Google technically respects robots txt and noindex metatags. OP is arguing the ethics of scraping, not if people are ignoring bot meta tags.

Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#75

Post by yxdfasdjkljasdf »

dsjoerg wrote: post #53
yxdfasdjkljasdf wrote: post #36

Earlier quoted context omitted.

That is not how HTTP works; your analogy is not correct. Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.

At a high enough frequency, scraping is indistinguishable from a DDoS attack. Do you believe DDoS attacks are OK? How do you draw the line?

There is a clear distinction in the two. You are presenting a straw-man argument.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#76

Post by yxdfasdjkljasdf »

manigandham wrote: post #49
yxdfasdjkljasdf wrote: post #36

Earlier quoted context omitted.

That is not how HTTP works; your analogy is not correct. Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.

Since there's no easy way to always reliably identify the requester, this gets complicated. Most scrapers - including this one - advertise how they use multiple servers/locations/ips/etc to get around this.

I fail to see a problem you are trying to present.

Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#77

Post by yxdfasdjkljasdf »

chdir wrote: post #62
yxdfasdjkljasdf wrote: post #27

Earlier quoted context omitted.

That's just wrong. You haven't explained how is it wrong and why. None of those thing are "wrong" by itself. It is the malicious use, of any tool, that is unethical. Is there a website where we can blacklist IP addresses of such violators ? What exactly do you think is being violated here?

Websites use captchas & IP based limits to prevent abuse of their resources & make it harder for copycats to mirror their data. There are often cases where copycats outrank original content in search rankings. (see this example : https://news.ycombinator.com/item?id=10103545 ). If I were a content owner/producer and I see automated scraping from IP addresses owned by Cloudscrape that violate the ToS, I would sadly tr…

(Each line responds to a paragraph in order)

Appeal to fear.

No comment.

No being able to determine a right cause doesn't prove a wrong.

Loaded question.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#78

Post by dsjoerg »

yxdfasdjkljasdf wrote: post #75
dsjoerg wrote: post #53

Earlier quoted context omitted.

At a high enough frequency, scraping is indistinguishable from a DDoS attack. Do you believe DDoS attacks are OK? How do you draw the line?

There is a clear distinction in the two. You are presenting a straw-man argument.

You haven't quite laid out your argument so I have to guess what it is.

When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK.

So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.

I find this an interesting question, because while I would love for protcols to also define ethics, I feel that would be scope creep for the poor protocol designers. There's a wide variety of conduct and ethics questions that a protocol cannot address.

Where I myself draw the line is at protocol behavior intentionally designed to obscure my intentions. For example, sending my requests from a wide variety of IP addresses is behavior that is specifically designed to obscure where I'm coming from; my only intent in doing so would be to circumvent the intent of the serving machine from providing lots of content to a single requestor. At that point I'm engaging in deceptive behavior; I've crossed an ethical line.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#79

Post by yxdfasdjkljasdf »

dsjoerg wrote: post #78
yxdfasdjkljasdf wrote: post #75

Earlier quoted context omitted.

There is a clear distinction in the two. You are presenting a straw-man argument.

You haven't quite laid out your argument so I have to guess what it is. When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK. So I'm left wondering what your argument actually is for why unwelcome scraping is ethically O…

When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK.

That wasn't a response made to your comment, and you are mixing two different arguments there. You guess in not correct.

So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.

I never even suggested such an argument.

The behavior you described in the last paragraph is only deceptive from the eyes of an information and privacy surveillant state actor. Anonymity is not unethical, it is a human right.

Re: Show HN: CloudScrape – Cloud-based web scraping platform

#80

Post by wrath »

dchuk wrote: post #66
wrath wrote: post #65

Earlier quoted context omitted.

Sorry, my bad... it's shader.io

Awesome, thank you. They do look interesting, but their pricing confuses me. Are they just selling individual proxies? Or does $1.80 get you single exit nodes or threads basically?

As far as I know, shader.io sells individual proxies. Luminati and Nohodo are exit nodes priced by the amount of bandwidth that you use.
Post reply on HN