Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudproxy – hide your scrapers IP behind the cloud

github.com

Post reply on HN
Print view View original

91–100 of 116 posts

Re: Cloudproxy – hide your scrapers IP behind the cloud

#91

Post by ClumsyPilot »

dstick wrote: post #85
skummetmaelk wrote: post #77

Earlier quoted context omitted.

It's more like you going to the public library to borrow every single book they have. Thereby forcing the library staff to handle all your requests. You would probably not expect to be allowed to do this in real life.

Good point! There is an increased pressure on a public service. Hadn’t thought about that parameter.

Except its aprivate service.

Stop trying to lawyer this through analigies people, think about the actual situation at hand, instead of drawing broken parallels to some hypothetical

Re: Cloudproxy – hide your scrapers IP behind the cloud

#92

Post by kevin_thibedeau »

TheFreim wrote: post #75
deanclatworthy wrote: post #33

Earlier quoted context omitted.

It doesn't really matter what's ethical and not, or what the site's wish - what matters is what the law says [1]. I don't want my neighbours smoking on the balcony below me, as I can smell their smoke but the law doesn't allow my building to ban it without consensus. Alas... [1] https://techcrunch.com/2021/06/14/supreme-court-revives-link...

I care since I try to be ethical in my actions.

Ethics requires reciprocity. These same sites care nothing about being a vector for JS malware.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#93

Post by moehm »

mkl95 wrote: post #63
temp8964 wrote: post #45

Earlier quoted context omitted.

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month. The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with…

> free captcha solver

I haven't looked into this space, but what is a free captcha solver? And what is the drawback? Wouldn't this defeat the purpose of a captcha?

Re: Cloudproxy – hide your scrapers IP behind the cloud

#94

Post by judge2020 »

wu_187 wrote: post #55
holstvoogd wrote: post #18

Earlier quoted context omitted.

As someone who has to deal with a lot of bots, bot networks and other weird scraper apps people use: The biggest issue is that most of these tools are not behaved very well. This tool is clearly designed to circumvent protections, rate limits mostly, against scraping that might be essential to keep things running. They follow links that are explicitly marked as do not follow, they do not even try to limit their rate,…

Exactly. We'd get customers thats sites would drown in bot traffic because they pulled the same shit, changing user agents, different ips, etc. I had to build custom mod security rules to block the patterns these boys would pull. What's funny is that the bots would have a site where you could control the crawl rates but it's just a placebo. They would crawl even if you requested them to stop. The issue with bots host…

Note that you can report spam to AWS https://support.aws.amazon.com/#/contacts/report-abuse

Re: Cloudproxy – hide your scrapers IP behind the cloud

#95

Post by blantonl »

arsome wrote: post #43
blantonl wrote: post #39

Earlier quoted context omitted.

no business is under any obligation to provide you access to their data and services for you to build a business on top of. If you feel that the owner of data or services maintains an illegal monopoly over that data, work to lobby government to resolve that issue. It's not your individual place to unilaterally decide that a business owes you anything other than what is described in their terms and conditions of use.…

> no business is under any obligation to provide you access to their data and services for you to build a business on top of. I mean, they're the ones who gave it to me, I'm just using it. Not my problem if they don't like it. I never saw any terms and conditions when my browser loaded the page, no reason I'm going to go looking for them before I scrape it. I have just as much right to access it as my web browser doe…

I'm not saying they owe me anything, they're offering it to me, I'm just taking it.

No you are not. If you are deploying proxy containered services on cloud computing hosts orchestrated by yourself to circumvent mechanisms to prevent such behavior you are delibertly and unethically acting in a manner directly not intended by the owner of such data.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#96

Post by judge2020 »

deanclatworthy wrote: post #33
TheFreim wrote: post #15

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

It doesn't really matter what's ethical and not, or what the site's wish - what matters is what the law says [1]. I don't want my neighbours smoking on the balcony below me, as I can smell their smoke but the law doesn't allow my building to ban it without consensus. Alas... [1] https://techcrunch.com/2021/06/14/supreme-court-revives-link...

> I don't want my neighbours smoking on the balcony below me, as I can smell their smoke but the law doesn't allow my building to ban it without consensus.

That's simply because you're in multi-family housing, it would be a different story if the neighbors smoked and put up a fan to blow the smoke into your yard/window/etc, and that's probably a more apt comparison to bots that literally fork bomb themselves to crawl your site at 500 requests per second.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#97

Post by judge2020 »

raxi wrote: post #86

When you order/cancel DigitalOcean instances, the IPs of new instances will be reused from recently cancelled. So, you will not get many IPs this way.

I wonder if there's some internal APIs the cloud providers share to let them know when customers release ownership of an IP so that its reputation can be reset.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#98

Post by cortesoft »

deanclatworthy wrote: post #33
TheFreim wrote: post #15

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

It doesn't really matter what's ethical and not, or what the site's wish - what matters is what the law says [1]. I don't want my neighbours smoking on the balcony below me, as I can smell their smoke but the law doesn't allow my building to ban it without consensus. Alas... [1] https://techcrunch.com/2021/06/14/supreme-court-revives-link...

Why can't we talk about ethics? There are a lot of things in this world that are legal, yet I choose not to do them because they are not ethical in my opinion.

We are allowed to talk about ethics separate from the law.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#99

Post by thespacecowboy »

TheFreim wrote: post #15

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

I think it depends on the industry. I've worked at a few ecommerce companies and they all either bought data collected with scrapers or had a scraper team. They also paid for bot protection on their website to stop other companies from scraping their data. I have no problem with this, as ultimately consumers get more competitive and usually lower prices. That is assuming that they are scraping a reasonable rate and not sending 500 requests a second.

What does concern me is the other uses that scraping tools have. For example, what's to stop me from writing a bot specifically to fuck with a competitor's analytics and a/b testing?

Re: Cloudproxy – hide your scrapers IP behind the cloud

#100

Post by arsome »

blantonl wrote: post #95
arsome wrote: post #43

Earlier quoted context omitted.

> no business is under any obligation to provide you access to their data and services for you to build a business on top of. I mean, they're the ones who gave it to me, I'm just using it. Not my problem if they don't like it. I never saw any terms and conditions when my browser loaded the page, no reason I'm going to go looking for them before I scrape it. I have just as much right to access it as my web browser doe…

I'm not saying they owe me anything, they're offering it to me, I'm just taking it. No you are not. If you are deploying proxy containered services on cloud computing hosts orchestrated by yourself to circumvent mechanisms to prevent such behavior you are delibertly and unethically acting in a manner directly not intended by the owner of such data.

How so? They're still offering it to me, just on a different IP. I fail to see how that becomes unethical. I'm accessing information they publish to the public at large.
Post reply on HN