Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudproxy – hide your scrapers IP behind the cloud

github.com

Post reply on HN
Print view View original

111–116 of 116 posts

Re: Cloudproxy – hide your scrapers IP behind the cloud

#111

Post by judge2020 »

Scoundreller wrote: post #105
judge2020 wrote: post #64

Earlier quoted context omitted.

This probably means incompetency on the CDN’s part - Cloudflare has a detection rule for fake googlebots and it checks by doing reverse DNS to see if it’s really Google. Doing this trick is more likely to get your IP marked for spam, at least if you crawl CF sites with it. https://developers.google.com/search/docs/advanced/crawling/...

How does Google get around cloaking? Don't they need to visit a site from time to time without coming in as Googlebot to make sure they're getting presented the same page as Googlebot? Or is that always done via manual review?

They might pretend to be on other networks they own when they do things like ad/policy reviews (eg. the Google Fi or Google Fiber ASNs[0]), but I don't know of anyone confirming that this happens.

0: https://bgp.he.net/AS16591

Re: Cloudproxy – hide your scrapers IP behind the cloud

#112

Post by mkl95 »

moehm wrote: post #93
mkl95 wrote: post #63

Earlier quoted context omitted.

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month. The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with…

> free captcha solver I haven't looked into this space, but what is a free captcha solver? And what is the drawback? Wouldn't this defeat the purpose of a captcha?

I'm using a ML-based captcha solver that is free software available on Github. So far it has solved 100% of the Amazon captchas it has encountered.

The reason Amazon have a reputation of being good at blocking IPs is that their responses are (purposefully?) obscure. The way it works, it filters out script kiddies and lets engineers through, which probably are a small minority of the people scraping Amazon.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#113

Post by cinntaile »

pseudo0 wrote: post #109
cinntaile wrote: post #23

Earlier quoted context omitted.

You have autobuying bots (sneakers or GPUs or other limited edition stuff) that need to do tricks like this. I wouldn't necessarily call them unethical.

In my opinion it's unethical if the website explicitly states in their terms of service that bot purchasing is not allowed. As far as I know, that is pretty common for websites that sell frequently scalped products, and near-ubiquitous among those that implement technical countermeasures. I'd expect cybercriminals and fraudsters also find a pool of disposable residential IPs to be very useful.

A Terms of Service is not a proxy for morals though. Companies will put whatever is convenient for them in there, I'm sure you can come up with several examples that you wouldn't consider ethical.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#114

Post by fomine3 »

tyingq wrote: post #30
EveYoung wrote: post #26

Earlier quoted context omitted.

If I remember correctly, Cloudflare WARP forwards your client IP in the header. So it probably won't be useful for scraping.

That's correct. More detail here: https://news.ycombinator.com/item?id=21070828

Really? I read that mmproxy can't be applied to general 3rd party websites.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#115

Post by claffin »

Hi all - I'm the creator of CloudProxy and only just came across this. Appreciate all the comments and stars (over 600 in less than two days). All the points shared are valid, you won't get the same effectiveness as residential IPs and may face issues with the proxies being blocked. That being said this solution, if you find they're not being blocked, is a lot cheaper and very quick. I've used it to scrape some major websites extensively without issue. I created it for my own use but then decided to open source it. Just hope you guys find it useful!

Re: Cloudproxy – hide your scrapers IP behind the cloud

#116

Post by jimmySixDOF »

cortesoft wrote: post #79
jimmySixDOF wrote: post #25

In a related note the US Supreme Court just 10 days ago vacated a previous ruling against LinkedIn blocking a Scraper service. So the issue is back to the 9th circuit for a new determination that might, if reversed, change the landscape for Cloudflare and anyone else these operators try to sit behind. https://www.scotusblog.com/case-files/cases/linkedin-corp-v-...

I don’t think the Supreme Court ruled they can’t block them with technical means, just that they can’t sue them if they get around the block.

You are basically correct although it seems to hinge on the definition of an authorized use(r) and also they only vacated the appeals court decision so Microsoft will continue with the same case now in the lower court.

There is a good discussion of the more nuanced situation here:

https://news.ycombinator.com/item?id=27510103

Post reply on HN