Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudproxy – hide your scrapers IP behind the cloud

github.com

Post reply on HN
Print view View original

61–70 of 116 posts

Re: Cloudproxy – hide your scrapers IP behind the cloud

#61

Post by helsinkiandrew »

temp8964 wrote: post #45
superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

Yes, all their product data is available for affiliates: https://webservices.amazon.com/paapi5/documentation/

Re: Cloudproxy – hide your scrapers IP behind the cloud

#62

Post by dstick »

oefrha wrote: post #24
robjan wrote: post #16

Earlier quoted context omitted.

There have been a number of Chrome extensions which monetize by selling your bandwidth to provide this kind of service. Not ethical but probably not a compromise.

Browser extensions, and VPNs. Notably, Luminati was allegedly selling HolaVPN users as exit nodes. https://www.trendmicro.com/vinfo/hk-en/security/news/cybercr...

This is not alledgedly but very much true. I was in the proxy business for a few years and in talks with the Luminati people to white-label their product for a very specific type of proxy.

It’s such a weird field to be in. It’s not illegal by definition of law, but you’re definitely in shady territory, with most of the customers being of the “get rich quick” persuasion. Or at the very least trying to cut corners. One way or another, they were not playing by the rules ;-)

Re: Cloudproxy – hide your scrapers IP behind the cloud

#63

Post by mkl95 »

temp8964 wrote: post #45
superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month.

The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with some middleware is a trivial task.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#64

Post by judge2020 »

Dunedan wrote: post #49
superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

For one pet project I had to crawl a rather popular site. While that worked in general I would frequently get internal server errors. Turns out that this was the response of their CDN when it detected bot-like behavior. That left me wondering how they prevent search engine crawlers from being detected as bots and getting throttled this way as well. Turns out they just check the user agent for that. As soon as I put "…

This probably means incompetency on the CDN’s part - Cloudflare has a detection rule for fake googlebots and it checks by doing reverse DNS to see if it’s really Google. Doing this trick is more likely to get your IP marked for spam, at least if you crawl CF sites with it.

https://developers.google.com/search/docs/advanced/crawling/...

Re: Cloudproxy – hide your scrapers IP behind the cloud

#66

Post by temp8964 »

mkl95 wrote: post #63
temp8964 wrote: post #45

Earlier quoted context omitted.

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month. The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with…

But other comments say scraping Amazon is kind of complicated because they ban IPs? I am not sure if you have a seller / affiliate account and then use your home IP to do scaping, will that impact your seller / affiliate account ?

Re: Cloudproxy – hide your scrapers IP behind the cloud

#67

Post by temp8964 »

helsinkiandrew wrote: post #61
temp8964 wrote: post #45

Earlier quoted context omitted.

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

Yes, all their product data is available for affiliates: https://webservices.amazon.com/paapi5/documentation/

Also, the document says "Product Advertising API is free". I am confused.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#68

Post by iforgotpassword »

superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

There's a bot in an IRC channel I've been on for over a decade that announces the of any link being mentioned in the chan. It's becoming less and less useful as it's running on someone's vps, and a lot of sites behind cloudflare don't yield anything as they're returning the "checking your browser" page to the bot. Then there are pages that are pure javascript a d don't even deliver a title tag, and others try to show a GDPR banner or paywall and thus yield some generic title and not whatever article the link is supposed to show.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#69

Post by dstick »

meltedcapacitor wrote: post #59
nlh wrote: post #44

Earlier quoted context omitted.

I used to have a similar attitude - most typically applied to rules & laws, etc. “If you don’t like the rule, go run for office and get the law changed.” But then I realized that’s just a cop out. A lot of structures and systems are in place specifically to make it difficult for people to change things and specifically to maintain monopolies. Nobody’s going to lobby government to resolve this issue. That’s the point.…

In the analog age, imagine someone made a card index of some interesting data you can come in a shop to consult, for a fee. Would breaking into their premises to steal or copy the cards be on the same side of the line as the digital variant?

Scraping a website in your analog analogy would be a store where those index cards would be plastered to the shop’s windows, visible from the outside. The “scraper” would come by every day and manually copy (as in, write in their own notebook) what was on those index cards that are visible from the street.

So the question is: would you consider that stealing?

Re: Cloudproxy – hide your scrapers IP behind the cloud

#70

Post by mkl95 »

temp8964 wrote: post #66
mkl95 wrote: post #63

Earlier quoted context omitted.

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month. The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with…

But other comments say scraping Amazon is kind of complicated because they ban IPs? I am not sure if you have a seller / affiliate account and then use your home IP to do scaping, will that impact your seller / affiliate account ?

You shouldn't use the same IP to continuously scrape Amazon. Personally I use a $8/month rotating proxy service that gives me a new proxy list every hour (webshare.io if it piques your interest, I'm in no way associated with them).

Also, in my experience Amazon's "ban" comes down to solving a captcha on every request, so it's more like some mild throttling than a real ban.

Post reply on HN