Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudproxy – hide your scrapers IP behind the cloud

github.com

Post reply on HN
Print view View original

101–110 of 116 posts

Re: Cloudproxy – hide your scrapers IP behind the cloud

#101

Post by DoctorOW »

moehm wrote: post #93
mkl95 wrote: post #63

Earlier quoted context omitted.

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month. The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with…

> free captcha solver I haven't looked into this space, but what is a free captcha solver? And what is the drawback? Wouldn't this defeat the purpose of a captcha?

I also haven't looked into this space outside of a quick DDG search after reading your comment. It looks like the big hole is that Amazon rolled their own captcha a while ago and haven't kept up with what automation can do now.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#102

Post by [deleted] »

TheFreim wrote: post #13
tushar-r wrote: post #10

This is relatively easy to block, given that these companies publish their IP ranges. Similar to the Tor block, blocking "datacenter IP's" simply becomes the norm. This is why you have companies offering "resi's" or residential proxies to bypass such blocks for some years now.

I heard a while back, on HN 8 I think, that a lot of residential proxies are computers that have been infected with malware. How true is that?

[deleted]

Re: Cloudproxy – hide your scrapers IP behind the cloud

#103

Post by twothamendment »

holstvoogd wrote: post #18
TheFreim wrote: post #15

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

As someone who has to deal with a lot of bots, bot networks and other weird scraper apps people use: The biggest issue is that most of these tools are not behaved very well. This tool is clearly designed to circumvent protections, rate limits mostly, against scraping that might be essential to keep things running. They follow links that are explicitly marked as do not follow, they do not even try to limit their rate,…

"... because some bot is trying to scrape a calendar and is now looking for events in 1850 or whatever."

Been there, done that - at least on the side of fixing it. Anyone who implements a calendar, don't make pervious and next links that let someone travel time forever.

I've always wondered how much bit traffic costs us - but never actually tried to figure it out. It is a good portion of our traffic - even when we block a lot of it.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#104

Post by aaadult »

A similar approach I use is google cloud sdk + ssh socks5 proxy.

Create some preemptible instances on google cloud first, then connect them with commands like "gcloud compute ssh instances-name -- -D localhost:port".

And the last step is to connect scrapper to those proxy ports over localhost.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#105

Post by Scoundreller »

judge2020 wrote: post #64
Dunedan wrote: post #49

Earlier quoted context omitted.

For one pet project I had to crawl a rather popular site. While that worked in general I would frequently get internal server errors. Turns out that this was the response of their CDN when it detected bot-like behavior. That left me wondering how they prevent search engine crawlers from being detected as bots and getting throttled this way as well. Turns out they just check the user agent for that. As soon as I put "…

This probably means incompetency on the CDN’s part - Cloudflare has a detection rule for fake googlebots and it checks by doing reverse DNS to see if it’s really Google. Doing this trick is more likely to get your IP marked for spam, at least if you crawl CF sites with it. https://developers.google.com/search/docs/advanced/crawling/...

How does Google get around cloaking? Don't they need to visit a site from time to time without coming in as Googlebot to make sure they're getting presented the same page as Googlebot?

Or is that always done via manual review?

Re: Cloudproxy – hide your scrapers IP behind the cloud

#106

Post by Scoundreller »

iforgotpassword wrote: post #68
superasn wrote: post #27

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range. I've had some success in scraping lately with a similar project called FlareSolverr(1). It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP b…

There's a bot in an IRC channel I've been on for over a decade that announces the of any link being mentioned in the chan. It's becoming less and less useful as it's running on someone's vps, and a lot of sites behind cloudflare don't yield anything as they're returning the "checking your browser" page to the bot. Then there are pages that are pure javascript a d don't even deliver a title tag, and others try to show…

I guess it's time for a user-side script that sends the http request through their daily-driver browser to see what it is, but then they're getting their home computer to visit any and every link... Maybe only when it sees cloudflare DNS...

Re: Cloudproxy – hide your scrapers IP behind the cloud

#107

Post by Scoundreller »

oefrha wrote: post #24
robjan wrote: post #16

Earlier quoted context omitted.

There have been a number of Chrome extensions which monetize by selling your bandwidth to provide this kind of service. Not ethical but probably not a compromise.

Browser extensions, and VPNs. Notably, Luminati was allegedly selling HolaVPN users as exit nodes. https://www.trendmicro.com/vinfo/hk-en/security/news/cybercr...

Also the economic basis of a lot of "cell-phone farming".

Re: Cloudproxy – hide your scrapers IP behind the cloud

#108

Post by Scoundreller »

0xy wrote: post #28
TheFreim wrote: post #13

Earlier quoted context omitted.

I heard a while back, on HN 8 I think, that a lot of residential proxies are computers that have been infected with malware. How true is that?

A significant portion are grey market "ISPs" that purport to sell residential services but actually never do. They sell these "residential" IPs to Amazon, other ecommerce retailers and shady people for an extreme price. In the e-commerce world, scraping is necessary to stay in business. Amazon has armies of scrapers constantly monitoring their competitors, and in some cases automatically undercutting price updates.

Same on the consumer side. I like setting up alerts on camelcamelcamel and seeing price history (to make sure I'm not getting screwed) and to buy buy buy when prices hit a certain theshold.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#109

Post by pseudo0 »

cinntaile wrote: post #23
pseudo0 wrote: post #20

Earlier quoted context omitted.

I've heard of services that offer free proxy bandwidth in exchange for the user acting as a residential proxy node for other users and/or paying clients. It's generally marketed as a way for the user to avoid geoblocks and such. If this is clearly stated to the user upfront rather than buried behind half a dozen dark patterns and fine print, it seems like this business model could be conducted ethically. When it come…

You have autobuying bots (sneakers or GPUs or other limited edition stuff) that need to do tricks like this. I wouldn't necessarily call them unethical.

In my opinion it's unethical if the website explicitly states in their terms of service that bot purchasing is not allowed. As far as I know, that is pretty common for websites that sell frequently scalped products, and near-ubiquitous among those that implement technical countermeasures.

I'd expect cybercriminals and fraudsters also find a pool of disposable residential IPs to be very useful.

Re: Cloudproxy – hide your scrapers IP behind the cloud

#110

Post by Scoundreller »

abc03 wrote: post #54
samjmck wrote: post #17

Earlier quoted context omitted.

Are there even ethical or “official” ways to source residential IPs? I thought most of them were from botnet infected IoT devices.

Packetstream.io buys bandwidth from users and is cheap. I‘m a customer.

Pretty good margins: They pay users $0.10/gb and they sell it to you for $1.
Post reply on HN