Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to scrape anything on the web and not get caught

tinyendian.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: How to scrape anything on the web and not get caught

#41

Post by nkozyra »

CapacitorSet wrote: post #34

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.

In addition, it does nothing to mitigate even remotely significant scrape detection. Doesn't talk about request fingerprints, patterns, user agents and headers, multi region access, etc.

This is a hobbyists' guide to scraping under the radar. Fine at that scale but quite incomplete for anything remotely mature or wide reaching.

Re: How to scrape anything on the web and not get caught

#42

Post by time0ut »

your-nanny wrote: post #30

What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?

I've used it to convert publicly available data to a more useful format. A local government agency made the data freely available as a series of thousands of HTML pages, but not in any other form. I didn't need to use any evasive tactics, but it was still wholesale repeated scraping.

Re: How to scrape anything on the web and not get caught

#43

Post by forcer »

ikeboy wrote: post #8
dewey wrote: post #6

Earlier quoted context omitted.

You could use https://oxylabs.io/ or buy some regular VPN accounts and build a http proxy wrapper around that. Extremely cheap and works well. There’s a lot of existing projects on Github for that too.

oxylabs.io seems more expensive than luminati, minimum of $178/month. Not clear if they charge for bandwidth. Problem with VPN is it's shared and hard to get a lot of IPs, any specific ones I could get say 100 dedicated US IPs for a reasonable price?

Depending on what you scrape we can help. we have proxies in 170+ countries, please get in touch at www.speedchecker.xyz , the pricing is cheaper than what you say above

Re: How to scrape anything on the web and not get caught

#45

Post by ospider »

CapacitorSet wrote: post #34

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.

I do web crawling for a living, the method mentioned in the article does not work for most sites

Re: How to scrape anything on the web and not get caught

#46

Post by alehul »

ospider wrote: post #45
CapacitorSet wrote: post #34

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.

I do web crawling for a living, the method mentioned in the article does not work for most sites

I've been working on a project that requires scraping from a large number of sites. Do you have any recommendations on better resources?

Re: How to scrape anything on the web and not get caught

#47

Post by nojvek »

amelius wrote: post #17

Could this be a solution?: run a website, and let your visitors do the crawling.

You download some popular iOS game. But in the background it’s fetching a List of urls, crawling it and sending results back to mothership.

How would an average joe even know?

Re: How to scrape anything on the web and not get caught

#48

Post by nojvek »

amelius wrote: post #24
Theodores wrote: post #9

I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that? This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond lookin…

Captchas are one (drastic) option.

Any image based captcha where one needs to identify words from an image can be easily broken by algorithms now.

Google would be the leader with re-captcha, but as a human, I fail a whole number of them. They are a very annoying experience to your users.

Re: How to scrape anything on the web and not get caught

#50

Post by robsun »

Some time ago I was looking for an apartment to buy. Sites in my country are bloated and terribly slow. Checking several offers took minutes. Moreover I live in a city where good offers are sold the same day they are published.

I decided to run scrapper to fetch all the data about available apartments in my city. Thanks to that I was able to browse offer at speed of tinder. It took me a few hours to write all the stuff, it saved me probably weeks.

To avoid getting caught I decided to setup TOR on my raspberry pi and use it as a proxy. It was extremely easy and reliable. Sites were so slow I didn't notice significant performance drop. I didn't care about changing proxies because TOR made it for me.

Except that it is good idea to change User-Agents and add some random delays between calls. Luckily for this case it was enough.

Post reply on HN