How to scrape anything on the web and not get caught • Hacker News Board

Re: How to scrape anything on the web and not get caught

#51

Post by robsun » Sun, Apr 22, 2018, 4:38 PM UTC

CapacitorSet wrote: post #34

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.

Same thoughts. Sadly we live in world when it is good enough to get recruiters' attention. Some time ago there was a post on HN about some bullshit like "everyone should have a blog".

Do you know any professionals with exceptional experience that is being shared on their blogs? For example, if someone is interested in .NET I can recommend this one: https://www.wiktorzychla.com

Re: How to scrape anything on the web and not get caught

#52

Post by megous » Sun, Apr 22, 2018, 5:31 PM UTC

salawat wrote: post #33

your-nanny wrote: post #30

What sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?

I'm kind of interested in an answer to this as well. I know the typical "travel site" or "comparison shop" use case. There is also the "darn it, I want this" use case. However, automated, periodic web scraping that mutates (I.e. ticket or reservation grabbing bots) has always felt a bit squicky to me in the same way DNS squatting does.

Website you like that invites discussion has a very shitty comment system, that doesn't notify you of the responses to your comments.

Or perhaps you want to monitor mentions of your name, and join the discussion.

Or you want to not lose your past thoughts, because discussions were deep and some may be interesting to re-read in the future.

Or a website is known to allow users delete their comments, or website itself bans users and hides all their content from others for flimsy reasons like "ISIS" in the title, no matter whether it's pro/against/neutral/irrelevant.

Or you want to organize the information differently than the site does.

Or the website is bloated and slow as hell, and you want to use it over gprs, so you create a lightweight/fast/better organized/filtered mirror.

Or the website doesn't have search or it is crap/slooow, and is not indexed in google, like some IRC logs out there.

Or you want to have content available offline.

...

Re: How to scrape anything on the web and not get caught

#53

Post by Bromskloss » Sun, Apr 22, 2018, 6:17 PM UTC

> proxy-lists getProxies --sources-white-list="gatherproxy,sockslist"

Is there a reason for using only gatherproxy and sockslist? There are more lists [0] available.

[0] https://github.com/chill117/proxy-lists/blob/4bb8064703b09ee...

Re: How to scrape anything on the web and not get caught

#54

Post by Bromskloss » Sun, Apr 22, 2018, 6:18 PM UTC

ospider wrote: post #45

CapacitorSet wrote: post #34

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.

I do web crawling for a living, the method mentioned in the article does not work for most sites

Do most sites have scraping detection at all? Are they even opposed to scraping?

Re: How to scrape anything on the web and not get caught

#55

Post by kapauldo » Sun, Apr 22, 2018, 8:34 PM UTC

ospider wrote: post #45

CapacitorSet wrote: post #34

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.

I do web crawling for a living, the method mentioned in the article does not work for most sites

What does work?

Re: How to scrape anything on the web and not get caught

#56

Post by codefined » Sun, Apr 22, 2018, 8:42 PM UTC

Bromskloss wrote: post #54

ospider wrote: post #45

Earlier quoted context omitted.

I do web crawling for a living, the method mentioned in the article does not work for most sites

Do most sites have scraping detection at all? Are they even opposed to scraping?

On a website I'd written we had pseudo-randomly generated URLs to show dynamic content (it was a game, the URL contained parameters). On each page we had this little widget that included five random configurations people might like to try.

A few times our website went down due to the load going >30, eventually I discovered Google was doing something funky, adding the dynamic domains to the "robot.txt" file fixed the issue. Then some other search engines / scrapers seemed to run into the same issue and started requesting hundreds of thousands of URLs per day (these pages were dynamically generated and took a moderate amount of compute power).

We eventually did have to implement basic anti-scraper rules because it was degrading the user experience.

Re: How to scrape anything on the web and not get caught

#57

Post by lazycouchpotato » Mon, Apr 23, 2018, 7:51 AM UTC

reallymental wrote: post #37

lazycouchpotato wrote: post #35

Earlier quoted context omitted.

Statistics I'd say is one of the useful cases of scraping. Back in 2013, a guy scraped the results of about 150,000 students giving their 10th grade finals of a particular examination board in India. He showed that not only was there no privacy of student's marks because the roll numbers were all linearly incremented, but there was also mass-scale manipulation of marks going on. The concept is simple but it's a very…

Facinating read, I cannot find out more about what happened to him after he was accused of 'hacking' the govt systems, do you have any more sources that sheds some light on that?

I tweeted out to him and he responded.

https://twitter.com/debarghya_das/status/988178000914022400