Scrape like the big boys

Post by [deleted] » Fri, Nov 05, 2021, 8:22 PM UTC

scarygliders wrote: post #31

ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Right with you there. I had a particularly bad time not so long ago, when a customer's site - a shop - was brought to its knees because someone, probably a competitor, hired some scraper-company of some sort to scrape every product and price. The scraper would systematically go through every single product page. And by scraper, I mean - 100's of them. All at the same time, using the old trick of 1 scraper requesting…

Post by matheusmoreira » Fri, Nov 05, 2021, 8:34 PM UTC

rootusrootus wrote: post #65

biosed wrote: post #37

I used to lead Sys Eng for a FTSE 100 company. Our data was valuable but only for a short amount of time. We were constantly scraped which cost us in hosting etc. We even seen competitors use our figures (good ones used it to offset their prices, bad ones just used it straight). As the article suggest, we couldn't block mobile operator IPs, some had over 100k customers behind them. Forcing the users to login did litt…

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

Post by 323 » Fri, Nov 05, 2021, 8:41 PM UTC

hall0ween wrote: post #103

Basic question, how does one profit from scraping data and what kinda data? Taking a stab at answering it: you scrape the data and build a business around selling it. Stock prices? But that's boring, plus how many others are doing it? I bet a lot.

Post by funnyflamigo » Fri, Nov 05, 2021, 9:06 PM UTC

jjeaff wrote: post #98

funnyflamigo wrote: post #88

Earlier quoted context omitted.

> Seriously, I hate scrapers. I hate the people who make scrapers. I hate their lack of ethics. Fuck those guys. Wait till you find out what half of Google's business is based on (spoiler - scraping). I really don't think scraping itself is an issue 90% of the time. It's the behavior of the out of control scrapers that are the problem. A well behaved scraper should barely be noticeable, if at all.

At least google's scraping does result in your website being discoverable by users. So you get something out of it. That's not to say that sometimes Google is missing or stealing data they scrape. But at least there is some benefit. Many other scrapers are merely taking the data to compete.

Post by bsder » Fri, Nov 05, 2021, 9:20 PM UTC

incolumitas wrote: post #48

IceWreck wrote: post #6

The author says proxys are expensive and then proceeds to spend a shitton of money buying all that hardware.

4G proxies are just soo much better than so called "residential" or straight datacenter proxies. It makes sense to create your own 4G proxy farm if you conduct business in that area. With only 10 dongles and 10 dataplans, you can have a lot of IP addresses that are extremely hard to block. It's an one time investment, paying proxy providers is a fixed cost.