Scrape like the big boys
incolumitas.com
Scrape like the big boys
1–10 of 196 posts
Re: Scrape like the big boys
#2Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece.
For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely accessible as possible.
But after this week, where some amateur did real damage to us with a ham-fisted attempt to scrape too much too quickly, we’re forced to degrade the experience for ALL users by introducing captchas and other techniques we’d really rather not.
Re: Scrape like the big boys
#3Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?
Re: Scrape like the big boys
#4Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?
Like everyone and their brother has a web spider. And some of them are VERY badly designed. We block them when they use too many resources, although we'd rather just let them be.
Can't speak for the op but we have APIs and move the ones scraping and reselling our content to APIs. The majority are just a worthless suck on resources though.
Re: Scrape like the big boys
#5Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?
We do offer an API - the scrapers are trying to circumvent using that, presumably.
Re: Scrape like the big boys
#6The author says proxys are expensive and then proceeds to spend a shitton of money buying all that hardware.
Re: Scrape like the big boys
#7Earlier quoted context omitted.
Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?
We do offer an API - the scrapers are trying to circumvent using that, presumably.
What is your site may I ask?
Just curious about the difference in value from using your API and web scraping as there is a cost to web scraping as well.
Re: Scrape like the big boys
#8Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…
[deleted]
Re: Scrape like the big boys
#9Earlier quoted context omitted.
Considering the demand for your content, why haven’t you created and provided an API? Maybe you could monetize?
We do offer an API - the scrapers are trying to circumvent using that, presumably.
Is it not viable to put majority of your data behind a login and so the bots only get a very limited snapshot while legitimate users get it through a free login?
I’m asking this because I’m going through very similar situation and would love to see other opinions around this.
Re: Scrape like the big boys
#10The author says proxys are expensive and then proceeds to spend a shitton of money buying all that hardware.
That was not the authors main argument against proxies, that was just an additional point. You ignored the primary argument in your judgment.
>>Because I could not fully trust the other customers with whom I shared the proxy bandwidth. What if I share proxy servers with criminals that do more malicious stuff than the somewhat innocent SERP scraping?