Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrape like the big boys

incolumitas.com

Post reply on HN
Print view View original

81–90 of 196 posts

Re: Scrape like the big boys

#81

Post by mdoms »

scarygliders wrote: post #31
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Right with you there. I had a particularly bad time not so long ago, when a customer's site - a shop - was brought to its knees because someone, probably a competitor, hired some scraper-company of some sort to scrape every product and price. The scraper would systematically go through every single product page. And by scraper, I mean - 100's of them. All at the same time, using the old trick of 1 scraper requesting…

If your site is so poorly written it can't handle a few hundred computers trying to do something as simple as loading your product pages then sorry, but that's on you. The information is on the public web and scrapers are as entitled to access it as any web browser.

Re: Scrape like the big boys

#82

Post by RobSm »

paco3346 wrote: post #42
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

I'm right there with you. I'm the lead engineer for an automotive SaaS provider (with ~6000 customers and ~4 billion requests per month) and we recently started moving all our services to Cloudflare's WAF to take advantage of their bot protection. We were getting scrapes from botnets in the 100000+ per minute range that was affecting performance. We chose to switch to the JS challenge screen as it requires no human i…

Why do you think those bots were scraping your data in the first place?

Re: Scrape like the big boys

#83

Post by RobSm »

paco3346 wrote: post #62
EdwardDiego wrote: post #46

Earlier quoted context omitted.

What were they scraping, if I can ask? Was it targeted or just wget -r style?

It was a hybrid of low-effort vulnerability scanning and targeted inventory scraping. Many dealerships in the automotive space will pay gray-hat third parties to scrape and compile data on their competitors. The irony for us as a provider is that it's one of our customers (party A) paying a third party to scrape data from another one of our customers (party B) which in turn affects the performance of party A's site.…

And how do you get your 'inventory data'? Aren't you scraping (or using scraped data) yourself? Oh the irony :)

Re: Scrape like the big boys

#84

Post by CWuestefeld »

jadell wrote: post #56
scarygliders wrote: post #32

Earlier quoted context omitted.

Do you honestly believe all site scraper people/companies are ethical enough to go to whoever pays /them/ to scrape data from a competitor's site and say "oh they offer an API to access this data let's pay for that", instead of "why pay for that data when we can scrape it right off their site"? Also, not all types of company will provide API endpoints. It all depends on the type of site - for example, an online shop…

I run a large scraper farm against several large sites. They're not online shops, and we don't compete with them. But they do have hundreds of thousands of data points that we use to provide reports and analytics for our clients, who also do not compete with the sites. I absolutely would pay for an API that provides that data. I'd be willing to pay 10x more than the cost of maintaining and running the scrapers. But t…

Have you tried approaching those sites and asking them to provide an API, pointing out that it would be easier for both of you in the long run? Or are you just assuming they wouldn't do it.

Because right now, I sure wish that the bots - which comprise probably 2/3 of my traffic - are causing me huge headaches and I wish that the people doing it would tell me what the heck they want.

Re: Scrape like the big boys

#85

Post by wolverine876 »

biosed wrote: post #37

I used to lead Sys Eng for a FTSE 100 company. Our data was valuable but only for a short amount of time. We were constantly scraped which cost us in hosting etc. We even seen competitors use our figures (good ones used it to offset their prices, bad ones just used it straight). As the article suggest, we couldn't block mobile operator IPs, some had over 100k customers behind them. Forcing the users to login did litt…

> But our most fruitful effort was when we removed limits and started giving "bad" data. By bad I mean alter the price up or down by a small percentage. ... If the customer made a transaction on the altered figure we we informed them and took it at the correct price.

Is that legal? It would be a big blow to trust if I was the customer, but that's without knowing what you were selling and in what market.

Re: Scrape like the big boys

#86

Post by anon9001 »

kerokerokero wrote: post #76

Thanks for the share. Great stuff. I used to scrape websites to generate content for higher SERPs. Ended up going into the adult industry lols. ( https://javfilms.net )

Neat! I've run across your site organically :P

I've always wondered, and since you're right here... how do sites like this make money?

It looks like you're probably crawling all the JAV vendors, finding free clips of today's releases, embedding them in your own site to draw traffic, and making money with affiliate links to buy the full content?

Am I missing anything? It seems hard to believe you'd get enough affiliate signups to make it worthwhile.

I can imagine your site as being a few hours a year of script maintenance and a money printer, or a 40hr/week SEO job with 1000s of similar sites across the adult industry.

I'd love to know anything you're willing to share about how the business works.

Re: Scrape like the big boys

#87

Post by joncp »

joekrill wrote: post #14

A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.

An all-too-common occurrence in HN comments as well.

Re: Scrape like the big boys

#88

Post by funnyflamigo »

scarygliders wrote: post #31
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Right with you there. I had a particularly bad time not so long ago, when a customer's site - a shop - was brought to its knees because someone, probably a competitor, hired some scraper-company of some sort to scrape every product and price. The scraper would systematically go through every single product page. And by scraper, I mean - 100's of them. All at the same time, using the old trick of 1 scraper requesting…

> Seriously, I hate scrapers. I hate the people who make scrapers. I hate their lack of ethics. Fuck those guys.

Wait till you find out what half of Google's business is based on (spoiler - scraping).

I really don't think scraping itself is an issue 90% of the time. It's the behavior of the out of control scrapers that are the problem. A well behaved scraper should barely be noticeable, if at all.

Re: Scrape like the big boys

#89

Post by funnyflamigo »

neals wrote: post #68

In a particularly hard to scrape website, using some kind of bot protection that I just couldn't reliably get working (if anybody wants to know what that was exactly, I'll go and check it) I now have a small Intel NUC running with firefox that listens to a local server and uses Temper Monkey to perform commands. Works like a charm and I can actualy see what it's doing and where it's going wrong. (though it's not scal…

I'm curious what bot protection it was? It couldn't have been trying too hard unless you were employing multiple anti-fingerprinting techniques, I'm assuming you used firefox's built in anti-fingerprinting?
Post reply on HN