Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrape like the big boys

incolumitas.com

Post reply on HN
Print view View original

51–60 of 196 posts

Re: Scrape like the big boys

#51

Post by jtdev »

scarygliders wrote: post #34
purerandomness wrote: post #19

Earlier quoted context omitted.

Why do you think are they trying to circumvent it? Does your API provide all the information that can be found on the site, or are they scraping because the API is incomplete? We've once had to scrape Amazon product pages because they have a lot of API endpoints, but those didn't contain the data we needed.

Why would Amazon wish to provide you with easy to access data on their products and prices when you could either be a competitor wishing to undercut those prices, or be a scraper company hired by such a competitor? In what universe is providing such a straightforward way of helping a competitor considered sane business practice?

In the end, they still get the data, just in a much less desirable way for both you and the customer.

Re: Scrape like the big boys

#52

Post by abc03 »

I scrap government sites a lot as they don't provide apis. For mobile proxies, I use the proxidize dongles and mobinet.io (free, with Android devices). As stated in the article, with cgNAT it's basically impossible to block them as in my case, half the country couldn't access the sites anymore (if you place them in several locations and use one carrier each there).

Re: Scrape like the big boys

#53

Post by matheusmoreira »

scarygliders wrote: post #32
kulikalov wrote: post #29

Earlier quoted context omitted.

Why not create api endpoint and charge mild cost for that data? You’ll make money instead of spending it.

Do you honestly believe all site scraper people/companies are ethical enough to go to whoever pays /them/ to scrape data from a competitor's site and say "oh they offer an API to access this data let's pay for that", instead of "why pay for that data when we can scrape it right off their site"? Also, not all types of company will provide API endpoints. It all depends on the type of site - for example, an online shop…

> Why would an online shop do that?

Because otherwise the HTML will become the API.

Re: Scrape like the big boys

#54

Post by matheusmoreira »

scarygliders wrote: post #34
purerandomness wrote: post #19

Earlier quoted context omitted.

Why do you think are they trying to circumvent it? Does your API provide all the information that can be found on the site, or are they scraping because the API is incomplete? We've once had to scrape Amazon product pages because they have a lot of API endpoints, but those didn't contain the data we needed.

Why would Amazon wish to provide you with easy to access data on their products and prices when you could either be a competitor wishing to undercut those prices, or be a scraper company hired by such a competitor? In what universe is providing such a straightforward way of helping a competitor considered sane business practice?

Because they will get the data regardless of what you do and if you don't make an API it will cost you more due to overhead.

Re: Scrape like the big boys

#55

Post by matheusmoreira »

purerandomness wrote: post #19
ebbp wrote: post #5

Earlier quoted context omitted.

We do offer an API - the scrapers are trying to circumvent using that, presumably.

Why do you think are they trying to circumvent it? Does your API provide all the information that can be found on the site, or are they scraping because the API is incomplete? We've once had to scrape Amazon product pages because they have a lot of API endpoints, but those didn't contain the data we needed.

This is the number one reason to scrape websites. It's always nice when there's an API with documentation and rate limiting rules you can follow. Sometimes the data I need just isn't there, though. Then I open up their site and find a huge amount of private API endpoints that do exactly what I want. Then I open up a ticket about it and it gets 200 replies but they ignore it for years. It's fucking stupid and it's really no wonder people scrape their site.

Re: Scrape like the big boys

#56

Post by jadell »

scarygliders wrote: post #32
kulikalov wrote: post #29

Earlier quoted context omitted.

Why not create api endpoint and charge mild cost for that data? You’ll make money instead of spending it.

Do you honestly believe all site scraper people/companies are ethical enough to go to whoever pays /them/ to scrape data from a competitor's site and say "oh they offer an API to access this data let's pay for that", instead of "why pay for that data when we can scrape it right off their site"? Also, not all types of company will provide API endpoints. It all depends on the type of site - for example, an online shop…

I run a large scraper farm against several large sites. They're not online shops, and we don't compete with them. But they do have hundreds of thousands of data points that we use to provide reports and analytics for our clients, who also do not compete with the sites.

I absolutely would pay for an API that provides that data. I'd be willing to pay 10x more than the cost of maintaining and running the scrapers.

But the sites being scraped have no interest in that.

Re: Scrape like the big boys

#57

Post by 0xdeadbeefbabe »

scarygliders wrote: post #41
zivkovicp wrote: post #36

Earlier quoted context omitted.

Well, you don't need an api, just a CSV file with a catalog. The scraping company WILL use the API/CSV file... they will probably also still charge their customer for scraping, so it's a win-win :D You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. You can be principled and insist on blocking bots and spen…

> You can think of it this way, the prices and product data are publicly visible already on the website, there are no real secrets, none of it is password protected. There's the problem right there. The prices and product data are publicy visible - because there is a target audience of /humans/ for whom the site is designed and intended to be used by. The site is not there to cater for a competitor's scrapers. I don'…

Let's not encourage these unethical people to even think of using human eyeballs and manual data entry for their scraping instead of bots. That sounds pretty darn unethical.

Re: Scrape like the big boys

#58

Post by matheusmoreira »

scarygliders wrote: post #31
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Right with you there. I had a particularly bad time not so long ago, when a customer's site - a shop - was brought to its knees because someone, probably a competitor, hired some scraper-company of some sort to scrape every product and price. The scraper would systematically go through every single product page. And by scraper, I mean - 100's of them. All at the same time, using the old trick of 1 scraper requesting…

> Seriously, I hate scrapers. I hate the people who make scrapers. I hate their lack of ethics. Fuck those guys.

Not everybody in this space is out to destroy your site. Some of us actively try to put as little load on your site as possible. My scraper puts less load on sites than I do when I browse them normally, I've measured it. Really sucks when we get lumped together with the other abusers and blocked.

Re: Scrape like the big boys

#59

Post by weird-eye-issue »

scarygliders wrote: post #34
purerandomness wrote: post #19

Earlier quoted context omitted.

Why do you think are they trying to circumvent it? Does your API provide all the information that can be found on the site, or are they scraping because the API is incomplete? We've once had to scrape Amazon product pages because they have a lot of API endpoints, but those didn't contain the data we needed.

Why would Amazon wish to provide you with easy to access data on their products and prices when you could either be a competitor wishing to undercut those prices, or be a scraper company hired by such a competitor? In what universe is providing such a straightforward way of helping a competitor considered sane business practice?

Man your comment is hilarious because in fact Amazon DOES provide an API for exactly that

Re: Scrape like the big boys

#60

Post by weird-eye-issue »

gmanis wrote: post #9
ebbp wrote: post #5

Earlier quoted context omitted.

We do offer an API - the scrapers are trying to circumvent using that, presumably.

Is it not viable to put majority of your data behind a login and so the bots only get a very limited snapshot while legitimate users get it through a free login? I’m asking this because I’m going through very similar situation and would love to see other opinions around this.

You are defining legitimate users as those that have a valid session cookie? Good luck
Post reply on HN