Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bypassing website anti-scraping protections

kb.apify.com

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Bypassing website anti-scraping protections

#61

Post by PaulHoule »

vinceguidry wrote: post #5
cm2187 wrote: post #2

There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

Think different. Try bing, it has an API.

I think bing is close to Google in quality. Some people might even like it better. On the other hand I think DDG is the Sprint of search engines.

Google used to have a search API and they discontinued it because they said most of the people who used it were SEO people.

People who do pay-per-click are into A/B testing and other quantitative testing. Google is all for you doing that if you pay for advertising. Their mainstay of anti-SEO is doing arbitrary and random things to make it impossible for SEOs to go at it quantitatively. (They have patents on this!)

One reason so many sites go to a harvesting business model is that once a site is established you can make the slightest change and then your search rankings plummet. If you depend on search engine traffic it is a huge risk that you can't do anything about unless you are about.com (bought a 'competitive' search engine and just might be able to make an antitrust case against Google.)

Re: Bypassing website anti-scraping protections

#62

Post by PaulHoule »

fauigerzigerk wrote: post #57
vinceguidry wrote: post #5

Earlier quoted context omitted.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

>Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Unfortunately that is not the case. Many paywalled sites will let googlebot index their content but block other crawlers. They may have good reasons for doing that in some cases, but as a consequence the level playing field you're talking about no longer exists. Also, the purpose of using Google as part of som…

I have had web crawlers from China crawl my site multiple times a day but never send me traffic. Same with Yandex. I like the bing search engine but often it does not like my site. If it doesn't send any traffic, why let them run up my AWS bill?

Re: Bypassing website anti-scraping protections

#63

Post by hangonhn »

PaulHoule wrote: post #61
vinceguidry wrote: post #5

Earlier quoted context omitted.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

Think different. Try bing, it has an API. I think bing is close to Google in quality. Some people might even like it better. On the other hand I think DDG is the Sprint of search engines. Google used to have a search API and they discontinued it because they said most of the people who used it were SEO people. People who do pay-per-click are into A/B testing and other quantitative testing. Google is all for you doing…

Can you elaborate more on this statement? "On the other hand I think DDG is the Sprint of search engines."

I've been interested in switching to DDG for a while but as a former Sprint customer, that statement scares me but maybe some explanation from you might understand your opinion better.

Re: Bypassing website anti-scraping protections

#64

Post by JeremyBanks »

PaulHoule wrote: post #61
vinceguidry wrote: post #5

Earlier quoted context omitted.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

Think different. Try bing, it has an API. I think bing is close to Google in quality. Some people might even like it better. On the other hand I think DDG is the Sprint of search engines. Google used to have a search API and they discontinued it because they said most of the people who used it were SEO people. People who do pay-per-click are into A/B testing and other quantitative testing. Google is all for you doing…

> I think bing is close to Google in quality. Some people might even like it better. On the other hand I think DDG is the Sprint of search engines.

Isn't DDG just a Bing wrapper with a few frills in the results?

Re: Bypassing website anti-scraping protections

#65

Post by fauigerzigerk »

PaulHoule wrote: post #62
fauigerzigerk wrote: post #57

Earlier quoted context omitted.

>Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Unfortunately that is not the case. Many paywalled sites will let googlebot index their content but block other crawlers. They may have good reasons for doing that in some cases, but as a consequence the level playing field you're talking about no longer exists. Also, the purpose of using Google as part of som…

I have had web crawlers from China crawl my site multiple times a day but never send me traffic. Same with Yandex. I like the bing search engine but often it does not like my site. If it doesn't send any traffic, why let them run up my AWS bill?

I understand that, but I think there are good reasons why we shouldn't always act in the narrowest sense of our self interest (provided we have enough financial wiggle room).

A search monopoly is not good for website owners. It makes us very dependent on the whims of that monopolist.

If you block all crawlers that don't already have a large market share and send back a lot of traffic, you're killing any possibility for new competitors to get a foot in the door.

Also, you're killing any chance for something unexpected to happen, such as someone having a great idea based on crawled data that could change all our lives for the better without ever sending traffic to your site.

Now, I'm not telling you what you can and cannot afford. If crawlers cost me a ton of money that I don't have I would certainly act exactly like you suggested.

Re: Bypassing website anti-scraping protections

#66

Post by zephyrfalcon »

vinceguidry wrote: post #5
cm2187 wrote: post #2

There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

The irony is that if every site protected themselves from web scraping, there would be no Google.

Re: Bypassing website anti-scraping protections

#68

Post by mlevental »

unreal37 wrote: post #59
learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

You had a paid API, and people wanted the information for free.... Not unexpected I guess.

you've misunderstood: the site used something like algolia (which the site paid for) to index. the scrapers were hitting that service (which was costing the site) rather than parsing the xml (which already had everything).

Re: Bypassing website anti-scraping protections

#69

Post by xenomachina »

greglindahl wrote: post #44
userbinator wrote: post #3

For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…

From my experience at blekko, 99.9% of the "people" who go deep into the results pages for a single query are actually bots. You're a very unusual user, and there are a lot of bots.

There's a difference between going deep into the results, and progressively refining a query. The former is pretty indicative of bot behavior -- humans rarely go past even the first page of results. I do the latter all the time, and this frequently gets me Google's captcha, especially if I'm doing something like using site: and inurl: operators.

Re: Bypassing website anti-scraping protections

#70

Post by learc83 »

unreal37 wrote: post #59
learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

You had a paid API, and people wanted the information for free.... Not unexpected I guess.

The paid API was just to handle the search of the company database (couldn't change that for political reasons).

They weren't getting any information that they couldn't get through the XML.

Post reply on HN