Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bypassing website anti-scraping protections

kb.apify.com

Post reply on HN
Print view View original

81–90 of 121 posts

Re: Bypassing website anti-scraping protections

#81

Post by bo1024 »

unreal37 wrote: post #59
learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

You had a paid API, and people wanted the information for free.... Not unexpected I guess.

After searching "algolia" mentioned below, I figured out the misunderstanding. The company was paying somebody else per search made on their web site. So every time a scraper called the website's search function, it cost the website money.

Re: Bypassing website anti-scraping protections

#82

Post by textmode »

vinceguidry wrote: post #5
cm2187 wrote: post #2

There is an irony in google preventing web scraping given that their business is pretty much built on web scraping.

Why is there irony in that? Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. Google protecting its site from scraping means you can't compete with Google using Google's own resources . That said, automated research fascinates me, I wouldn't want to scrape Google to make my own Google, but rather to make private repositories of information that I can then quer…

But as he describes it, the GP is not trying to "compete" with Google (whatever that means), he is only trying to do some comprehensive searches.

He is not selling advertising.

He is not even running a public website.

Google is preventing you from using automation to create private (i.e. personal) respositories of information, even when that information is public and (ironically) Google itself relied on automation ("bots") to collect it.

Re: Bypassing website anti-scraping protections

#83

Post by sydd »

fauigerzigerk wrote: post #65
PaulHoule wrote: post #62

Earlier quoted context omitted.

I have had web crawlers from China crawl my site multiple times a day but never send me traffic. Same with Yandex. I like the bing search engine but often it does not like my site. If it doesn't send any traffic, why let them run up my AWS bill?

I understand that, but I think there are good reasons why we shouldn't always act in the narrowest sense of our self interest (provided we have enough financial wiggle room). A search monopoly is not good for website owners. It makes us very dependent on the whims of that monopolist. If you block all crawlers that don't already have a large market share and send back a lot of traffic, you're killing any possibility f…

> It makes us very dependent on the whims of that monopolist.

Very true. Only allow Google and you are helping them to build their monopoly. And if they have full monopoly they do what they want - including asking you money to be included in the search results.

Re: Bypassing website anti-scraping protections

#84

Post by 0x00000000 »

textmode wrote: post #45
0x00000000 wrote: post #23

Earlier quoted context omitted.

Instagram is the worst I have come across. If you are on a page with 1000+ pictures trying to find something near the bottom, you have to let it load each new group sequentially, then after a while it starts timing you out for like 60 seconds or longer every couple times you load more. God forbid you accidentally navigate away while scrolling you have to start all over again from the top. Due to recent events it seem…

This sounds like an issue that is specific to Javascript-controlled browsers. If using a traditional, non-Javascript tcp/tls/http client it is trivial to extract the image urls and other information from the page using a single HTTP request (and from each successive page using more HTTP requests in a single connection, if "has_next_page" is "true"). No "API" needed. Can you provide an example of a single page with 10…

https://www.instagram.com/ryuji513

It looks like it just hits https://www.instagram.com/graphql/query/... every time you scroll down so if you scroll too fast it just hammers it and throttles your requests to that endpoint.

Re: Bypassing website anti-scraping protections

#86

Post by tracker1 »

hangonhn wrote: post #63
PaulHoule wrote: post #61

Earlier quoted context omitted.

Think different. Try bing, it has an API. I think bing is close to Google in quality. Some people might even like it better. On the other hand I think DDG is the Sprint of search engines. Google used to have a search API and they discontinued it because they said most of the people who used it were SEO people. People who do pay-per-click are into A/B testing and other quantitative testing. Google is all for you doing…

Can you elaborate more on this statement? "On the other hand I think DDG is the Sprint of search engines." I've been interested in switching to DDG for a while but as a former Sprint customer, that statement scares me but maybe some explanation from you might understand your opinion better.

I'm not sure about the comparison itself... I've tried DDG several times, I search for technical things in generic ways a lot. DDG almost never gives me what I want in the first page. Google almost always does.

Re: Bypassing website anti-scraping protections

#87

Post by leesalminen »

tracker1 wrote: post #86
hangonhn wrote: post #63

Earlier quoted context omitted.

Can you elaborate more on this statement? "On the other hand I think DDG is the Sprint of search engines." I've been interested in switching to DDG for a while but as a former Sprint customer, that statement scares me but maybe some explanation from you might understand your opinion better.

I'm not sure about the comparison itself... I've tried DDG several times, I search for technical things in generic ways a lot. DDG almost never gives me what I want in the first page. Google almost always does.

Same here. It’s hard to blame DDG though- Google’s search index of Stack Overflow is better than SO’s own.

Re: Bypassing website anti-scraping protections

#89

Post by MightySCollins »

learc83 wrote: post #31

A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…

We have kind of the same thing. All data is in an API which is less than a cheap VPS and display messages to them about using the API if they get blocked but they just come back with a new IP every time.

Re: Bypassing website anti-scraping protections

#90

Post by PaulHoule »

tracker1 wrote: post #86
hangonhn wrote: post #63

Earlier quoted context omitted.

Can you elaborate more on this statement? "On the other hand I think DDG is the Sprint of search engines." I've been interested in switching to DDG for a while but as a former Sprint customer, that statement scares me but maybe some explanation from you might understand your opinion better.

I'm not sure about the comparison itself... I've tried DDG several times, I search for technical things in generic ways a lot. DDG almost never gives me what I want in the first page. Google almost always does.

Maybe if I used DDG more I would learn to parse the results better but the first thing I see are many results that have a "dark pattern" appearance to me.

Often I do get a good result on the first page but often results #1-#N vary from third rate to non-sequitur and then result #N+1 is the one that should be at #1, where maybe N is drawn from Uniform(3,6). I see this so much I can't imagine it is an accident. If anything it seems to be 70% more evil than Google.

Post reply on HN