Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoiding bot detection: How to scrape the web without getting blocked?

github.com

Post reply on HN
Print view View original

271–280 of 312 posts

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#271

Post by matheusmoreira »

gzer0 wrote: post #137
abadger9 wrote: post #102

I'm a lead engineer on the search team of a publicly traded company who's bread and butter is this domain. I was curious about this list, it candidly misses the mark- the tech mentioned in this blog is what you might get if you hired a competent consultant to build out a service without having domain knowledge. In my experience, what's being used on the bleeding edge is two steps ahead of this.

I have a considerable amount of experience in the industry. Some of these so-called "advanced" techniques: * We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below). * We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point. * All you need…

> Mobile LTE data users are behind CGNATfor IPV4.

> You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.

This is similar how Tor is supposed to work in practice. Make everyone look like the same user so these companies can't tell who's who.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#272

Post by marginalia_nu »

matheusmoreira wrote: post #266
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script? Why is Google the only one who can do it? Must they have yet another monopoly? > Bots with seconds between the requests rarely get blocked even by CDNs. I've had scripts that made 1 request per day get…

> Why is Google the only one who can do it? Must they have yet another monopoly?

I run a search engine and do my own crawling, and this does not correspond to my view of reality at all.

I have had almost no problems with getting blocked. If I have gotten blocked, it's usually been my own fault and I've been able to get unblocked by sending them an email explaining I run a search engine and asking for forgiveness because my bot wasn't behaving well.

The bots that do get blocked are bots in most cases bots that misbehave, ignore robots.txt, fetch the same resources repeatedly or with insane crawl-delays.

There are a few rare exceptions, but the whole "why does Google get a free pass when I don't?"-angle just doesn't hold water at all.

> I've had scripts that made 1 request per day get blocked for no reason. Not to mention the endless cloudflare javascript bullshit they made me support for it to even work.

Google doesn't repeat requests every day. I don't repeat requests every day. That's a weird thing to do, and it's well within a site owner's prerogative to block that nonsense.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#273

Post by jhgb »

marginalia_nu wrote: post #234
hattmall wrote: post #181

Earlier quoted context omitted.

403 is per request, not requester. I get random 403s when just browsing some websites. Does that mean I should close the browser and not hit refresh for fear of breaking some wire fraud unauthorized access law?

If you go by the semantics of what the 403 code means, absolutely, that's excatly what the status code means. In practice there's of course nuance, like anyone will occasionally type in the wrong password on a log-in screen, maybe try again and then realize it was the wrong log-in prompt. That's mostly fine. That's different from deliberate trying to circumvent a measure like this. If you are doing the stuff in the l…

I thought the semantics of 403 was "look for the actual semantics in the response body".

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#274

Post by matheusmoreira »

marginalia_nu wrote: post #272
matheusmoreira wrote: post #266

Earlier quoted context omitted.

> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script? Why is Google the only one who can do it? Must they have yet another monopoly? > Bots with seconds between the requests rarely get blocked even by CDNs. I've had scripts that made 1 request per day get…

> Why is Google the only one who can do it? Must they have yet another monopoly? I run a search engine and do my own crawling, and this does not correspond to my view of reality at all. I have had almost no problems with getting blocked. If I have gotten blocked, it's usually been my own fault and I've been able to get unblocked by sending them an email explaining I run a search engine and asking for forgiveness beca…

> Google doesn't repeat requests every day. I don't repeat requests every day.

I scrape one site whose content changes every second. How is it "nonsense" to make one request every 24 hours? I make hundreds, thousands when I browse their site normally using my browser.

We've got people in this very thread talking about bots making hundreds of requests per second. How is one request every 24 hours harming anyone? People told me to make one request per hour to avoid hammering their servers, I decided to wait a day instead. It boggles my mind that this generous interval could possibly be considered abuse. How long should the interval be then? A month? A year? Infinitely long so the scraper never makes requests?

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#275

Post by charcircuit »

leppr wrote: post #267
charcircuit wrote: post #141

Earlier quoted context omitted.

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

With many of these big anti-bot services like Google ReCaptcha, it's not even specialized anonymity tools that can cause shadow banning, just unusual user-agents. All of these have independently caused me to get into endless ReCaptcha loops: firefox on android, smartphone with unusual screen resolution, clean browser profile with VPN. It's so common that I now default to using duckduckgo, which never blocks me. I dou…

As a person who also uses mobile Firefox, I don't feel like I personally have issues with recaptcha.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#276

Post by charcircuit »

selfhoster11 wrote: post #229
charcircuit wrote: post #141

Earlier quoted context omitted.

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

> The internet can't function if people are allowed to be anonymous. You must have missed the first 20 or so years of its existence, if that's your position.

Okay let's go back to before I was born when people still used IRC and let's say you hated someone else's IRC server. You can just use a program to flood their server with garbage messages. In order to try and stop this spam they first try and deanonymize where this traffic is coming from. This can be done by looking at the IP that these bots are coming from. Now they can gline you and the flood ends. Now let's say the internet didn't leak your IP deanonymizing you. What are they to do? They essentially are forced to lock down the server and whitelist it. They can not allow anonymous users to join or else risk just being flooded.

Stopping abuse has always been a game of trying to deanonymize users in order to try and ban the harmful ones.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#277

Post by colinmhayes »

matheusmoreira wrote: post #266
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script? Why is Google the only one who can do it? Must they have yet another monopoly? > Bots with seconds between the requests rarely get blocked even by CDNs. I've had scripts that made 1 request per day get…

I built a search engine in school. I was lazy and stupid with the scraper and ended up writing a bug that caused it to loop on certain sites. That lead to the entire Duke law school site being DOS's during their class sign up period. Sorry for ruining scraping for everyone, but this is why websites don't want people scraping them.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#279

Post by bryan_w »

charcircuit wrote: post #141
azalemeth wrote: post #106

Earlier quoted context omitted.

You very much missed the false positive rate! I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. I had to try to log in to my bank about ten times today because their js-crap website didn't like me ( grumble why does it even need to ask for my desktop's accelerometer data via js...) Stuff like this is a pain beyond pain. I rea…

>I'm fed up of being classed as a bot just because I browse with uMatrix, a Linux user agent, and a ton of ad filtering and anonymisation tech. Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

To kinda tweak this since people do tend to like their anonymity, "Do you have to be anonymous to all parties, all the time?"

Parent poster trusts his bank, and his bank would trust his once it knows he's not an fraudster, so maybe it's in everyone's interest to just allow the javascript for that one site.

Re: Avoiding bot detection: How to scrape the web without getting blocked?

#280

Post by hbgl »

matheusmoreira wrote: post #266
marginalia_nu wrote: post #36

If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Trying to circumvent it is besides being legally questionable, a serious pain in the ass for the site owner and makes websites more prone to attempt to block bots in general. Also, in my experience, most websites that block your bot, block your bot because your bot is too aggressive, or because you are fetching…

> If someone is signalling to you you that they do not want your bot on their site, then maybe respect that? Maybe respect user freedom? If I can access the data using my browser, why can't I access it using my script? Why is Google the only one who can do it? Must they have yet another monopoly? > Bots with seconds between the requests rarely get blocked even by CDNs. I've had scripts that made 1 request per day get…

> Why is Google the only one who can do it? Must they have yet another monopoly?

Google can do it because most website operators want them to index their site. Plus, it is trivial to tell google to stop. That goes for all search engines.

Post reply on HN